Paper deep dive

Get Confused Cautiously: Textual Sequence Memorization Erasure with Selective Entropy Maximization

Zhaohan Zhang, Ziquan Liu, Ioannis Patras

Year: 2024Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 82

Models: GPT-Neo-125M, GPT-Neo-1.3B, GPT-Neo-2.7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 6:16:02 PM

Summary

The paper introduces Entropy Maximization with Selective Optimization (EMSO), a framework for erasing Textual Sequence Memorization (TSM) in Large Language Models (LLMs). EMSO uses a reference-free contrastive gradient metric to identify and update only the most influential weights for entropy maximization, effectively balancing TSM erasure with the preservation of model utility without requiring additional data or models.

Entities (4)

LLMs · technology · 99%EMSO · framework · 98%TSM · phenomenon · 95%Contrastive Gradient Metric · methodology · 92%

Relation Signals (3)

EMSO → appliedto → LLMs

confidence 95% · To achieve a better trade-off between the effectiveness of TSM erasure and model utility in LLMs, our paper proposes a new framework

EMSO → erases → TSM

confidence 95% · proposes a new framework based on Entropy Maximization with Selective Optimization (EMSO) for TSM erasure

Contrastive Gradient Metric → optimizes → EMSO

confidence 90% · the updated weights are chosen with a novel contrastive gradient metric

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Language Models (LLMs) have been found to memorize and recite some of the textual sequences from their training set verbatim, raising broad concerns about privacy and copyright issues when using LLMs. This Textual Sequence Memorization (TSM) phenomenon leads to a high demand to regulate LLM output to prevent it from generating certain memorized text to meet user requirements. However, our empirical study reveals that existing methods for TSM erasure fail to forget massive memorized samples without substantially jeopardizing the model utility. To achieve a better trade-off between the effectiveness of TSM erasure and model utility in LLMs, our paper proposes a new framework based on Entropy Maximization with Selective Optimization (EMSO), where the updated weights are chosen with a novel contrastive gradient metric without any participation of additional model or data. Our analysis shows that training with the entropy maximization loss has a more stable optimization process and better keeps model utility than existing methods. The contrastive gradient metric localizes the most influential weight for TSM erasure by taking both the gradient magnitude and direction into consideration. Extensive experiments across three model scales demonstrate that our method excels in handling large-scale forgetting requests while preserving model ability in language generation and reasoning.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

81,400 characters extracted from source content.

Expand or collapse full text

Get Confused Cautiously: Textual Sequence Memorization Erasure with Selective Entropy Maximization Zhaohan Zhang, Ziquan Liu, Ioannis Patras Queen Mary University of London, London, UK zhaohan.zhang, ziquan.liu, i.patras@qmul.ac.uk Preprint. Zhaohan Zhang is the corresponding author. Abstract Large Language Models (LLMs) have been found to memorize and recite some of the textual sequences from their training set verbatim, raising broad concerns about privacy and copyright issues when using LLMs. This Textual Sequence Memorization (TSM) phenomenon leads to a high demand to regulate LLM output to prevent it from generating certain memorized text to meet user requirements. However, our empirical study reveals that existing methods for TSM erasure fail to forget massive memorized samples without substantially jeopardizing the model utility. To achieve a better trade-off between the effectiveness of TSM erasure and model utility in LLMs, our paper proposes a new framework based on Entropy Maximization with Selective Optimization (EMSO), where the updated weights are chosen with a novel contrastive gradient metric without any participation of additional model or data. Our analysis shows that training with the entropy maximization loss has a more stable optimization process and better keeps model utility than existing methods. The contrastive gradient metric localizes the most influential weight for TSM erasure by taking both the gradient magnitude and direction into consideration. Extensive experiments across three model scales demonstrate that our method excels in handling large-scale forgetting requests while preserving model ability in language generation and reasoning. Get Confused Cautiously: Textual Sequence Memorization Erasure with Selective Entropy Maximization Zhaohan Zhang†thanks: Preprint. Zhaohan Zhang is the corresponding author., Ziquan Liu, Ioannis Patras Queen Mary University of London, London, UK zhaohan.zhang, ziquan.liu, i.patras@qmul.ac.uk 1 Introduction Large Language Models (LLMs) are a series of transformers-based models pre-trained on an enormous corpus with trillions of tokens, achieving human-level performance on language abilities. Vaswani et al. (2017); Brown et al. (2020); Touvron et al. (2023); Achiam et al. (2023). While the utility of LLMs greatly benefits from scaling laws Kaplan et al. (2020), recent studies reveal that LLMs have Textual Sequence Memorization (TSM), i.e., memorizing and emitting training samples verbatim, including Personally Identifiable Information (PII) and copyrighted content Carlini et al. (2021); Huang et al. (2022); Jagielski et al. (2022). This phenomenon raises serious concerns about violating the regulation of the right to be forgotten (RTBF) Mantelero (2013); Graves et al. (2021). Hence, erasing TSM from LLMs is in great demand to protect PII and intellectual property. Figure 1: Illustration of erasure-utility trade-off with the example of three methods: Gradient Ascent (GA) Jang et al. (2023), Deliberate Imagaination (DI) Dong et al. (2024) and our EMSO. The top figure shows the exemplary erasure-utility trade-off with different cases. The bottom figure demonstrates the quantitative erasure-utility trade-off with the correlation between TSM metric (Memorization Accuracy, MA) and generation quality metric (MAUVE). There are two major types of memorization erasure for LLMs in existing literature. 1) Knowledge erasure focuses on the removal or modification of abstract knowledge, such as factual associations Wang et al. (2024b); Meng et al. or hazardous knowledge Li et al. (2024); Liu et al. (2024) within LLMs. These works evaluate the model acquisition of unwanted knowledge using question-answering tasks in the form of classification. For example, Weapons of Mass Destruction Proxy (WMDP) benchmark Li et al. (2024) constructs a dataset of multiple-choice questions to serve as a proxy measurement of hazardous knowledge and evaluate the efficacy of knowledge erasure with classification accuracy drop. 2) Textual sequence memorization erasure prevents the model from generating sequences with high verbatim similarity with training data Carlini et al. (2021, 2022); Barbulescu and Triantafillou (2024). Compared with knowledge memorization in classification tasks, TSM has a closer relationship with current privacy and copyright challenges in LLMs given the fact that the most popular LLMs are generative models. Thus, recent works within the scope of TSM erasure are commonly evaluated on open-end generation tasks (i.e., continuation based on given prefix) Jang et al. (2023); Kassem et al. (2023); Yao et al. (2023). This work focuses on erasing TSM of user-designated data from LLMs. Memorized data is deeply tied to general language modelling Huang et al. (2024), making it hard to remove without reducing model utility. As shown in Fig.1, current methods often either erase TSM or maintain model utility, creating an erasure-utility trade-off dilemma. Existing erasure methods rely on references like memorized models Ilharco et al. (2022); Li et al. (2023); Eldan and Russinovich (2023) or retained data Liu et al. (2022); Wang et al. (2023) to manage model utility. However, these references introduce challenges: memorized models can compromise privacy, while acquiring and maintaining retained data can be impractical. Furthermore, Maini et al. (2024) has observed instances of model collapse when attempting to erase extensive memorized data in one operation. Recognizing the limitations of previous work mentioned above, we aim to improve the TSM erasure while preserving model utility with three desired properties: (i) erasing without involving a memorized model to avoid privacy issues; (i) erasing with only access to forget set without a retain set; (i) erasing with a large-scale forget set to accommodate large-scale erasure requests. To tackle these challenges, we design a novel framework for TSM erasure, entropy maximization with selective optimization (EMSO). The proposed objective function is to increase the entropy of the predictive distribution on a forget set to encourage more diverse output instead of penalizing the generation of memorized tokens. Moreover, to keep the original model utility, we apply a minimally invasive surgery to the model by only updating the most significant weights for entropy maximization. To be specific, we design a novel reference-free metric that takes both gradient magnitude and direction into consideration. This metric helps to locate weights that contribute positively to entropy maximization while negatively to token memorization. Extensive experiments show that our method achieves a better erasure-utility trade-off when processing massive erasure requests compared with recent baselines. Our contribution is summarized as follows: • We introduce a reference-free optimization objective to enhance forgetting of large-scale memorized data in LLMs. This objective aims to increase predictive distribution entropy, which proves to be a more stable optimization target compared to commonly used gradient ascent and label smoothing methods, supported by both theoretical analysis and empirical findings. • We propose a selective optimization approach that only updates salient weights selected by a contrastive gradient metric to achieve a better erasure-utility trade-off. The metric prefers a weight that is significant only for entropy maximization but not memorization based on gradient magnitude and direction. • Our empirical study demonstrates that our EMSO method for TSM erasing achieves the best trade-off between information leakage and model utility on a large-scale forget dataset across various metrics and model sizes. Figure 2: Framework of EMSO. We select the most significant weights for forgetting based on the gradient magnitude and direction so that the weight is effective at entropy maximization instead of data memorization. 2 Related Works Knowledge Unlearning for LLMs. Machine unlearning aims at removing model memorization about sensitive data. In contrast to traditional unlearning approaches in classification tasks Bourtoule et al. (2021); Chundawat et al. (2023); Jia et al. (2023), the concept of machine unlearning in generative LLMs shifts focus to the characteristics of model output. Specifically, it focuses on mitigating harmful or biased information in generated content. Abstract harmful knowledge is one of the targets for LLM unlearning, which bears similarities with safety alignment but primarily uses negative samples Li et al. (2024); Liu et al. (2024); Yao et al. (2023). Question-answering-based benchmarks such as TOFU Maini et al. (2024) and WMDP Li et al. (2024) are established for evaluating model acquisition of the knowledge. Given the objective and testbase, rejection-based methods such as Direct Preference Optimization (DPO) Rafailov et al. (2024) are suitable for encouraging the model to answer malicious questions. The evaluation metrics for quantifying hazardous knowledge in LLMs include accuracy on malicious multiple-choice questions Li et al. (2024) and GPT-as-a-Judge score Liu et al. (2024). TSM Erasure in LLMs TSM refers to LLMs’ ability to memorize and emit training samples verbatim, which is an undesired attribute to be erased/unlearned Carlini et al. (2021, 2022). Different from undesired knowledge, TSM is defined over certain training data points. For evaluation, the updated model is asked to generate continuation based on the prefix of memorized data. Training Data Extraction Challenge 111https://github.com/google-research/lm-extraction-benchmark serves as a persuasive benchmark for probing TSM in GPT-Neo model family. Concurrent work MUSE Shi et al. (2024) provides news and books corpus for evaluating verbatim memorization. The extent of memorization is calculated by the similarity between the output (before or after decoding) of the original model and updated model Jang et al. (2023); Barbulescu and Triantafillou (2024); Wang et al. (2024a). Gradient ascent Jang et al. (2023) is a straightforward approach for erasing TSM by maximizing the probability of wrong prediction for samples in the forget set. Other objectives such as Deliberate Imagination Dong et al. (2024) and Negative Preference Optimization Zhang et al. (2024) are proposed to avoid model collapse during model updation. Recent works also try to localize the specific model units where the TSM is stored. For example, Wu et al. localizes privacy neurons with gradient integration and deactivates the identified neurons to protect private information. Jia et al. (2023); Fan et al. (2024) indicate weight saliency is informative for locating model units that are beneficial to unlearning. Our work, as a localization-informed method for TSM erasure, differs from the above-mentioned works in (i) proposing a new stabler objective for TSM erasure; (i) taking gradient direction into consideration without retain data for localizing important weights. 3 Methodology EMSO reference-freely removes TSM by selecting weights to be updated by contrastive gradient metric and optimizing towards entropy maximization objective. The workflow of EMSO is shown in Figure 2. 3.1 Problem Definition Let i=(x1,…,xp,…,xp+q)subscriptsubscript1…subscript…subscript x_i=(x_1,...,x_p,...,x_p+q)italic_xitalic_i = ( x1 , … , xitalic_p , … , xitalic_p + q ) be a textual sequence where x1:psubscript:1x_1:px1 : p is the prefix and xp+1:qsubscript:1x_p+1:qxitalic_p + 1 : q is the original continuation. Given a forget set Df∈DsubscriptD_f∈ DDitalic_f ∈ D, where D is the pre-training dataset for an LLM θosubscript _oθitalic_o, the objective of TSM erasure is to obtain an updated model θusubscript _uθitalic_u which performs exactly the same as a model only trained on D⁢\⁢Df D_fD \ Ditalic_f, i.e., dataset which is obtained by deleting DfsubscriptD_fDitalic_f from D. This goal implies that the updated model θusubscript _uθitalic_u should keep its utility on D⁢\⁢Df D_fD \ Ditalic_f as same as the original model θosubscript _oθitalic_o while showing "unmemorization" effect222The unmemorization effect refers to model’s disability to recite the text sequence in DfsubscriptD_fDitalic_f verbatim. on DfsubscriptD_fDitalic_f. Ideally, the updated model θusubscript _uθitalic_u can be obtained by pre-training an LLM from scratch with D⁢\⁢Df D_fD \ Ditalic_f. However, due to the prohibitive computational cost it requires Yao et al. (2023), such a solution is commonly recognized as unrealistic Liu et al. (2024); Wang et al. (2023); Jang et al. (2023). In this work, we aim to directly update θosubscript _oθitalic_o with access only to forget set DfsubscriptD_fDitalic_f to approximate the performance of θusubscript _uθitalic_u on both DfsubscriptD_fDitalic_f and D⁢\⁢Df D_fD \ Ditalic_f. 3.2 Entropy Maximization Entropy is the measurement of the uncertainty of a probability distribution P. In the context of LLM generation, a larger entropy on the next token probability Pθ⁢(xi|x<i)subscriptconditionalsubscriptsubscriptabsentP_θ(x_i|x_<i)Pitalic_θ ( xitalic_i | x< i ) indicates that the model is uncertain about its decision on the current decoding token, leading to a higher probability to select other reasonable tokens and output more diverse content. Importantly, this diversity helps prevent the model from memorizing specific sequences. We find entropy maximization is Thus, we propose to maximize the entropy of Pθ⁢(xi|x<i)subscriptconditionalsubscriptsubscriptabsentP_θ(x_i|x_<i)Pitalic_θ ( xitalic_i | x< i ) on DfsubscriptD_fDitalic_f by minimizing the following loss function ℒE⁢Msubscriptℒ _EMLitalic_E M =1q⁢∑i=1q∑y∈||Pθ,yi⁢log⁢Pθ,yi,absent1superscriptsubscript1subscriptsubscriptsuperscriptlogsubscriptsuperscript = 1q _i=1^q _y∈|V|P^i_θ,y% logP^i_θ,y,= divide start_ARG 1 end_ARG start_ARG q end_ARG ∑i = 1q ∑y ∈ | V | Pitalic_iitalic_θ , y log Pitalic_iitalic_θ , y , (1) Pθ,yisubscriptsuperscript P^i_θ,yPitalic_iitalic_θ , y =P⁢(x(p+i)=y|x<(p+i−1);θ),absentsubscriptconditionalsubscriptabsent1 =P(x_(p+i)=y|x_<(p+i-1);θ),= P ( x( p + i ) = y | x< ( p + i - 1 ) ; θ ) , (2) where p,qp,qp , q are the lengths of the prefix and continuation, respectively. Pθ,yisubscriptsuperscriptP^i_θ,yPitalic_iitalic_θ , y denotes the probability of predicting the i-th token to be y, VV is the vocabulary and |||V|| V | is its cardinality. Compared with commonly used objectives, we theoretically prove that entropy maximization objective helps stabilize the model updation process during TSM erasure in the following section. Comparison to Label Smoothing Loss and Gradient Ascent Loss. Label smoothing loss Müller et al. (2019); Dong et al. (2024) and gradient ascent loss Liu et al. (2022); Jang et al. (2023); Wang et al. (2023) have emerged as two popular objectives for TSM erasure. As a new learning objective, our EM loss is more stable during the optimization. The gradient analysis shows that the minimizer of the label smoothing loss is identical to the maximizer of the EM loss. For each token i, the label smoothing loss is as follows, ℒl⁢s=−γ⁢∑j=1||log⁡p^i⁢j,subscriptℒsuperscriptsubscript1subscript L_ls=-γ _j=1^|V| p% _ij,Litalic_l s = - γ ∑j = 1| V | log over start_ARG p end_ARGi j , (3) where p^i⁢j=P⁢(x(p+i)=j|x<(p+i−1);θ)subscript^subscriptconditionalsubscriptabsent1 p_ij=P(x_(p+i)=j|x_<(p+i-1);θ)over start_ARG p end_ARGi j = P ( x( p + i ) = j | x< ( p + i - 1 ) ; θ ) for simplicity and γ is the hyperparameter of the label smoothing loss. As p^i⁢jsubscript p_ijover start_ARG p end_ARGi j is the output of the softmax function with hi⁢jsubscriptℎh_ijhitalic_i j as the input, we take the derivative of the loss function with respect to the input hi⁢jsubscriptℎh_ijhitalic_i j, ∂ℒl⁢s∂hi⁢ksubscriptℒsubscriptℎ ∂L_ls∂ h_ikdivide start_ARG ∂ Litalic_l s end_ARG start_ARG ∂ hitalic_i k end_ARG =−γ⁢∑j=1||1p^i⁢j⁢∂p^i⁢j∂hi⁢kabsentsuperscriptsubscript11subscript^subscript^subscriptℎ =-γ _j=1^|V| 1 p_ij % ∂ p_ij∂ h_ik= - γ ∑j = 1| V | divide start_ARG 1 end_ARG start_ARG over start_ARG p end_ARGi j end_ARG divide start_ARG ∂ over start_ARG p end_ARGi j end_ARG start_ARG ∂ hitalic_i k end_ARG (4) =−γ⁢∑j≠k||1p^i⁢j⁢(−p^i⁢j⁢p^i⁢k)−γ⁢1p^i⁢k⁢(p^i⁢k−p^i⁢k2)absentsuperscriptsubscript1subscript^subscript^subscript^1subscript^subscript^superscriptsubscript^2 =-γ _j≠ k^|V| 1 p_ij(-% p_ij p_ik)-γ 1 p_ik( p_ik- p_% ik^2)= - γ ∑j ≠ k| V | divide start_ARG 1 end_ARG start_ARG over start_ARG p end_ARGi j end_ARG ( - over start_ARG p end_ARGi j over start_ARG p end_ARGi k ) - γ divide start_ARG 1 end_ARG start_ARG over start_ARG p end_ARGi k end_ARG ( over start_ARG p end_ARGi k - over start_ARG p end_ARGi k2 ) =−γ⁢[−(||−1)⁢p^i⁢k+1−p^i⁢k]absentdelimited-[]1subscript^1subscript =-γ[-(|V|-1) p_ik+1- p_ik]= - γ [ - ( | V | - 1 ) over start_ARG p end_ARGi k + 1 - over start_ARG p end_ARGi k ] =−γ⁢(1−||⁢p^i⁢k).absent1subscript =-γ(1-|V| p_ik).= - γ ( 1 - | V | over start_ARG p end_ARGi k ) . It is trivial to get that the minimizer of the function is ∀kfor-all∀ k∀ k, p^i⁢k=1/||subscript^1 p_ik=1/|V|over start_ARG p end_ARGi k = 1 / | V |, which is equivalent to the optimum of the maximum entropy loss. We next derive the gradient of ℒE⁢MsubscriptℒL_EMLitalic_E M with respect to the logits, ∂ℒE⁢M∂hi⁢k=∑j=1||(log⁡p^i⁢j+1)⁢∂p^i⁢j∂hi⁢k.subscriptℒsubscriptℎsuperscriptsubscript1subscript^1subscript^subscriptℎ ∂L_EM∂ h_ik= _j=1^|% V|( p_ij+1) ∂ p_ij∂ h_ik.divide start_ARG ∂ Litalic_E M end_ARG start_ARG ∂ hitalic_i k end_ARG = ∑j = 1| V | ( log over start_ARG p end_ARGi j + 1 ) divide start_ARG ∂ over start_ARG p end_ARGi j end_ARG start_ARG ∂ hitalic_i k end_ARG . (5) Comparing the gradient 4 with the gradient 5, the only difference is the first term. As p^i⁢j∈[0,1]subscript^01 p_ij∈[0,1]over start_ARG p end_ARGi j ∈ [ 0 , 1 ], the scale and gradient of log⁡p^i⁢jsubscript p_ijlog over start_ARG p end_ARGi j is much smaller than that of −1/p^i⁢j1subscript^-1/ p_ij- 1 / over start_ARG p end_ARGi j, we provide an illustration in Appendix A for reference. Note that the gradient scale analysis result is also applicable to gradient ascent loss (details are in Appendix A), indicating that the gradient ascent loss also has the risk of unstable optimization. In summary, our entropy maximization loss has the same optimization objective but much more stable gradients compared with the label smoothing loss and gradient ascent. In Section 4.2, our experiment results corroborate the gradient analysis. 3.3 Weight Selection with Contrastive Gradient To achieve a better trade-off between erasure effectiveness and model utility, we propose to only finetune weights that are salient to forgetting and keep other weights the same as the original to preserve model utility. We select weights from all attention heads and the multi-layer perceptron (MLP) block in every layer l because they are components of the "residual block" which acts as communication channels in transformers-based architectures Elhage et al. (2021). Inspired by the gradient-based input salient maps Adebayo et al. (2018); Yona and Greenfeld (2021), we use weight saliency ∇θE⁢M∈ℝL×C×D∇subscriptsuperscriptℝ∇ _EM ^L× C× D∇ θitalic_E M ∈ blackboard_RL × C × D with respect to ℒE⁢MsubscriptℒL_EMLitalic_E M as a metric for selecting influential weights333L is the number of layers, C is the number of candidate blocks, i.e., attention heads and MLP blocks, D denotes the dimension of weight vector. Please note that for simplicity of notation, the denotation assumes that D is the same across layers and components.: ∇θE⁢M=∂ℒE⁢M∂θ∇subscriptsubscriptℒ∇ _EM= _EM∂θ∇ θitalic_E M = divide start_ARG ∂ Litalic_E M end_ARG start_ARG ∂ θ end_ARG (6) However, maximizing the entropy of output distribution updates θosubscript _oθitalic_o towards a more diverse output but not precise "unmemorization". Thus, we design a contrastive gradient strategy to select weights that are both salient with respect to ℒE⁢MsubscriptℒL_EMLitalic_E M and contributive to unmemorization. Taking inspiration from previous works Zhang et al. (2023); Eldan and Russinovich (2023) which train a memorization model on the forget set by minimizing ℒN⁢L⁢L=−1q⁢∑i=1qlog⁢(Pθ,xp+i)subscriptℒ1superscriptsubscript1logsubscriptsuperscriptsubscriptL_NLL=- 1q _i=1^qlog(P^i_θ,x_p+i)Litalic_N L L = - divide start_ARG 1 end_ARG start_ARG q end_ARG ∑i = 1q log ( Pitalic_iitalic_θ , x start_POSTSUBSCRIPT p + i end_POSTSUBSCRIPT ), we consider the gradient direction with respect to ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L minimization as "memorization direction". Thus, we propose an updated metric M∈ℝL×CsuperscriptℝM ^L× CM ∈ blackboard_RL × C taking both direction and magnitude into consideration: M=cos(∇θN⁢L⁢L,∇θE⁢M)|∇θE⁢M|D,∇θN⁢L⁢L=∂ℒN⁢L⁢L∂θ,formulae-sequencecos∇subscript∇subscript∇subscript∇subscriptsubscriptℒ splitM=cos(∇ _NLL,&∇ _EM) |∇% _EM| D,\\ ∇ _NLL=& _NLL∂θ, splitstart_ROW start_CELL M = cos ( ∇ θitalic_N L L , end_CELL start_CELL ∇ θitalic_E M ) divide start_ARG | ∇ θitalic_E M | end_ARG start_ARG square-root start_ARG D end_ARG end_ARG , end_CELL end_ROW start_ROW start_CELL ∇ θitalic_N L L = end_CELL start_CELL divide start_ARG ∂ Litalic_N L L end_ARG start_ARG ∂ θ end_ARG , end_CELL end_ROW (7) where cos( ⋅·⋅ ) is cosine similarity and |⋅||·|| ⋅ | is l1subscript1l_1l1 norm function. We scale the l1subscript1l_1l1 norm by 1D1 1 Ddivide start_ARG 1 end_ARG start_ARG square-root start_ARG D end_ARG end_ARG to eliminate the effect of various dimensions of different model components. The cosine similarity measures the disagreement of optimization direction between ℒE⁢MsubscriptℒL_EMLitalic_E M and ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L. |∇θE⁢M|∇subscript|∇ _EM|| ∇ θitalic_E M | measures the parameter saliency to the optimization of ℒE⁢MsubscriptℒL_EMLitalic_E M. Note that the direction of ∇θN⁢L⁢L∇subscript∇ _NLL∇ θitalic_N L L represents memorization and the direction of ∇θE⁢M∇subscript∇ _EM∇ θitalic_E M is the updation direction. If the cosine similarity is large (positive), it means this weight is optimized towards memorization, which is not desirable for a good trade-off. If the cosine similarity is small (negative), this weight is updated towards forgetting. Therefore, the selected weight should be optimized towards "forget direction" and be salient to ℒE⁢MsubscriptℒL_EMLitalic_E M, see Figure 2 for the illustration. Thus, we obtain the block-wise weight mask m according to M: m=1⁢(t⁢o⁢p⁢k⁢(−M)),m1m=1(topk(-M)),m = 1 ( t o p k ( - M ) ) , (8) where 1⁢(top⁢k⁢(g))1topg1(topk(g))1 ( top k ( g ) ) is an element-wise indicator which labels 1 for the top-k element in g. In practice, we empirically observe that setting k to 2 yields sufficiently effective performance. We show the influence of different k in Appendix B. The updating process of the original model θosubscript _oθitalic_o can be expressed as: θu←θo−α⁢m⊙∇θE⁢M,←subscriptsubscriptdirect-productm∇subscript _u← _o- ∇ _EM,θitalic_u ← θitalic_o - α m ⊙ ∇ θitalic_E M , (9) where ⊙direct-product ⊙ denotes element-wise product and α is learning rate. Our experiment selects one batch randomly as input to calculate m. 4 Experiments Memorization Language Generation Ability Ranking Method EL3 ↓ ↓ MA ↓ ↓ S ↓ ↓ Perplexity ↓ ↓ Rep2 ↓ ↓ Div3 ↑ ↑ Coherence ↑ ↑ MAUVE ↑ ↑ Erasure Generation Avg. GPT-Neo-125M Original 0.212 0.789 0.587 27.69 0.123 0.923 0.566 0.702 N/A N/A N/A w/ M TA 0.117 0.677 0.500 26.48 0.346 0.737 0.554 0.276 4 4 4 CD 0.105 0.621 0.419 48.26 0.17 0.861 0.570 0.445 2 3 =2 w/ RD GD 0 0 0.021 5.30 0.956 0.037 0.092 0.023 N/A N/A N/A KL 0 0.007 0.023 1.91 0.990 0.010 0.032 0.038 N/A N/A N/A w/o REF GA 0 0 0.012 2.36 0.990 0.010 0.051 0.011 N/A N/A N/A DI 0.109 0.744 0.485 47.64 0.060 0.965 0.555 0.568 3 2 =2 EMSO (ours) 0.065 0.615 0.459 27.33 0.105 0.940 0.572 0.610 1 1 1 GPT-Neo-1.3B Original 0.371 0.953 0.688 16.99 0.090 0.94 0.597 0.762 N/A N/A N/A w/ M TA 0.151 0.682 0.467 18.98 0.377 0.733 0.552 0.328 3 3 3 CD 0.263 0.788 0.494 52.43 0.376 0.644 0.527 0.286 4 4 4 w/ RD GD 0 0.002 0.006 535.82 0.038 0.922 0.011 0.02 N/A N/A N/A KL 0 0 0 5.71 0.944 0.051 0.034 0.01 N/A N/A N/A w/o REF GA 0 0 0.061 3.49 0.984 0.023 0.066 0.015 N/A N/A N/A DI 0.138 0.751 0.457 64.27 0.057 0.967 0.527 0.591 2 2 2 EMSO (ours) 0.135 0.623 0.431 21.92 0.090 0.900 0.598 0.694 1 1 1 GPT-Neo-2.7B Original 0.377 0.966 0.744 13.83 0.083 0.953 0.599 0.790 N/A N/A N/A w/ M TA 0.059 0.441 0.037 11.91 0.611 0.474 0.532 0.137 N/A N/A N/A CD 0.287 0.858 0.493 36.12 0.348 0.686 0.476 0.371 3 3 3 w/ RD GD 0 0 0.061 477.89 0.019 0.759 0.061 0.028 N/A N/A N/A KL 0 0 0.006 2.51 0.938 0.065 0.030 0.035 N/A N/A N/A w/o REF GA 0 0 0.033 2.07 0.992 0.012 0.032 0.010 N/A N/A N/A DI 0.225 0.792 0.525 22.78 0.059 0.952 0.583 0.692 2 2 2 EMSO (ours) 0.242 0.701 0.415 19.06 0.095 0.947 0.608 0.713 1 1 1 Table 1: Experiment result of different TSM erasure methods on models with various scales. The best and the second-best results are highlighted in bold and underline respectively. We rank the erasure and generation ability of an updated model by the times they achieve best/second best in corresponding metrics. We mark the collapse models with red and do not count collapse models in the ranking. 4.1 Experiment Setup We describe the models, data, baselines and evaluation metrics of our experiment in this section. Detailed experiment setup is in Appendix C. 4.1.1 Test Model and Forget Set Model. We use the GPT-Neo model family (with 125M, 1.3B, 2.7B parameters) for evaluation because (i) they are proven to memorize and emit training sample verbatim and (i) they are widely used in previous works Jang et al. (2023); Dong et al. (2024); Barbulescu and Triantafillou (2024) to evaluate TSM. Data. We use the dataset from Training Data Extraction Challenge as the forget set, which is a subset of Pile Corpora Gao et al. (2020) and demonstrated to be easy-to-extract from pretrained GPT-Neo model family. This dataset consists of 15,000 text sequences with a length of 200 tokens, which is ideal for evaluating TSM erasure with large forgetting requests. Compared with TOFU benchmark Maini et al. (2024), the Extraction Challenge data is naturally memorized in the pretrainng stage of LLMs while TOFU simulates memorization by post hoc finetuning on fictional data. Thus, we choose Extraction Challenge data instead of TOFU to test in a practical setting. 4.1.2 Comparison Methods We compare our method with seven state-of-the-art methods to reveal its effectiveness and model utility after model updating. We divided the methods into three categories: (i) Updating with Memorized Model (w/ M), which trains a model overfitting on forget set to act as a reference for forgetting, including Task Arithmetic (TA) Ilharco et al. (2022) and Contrastive Decoding (CD) Li et al. (2023) (i) Updating with Retain Data (w/ RD), which assumes the existence of Dr∈D⁢\⁢Dfsubscript _r∈ D D_fDitalic_r ∈ D \ Ditalic_f to maintain the model utility, including Gradient Difference (GD) Liu et al. (2022) and KL Divergence (KL) Wang et al. (2023). (i) Updating without Reference (w/o REF), which is a challenging setting that only requires the forget set and original model to complete the updation process, including Gradient Ascent (GA) Jang et al. (2023) and Deliberate Imagination (DI) Dong et al. (2024). Our method lies in the w/o REF category. The detailed description for comparison methods is in Appendix D. 4.1.3 Evaluation Metrics Evaluation Metrics for Memorization. We use three different metrics to comprehensively evaluate the effectiveness of TSM erasure from exact memorization Tirumala et al. (2022) and approximate memorization Ippolito et al. (2022) perspectives: (i) Extraction Likelihood (EL) Jang et al. (2023) compares n-grams overlap between generation from an updated model and the original continuation. The definition for EL is: ELn⁢(x)=∑i=1p+q−nOverlapn⁢(fθ⁢(x1:i),xi:p+q)p+q−n,Overlapn(a,b)=|n-gram⁢(a)∩n-gram⁢(b)||n-gram(a)|,formulae-sequencesubscriptELxsuperscriptsubscript1subscriptOverlapsubscriptsubscript:1subscript:subscriptOverlapn-gramn-gramn-gram(a) splitEL_n(x)=& _i=1^p+q-n% Overlap_n(f_θ(x_1:i),x_i:p+q)p+q-n,\\ Overlap_n&(a,b)= |n-gram(a) -gram(b)||% n-gram(a)|, splitstart_ROW start_CELL ELn ( x ) = end_CELL start_CELL divide start_ARG ∑i = 1p + q - n Overlapn ( fitalic_θ ( x1 : i ) , xitalic_i : p + q ) end_ARG start_ARG p + q - n end_ARG , end_CELL end_ROW start_ROW start_CELL Overlapn end_CELL start_CELL ( a , b ) = divide start_ARG | n-gram ( a ) ∩ n-gram ( b ) | end_ARG start_ARG | n-gram(a) | end_ARG , end_CELL end_ROW where fθ⁢(x1:i)subscriptsubscript:1f_θ(x_1:i)fitalic_θ ( x1 : i ) is the generation from model θ given prefix x1:isubscript:1x_1:ix1 : i and n-gram(⋅·⋅) is a list of n-grams for given sequence. (i) Memorization Accuracy (MA) Tirumala et al. (2022) for quantifying the model memorization of given sequence x: MA⁢(x)=∑i=p+1p+q−11⁢(argmax⁢(Pθ,i)=xi)q−1.MAxsuperscriptsubscript111argmaxsubscriptsubscript1 (x)= _i=p+1^p+q-11(% argmax(P_θ,i)=x_i)q-1.MA ( x ) = divide start_ARG ∑i = p + 1p + q - 1 1 ( argmax ( Pitalic_θ , i ) = xitalic_i ) end_ARG start_ARG q - 1 end_ARG . (i) Semantic Similarity (S) for evaluating the semantic-level resemblance between model generation fθ⁢(x1:p)subscriptsubscript:1f_θ(x_1:p)fitalic_θ ( x1 : p ) and original continuation xp+1:qsubscript:1x_p+1:qxitalic_p + 1 : q. We extract semantic embedding from text sequence with MiniLM Wang et al. (2020) and compute the cosine similarity between the embeddings. We introduce the metrics and datasets for model utility evaluation in Appendix E. Figure 3: Experiment results for language reasoning ability evaluation with GPT-Neo-125M (left), 1.3B (middle), and 2.7B (right) as target model. We report the average accuracy of the updated model on all six tasks in the legend. Our EMSO achieves the best performance on all three models compared with baselines. Figure 4: Illustration of erasure-utility trade-off for different methods on GPT-Neo-125M. We vary the erasure strength for different methods. The one with high MAUVE and low Perplexity while low on MA and EL3subscriptEL3EL_3EL3 is considered better (i.e., closer to the lower-right corner for MAUVE-MA and MAUVE-EL3subscriptEL3EL_3EL3 figures and to the lower-left corner for Perplexity-MA and Perplexity-EL3subscriptEL3EL_3EL3 figures). We do not plot the trade-off line for GD, KL, and GA here because they collapse before completing a single training epoch. 4.2 Experiment Results We show experiment results in Table 1 together with the erasure-utility trade-off curve in Figure 4 and language reasoning ability evaluation in Figure 3. We unveil the following five key insights: 1) Best performer. Our method shows the best erasure-utility tradeoff among all competitors with erasure and utility all ranking first among 125M, 1.3B, and 2.7B models. Moreover, Figure 4 shows that our EMSO achieves comparable erasure effectiveness while sacrificing MAUVE by less than 0.1 and keeping Perplexity at the same level. In contrast, CD, TA, and DI all compromise either MAUVE (TA, CD) or Perplexity (DI, CD) greatly to get satisfactory erasure performance. 2) Model collapse. All methods based on ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L i.e., GD, KL, GA, completely collapse. We categorize such collapse into two classes: text degeneration and gibberish generation. Text degeneration means the model starts to repeat the same token, indicated by extremely low perplexity, high repetition and low MAUVE. Gibberish generation means that the model outputs nonsense content, reflected by high perplexity and low MAUVE. We observe that GD for the 1.3B and 2.7B model fall into gibberish generation while other ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L-based methods show text degeneration. The unsatisfied performance of ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L-based methods demonstrates that optimizing ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L fails to keep the utility after erasure when processing massive requests even if the retain data is available. 3) Less affected reasoning ability. As shown in Figure 3, compared with the significant deterioration in language generation ability after updation, the language reasoning ability of the updated model appears to be more stable. Our EMSO still stands out among all competitors with average accuracy on six tasks dropping by 0.33% and 1.05% on 1.3B and 2.7B models and increasing by 0.78% on 125M models. The fluctuation of reasoning ability is within 3% except for collapsed models. It demonstrates that in LLMs, generation and reasoning ability are orthogonal to some extent and TSM erasure tends to destroy model capability in generation rather than reasoning. A similar phenomenon is also observed by Barbulescu and Triantafillou (2024). 4) TSM erasure is difficult to scale up. As model parameters scale from 125M to 2.7B, model memorization is stronger and harder to erase. When we scale the original model from 125M to 2.7B, EL3subscriptEL3EL_3EL3, MA and S increase by 0.165, 0.177 and 0.157 respectively. Moreover, with the same erasure strength, all erasure methods are less effective when applied to larger models. For example, the 2.7B model achieves 0.792 in MA updated by DI, which is even higher than that in the smaller original 125M model. 5) Evaluation bias. There exists a bias in different memorization metrics. In the case of DI, it always performs better in EL3subscriptEL3EL_3EL3 but weak in MA. For instance, when updating the 2.7B model, DI excels our method by 0.017 in EL3subscriptEL3EL_3EL3 but falls far behind in MA by 0.091. Thus, it is necessary to use diverse metrics to evaluate erasure effectiveness comprehensively to avoid possible bias. Method EL3↓ ↓ MA↓ ↓ Perplexity↓ ↓ MAUVE↑ ↑ Select & NLL 0.012 0.008 5.218 0.006 Random & EM 0.201 0.785 29.695 0.701 w/o Dir 0.080 0.590 28.112 0.529 Full & EM 0.074 0.598 37.423 0.387 Ours 0.070 0.573 26.831 0.602 Table 2: Ablation study results using different variants. The best and the second-best result is highlighted in bold and underlined text respectively. The collapsed model is marked with red and is not included when comparing results. 4.3 Ablation Study To validate the necessity of every component in our proposed method, we conduct ablation studies with the following settings. 1) Select & NLL erases TSM of forget set data by updating top-2 salient blocks with ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L. 2) Random & EM randomly selects blocks and finetunes them with ℒE⁢MsubscriptℒL_EMLitalic_E M. 3) w/o Dir selects top-k blocks with the largest |∇θE⁢M|∇subscript|∇ _EM|| ∇ θitalic_E M | to update with ℒE⁢MsubscriptℒL_EMLitalic_E M. 4) Full & EM updates the whole model with ℒE⁢MsubscriptℒL_EMLitalic_E M. The experiment results are reported in Table 2. Unsurprisingly, fine-tuning a model with ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L again leads to model collapse even if we only update the most salient weight. Randomly picking weights brings little change to the model as all metrics stay close to the original model. Moreover, our EMSO reduces MA by 0.017 and improves MAUVE by 0.073 compared with w/o Dir, demonstrating that taking the direction into consideration helps accurately locate blocks that are influential in updation and boost the erasure-utility trade-off. Fine-tuning the whole model with ℒE⁢MsubscriptℒL_EMLitalic_E M jeopardizes the model utility substantially to achieve similar erasure effectiveness of our method. These results corroborate the function of each component of our proposed EMSO for improving erasure effectiveness while preserving model utility. Figure 5: Illustration of the output change for memorized data and non-memorized data. We calculate the exact match between the original model and the updated model output given prefix from the forget set. 4.4 Discussion Analysis on Entropy Maximization Loss. We study different objectives’ effectiveness on memorized and non-memorized data to better understand the reason why entropy maximization loss helps achieve a better trade-off between TSM erasure and model utility. In practice, it is normal that the forget set consists of both memorized and non-memorized data because users are agnostic of whether their requests are memorized by a model in most cases. As a simulation, we sample 20 memorized data and 100 non-memorized data from the mem-nonmem split of the Pile dataset444https://github.com/googleinterns/localizing-paragraph-memorization/tree/main/paragraphs/gpt-neo-125M/preds. This dataset quantifies model memorization with exact match (EMatch) Nasr et al. (2023) which counts the number of matches between greedy-decoded tokens and ground truth tokens given the same prefix until the first mismatch. Since the length of continuation in this dataset is 50, EMatch=50EMatch50EMatch=50EMatch = 50 is the maximum value and means the model repeats the text sequence verbatim. We update model to forget the samples with ℒE⁢MsubscriptℒL_EMLitalic_E M and ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L, respectively, and calculate the EMatch between outputs from the original model and the updated model given the prefix of requests. As shown in Fig. 5, ℒE⁢MsubscriptℒL_EMLitalic_E M and ℒN⁢L⁢LsubscriptℒL_NLLLitalic_N L L have similar effects on forgetting memorized data. However, EMSO preserves non-memorized samples better while NLL-updated model changes completely on its generation on the non-memorized set. We hypothesize that it is because NLL is a targeted objective for penalizing the probability of generating tokens while EMSO works in an untargeted fashion thus preserving the model’s original ability on non-memorized data. Analysis on Selected Blocks. We count the selection frequency of fine-tuning blocks and report the result in Table 4. The selection process is conducted seven times before early stopping thus the max frequency should be seven. EMSO tends to select blocks at shallow layers with a total frequency of 11 out of 14, indicating that memorization is affected largely by shallow layers. In addition, WvsubscriptWW_vWv in shallow layers are most frequently selected, e.g., the L2⁢Wv⁢H11subscriptL2subscriptWsubscriptH11L_2W_vH_11L2 Wv H11555We name the blocks according to their position and function in the model with the format Llayer numberWksubscriptWW_kWk, WqsubscriptWW_qWq, WvsubscriptWW_vWv, WosubscriptWW_oWo, Cf⁢csubscriptCC_fcCf c, Cp⁢r⁢o⁢jsubscriptCC_projCp r o jHattention head number, where WksubscriptWW_kWk, WqsubscriptWW_qWq, WvsubscriptWW_vWv, WosubscriptWW_oWo represent the linear transformation matrix for K,Q,V and output in attention mechanism and Cf⁢csubscriptCC_fcCf c, Cp⁢r⁢o⁢jsubscriptCC_projCp r o j represent up projection and down projection matrix in MLP. is selected in every round, suggesting the value matrix is most significant among the K, Q, V in the attention mechanism of LLMs regarding memorization. This observation is consistent with He et al. (2024). Moreover, we study the forward attention patterns of L2⁢H11subscriptL2subscriptH11L_2H_11L2 H11 to interpret its role in model memorization in Appendix F. 5 Conclusion This paper presents EMSO, a framework for textual sequence memorization erasure and better erasure-utility trade-off for LLMs when processing massive requests for verbatim memorization erasure. We first show that entropy maximization is a better objective for TSM erasure than the commonly used NLL and label smoothing loss. Then we provide theoretical analysis on the gradient of different updating objectives and find out entropy maximization provides more stable gradients, helps stabilize the updation process and avoids model collapse. Moreover, we minimally invade the model by only updating blocks selected by a contrastive gradient metric for selecting salient blocks that optimize towards forgetting direction to get a better erasure-utility trade-off. Our experiment results demonstrate the efficacy of our method compared with six baselines. The discussion on erasure effectiveness for memorized and non-memorized data and the pattern of selected blocks also sheds light on studying TSM from data and model structure perspectives in LLMs. Limitations In this work, we step forward to achieving a better erasure-utility trade-off when erasing model TSM about massive data. However, several limitations still exist in our proposed method EMSO. First, although EMSO performs best among all baseline methods in terms of TSM erasure, there is still a portion of requests that are not erased completely. Second, despite the effectiveness of our contrastive gradient metric, it needs more memory to store both ∇ℒE⁢M∇subscriptℒ _EM∇ Litalic_E M and ∇ℒN⁢L⁢L∇subscriptℒ _NLL∇ Litalic_N L L in the training stage, which limits its application on larger models. Third, our methods greatly change the meaning of data in the forget set. Future work will focus on automatically detecting and editing only the privacy information in textual sequences while preserving the overall semantics of requests. Ethics Statement The goal of our work is to protect user privacy from leaking by LLMs. We redact the accurate privacy information used in the examples. All the datasets used in this work are public. We use the datasets consistent with their intended use. References Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Adebayo et al. (2018) Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 9525–9536. Barbulescu and Triantafillou (2024) George-Octavian Barbulescu and Peter Triantafillou. 2024. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097. Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439. Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650. Chundawat et al. (2023) Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. 2023. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7210–7217. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Dong et al. (2024) Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. 2024. Unmemorization in large language models via self-distillation and deliberate imagination. arXiv preprint arXiv:2402.10052. Eldan and Russinovich (2023) Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238. Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1:1. Fan et al. (2024) Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. 2024. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations. Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP). Graves et al. (2021) Laura Graves, Vineel Nagisetty, and Vijay Ganesh. 2021. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516–11524. Hamborg et al. (2017) Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: a generic news crawler and extractor. He et al. (2024) Haoze He, Juncheng Billy Li, Xuan Jiang, and Heather Miller. 2024. Sparse matrix in large language model fine-tuning. arXiv e-prints, pages arXiv–2405. Huang et al. (2022) Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2038–2047. Huang et al. (2024) Jing Huang, Diyi Yang, and Christopher Potts. 2024. Demystifying verbatim memorization in large language models. arXiv preprint arXiv:2407.17817. Ilharco et al. (2022) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations. Ippolito et al. (2022) Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. 2022. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546. Jagielski et al. (2022) Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, et al. 2022. Measuring forgetting of memorized training examples. In The Eleventh International Conference on Learning Representations. Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408. Jia et al. (2023) Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. 2023. Model sparsity can simplify machine unlearning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 51584–51605. Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577. Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Kassem et al. (2023) Aly Kassem, Omar Mahmoud, and Sherif Saad. 2023. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4360–4379. Li et al. (2024) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, et al. 2024. The wmdp benchmark: Measuring and reducing malicious use with unlearning. In Forty-first International Conference on Machine Learning. Li et al. (2023) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In The 61st Annual Meeting Of The Association For Computational Linguistics. Liu et al. (2022) Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR. Liu et al. (2024) Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058. Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. Mantelero (2013) Alessandro Mantelero. 2013. The eu proposal for a general data protection regulation and the roots of the ‘right to be forgotten’. Computer Law & Security Review, 29(3):229–235. (37) Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. In International Conference on Learning Representations. Müller et al. (2019) Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? Advances in neural information processing systems, 32. Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035. Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. Shi et al. (2024) Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. 2024. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460. Su et al. (2022) Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35:21548–21561. Tirumala et al. (2022) Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. Wang (2021) Ben Wang. 2021. Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX. https://github.com/kingoflolz/mesh-transformer-jax. Wang et al. (2023) Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. 2023. Kga: A general machine unlearning framework based on knowledge gap alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13264–13276. Wang et al. (2024a) Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. 2024a. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. arXiv preprint arXiv:2402.05813. Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788. Wang et al. (2024b) Yu Wang, Ruihan Wu, Zexue He, Xiusi Chen, and Julian McAuley. 2024b. Large scale knowledge washing. arXiv preprint arXiv:2405.16720. (54) Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. Yao et al. (2023) Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning. arXiv preprint arXiv:2310.10683. Yona and Greenfeld (2021) Gal Yona and Daniel Greenfeld. 2021. Revisiting sanity checks for saliency maps. In eXplainable AI approaches for debugging and diagnosis. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. Zhang et al. (2023) Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. 2023. Composing parameter-efficient modules with arithmetic operation. In Thirty-seventh Conference on Neural Information Processing Systems. Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868. Appendix A Difference in Minimizer for Label Smoothing Loss and Entropy Maximization Loss Figure 6: Illustration of minimizer scale and gradient difference between label smoothing loss and entropy maximization loss. The scale and gradient of entropy maximization loss are much smaller than those of label smoothing loss, indicating entropy maximization provides a more stable updating process. Following the notion in Section 3.2. The gradient ascent loss aims to maximize the following objective: ℒN⁢L⁢L=−log⁡p^i⁢j,subscriptℒsubscript L_NLL=- p_ij,Litalic_N L L = - log over start_ARG p end_ARGi j , (10) As p^i⁢jsubscript p_ijover start_ARG p end_ARGi j is the output of the softmax function with hi⁢jsubscriptℎh_ijhitalic_i j as input, we take the derivative of the loss function with respect to the input hi⁢jsubscriptℎh_ijhitalic_i j. ∂ℒN⁢L⁢L∂hi⁢ksubscriptℒsubscriptℎ ∂L_NLL∂ h_ikdivide start_ARG ∂ Litalic_N L L end_ARG start_ARG ∂ hitalic_i k end_ARG =−1p^i⁢j⁢∂p^i⁢j∂hi⁢kabsent1subscript^subscript^subscriptℎ =- 1 p_ij ∂ p_ij∂ h_% ik= - divide start_ARG 1 end_ARG start_ARG over start_ARG p end_ARGi j end_ARG divide start_ARG ∂ over start_ARG p end_ARGi j end_ARG start_ARG ∂ hitalic_i k end_ARG (11) Comparing Equation 11 and Equation 4, they share the same minimizer −1p^i⁢j1subscript^- 1 p_ij- divide start_ARG 1 end_ARG start_ARG over start_ARG p end_ARGi j end_ARG, indicating gradient acsend loss also have greater gradient which deteriorates stable optimization. Appendix B Influence of the number of selected blocks In this section, we study how the number of selected blocks would affect the erasure-utility trade-off. The experiment results are shown in Table 3. Num. Blocks EL3subscriptEL3EL_3EL3 MA Perplexity MAUVE 1 0.131 0.720 26.86 0.694 2 0.065 0.615 27.33 0.701 3 0.078 0.641 25.01 0.657 4 0.094 0.666 26.38 0.612 Table 3: Experiment Results on different numbers of selected blocks Fine-tuning 2 blocks with entropy maximization objective leads to the best erasure-utility trade-off with the lowest EL3subscriptEL3EL_3EL3 and MA, and highest MAUVE. Only updating one block leads to more information leakage as EL3subscriptEL3EL_3EL3 and MA increase by 0.066 and 0.105, respectively. Interestingly, selecting more updating blocks not only does not help erase TSM but also impairs model utility indicated by dropping on MAUVE. Appendix C Inplementation Details We report all hyperparameter settings and hardware information in our experiments. For updating GPT-Neo-125M, 1.3B, and 2.7B models, we set the batch size to 64, 16, and 8, respectively, and set gradient accumulation to 1, 4, 8 for simulating the same updation steps. To make a fair comparison, we set the learning rate as 1⁢e−51superscript51e^-51 e- 5 for all methods with AdamW as optimizer Loshchilov and Hutter (2018). We set early stop criteria for the updating process to be perplexity increased by 3% on the WikiText-103 validation set. We require the updating process to complete at least one epoch to make sure all forgetting requests are processed. We use 10,000 textual sequences randomly sampled from C News Hamborg et al. (2017) as retain data for GD and KL. For w/ M methods, we train the memorized model for 10 epochs on the forget set. For methods that require erasure strength γ setting, i.e., TA, CD, DI, we set γ to 0.05, 0.3, 3, respectively for results in Table 1. For the experiment result in Figure 4, we take γ from range [0.04,0.05,0.08,0.1]0.040.050.080.1[0.04,0.05,0.08,0.1][ 0.04 , 0.05 , 0.08 , 0.1 ], [0.5,0.6,0.7,0.8]0.50.60.70.8[0.5,0.6,0.7,0.8][ 0.5 , 0.6 , 0.7 , 0.8 ], [3,5,8,10]35810[3,5,8,10][ 3 , 5 , 8 , 10 ] for TA, CD, DI, respectively and report EMSO results after epochs from 1 to 7 since training epoch is the key parameter for controlling erasure strength in EMSO. We conduct all our experiments on a single NVIDIA Tesla A100 80GB GPU. Figure 7: KQ attention pattern analysis across all attention head in layer 2. We especially pay attention to head 11 because it is the most frequently selected head during weight selection process in EMSO. Appendix D Detailed Description of Comparison Methods We introduce the details about comparison methods which can be categorized as w/ M, w/ RD, and w/o REF: 1. The w/ M methods are as follows: Task Arithmetic (TA) Ilharco et al. (2022), which erases TSM by subtracting the memorization model weight from the original model and the process can be formularized as: θTA=θo−γ⋅θMemo.subscriptTAsubscript⋅subscriptMemo _TA= _o-γ· _Memo.θTA = θitalic_o - γ ⋅ θMemo . (12) where θTAsubscriptTA _TAθTA, θosubscript _oθitalic_o, θMemosubscriptMemo _MemoθMemo are the parameters of the updated model, original model, and memorized model, respectively. γ controls the erasure strength, larger γ indicates model memorization about the forget set is removed more completely at the cost of more severe utility destruction. Contrastive Decoding (CD) Li et al. (2023), which steers original model output away from the memorized model with the following operation: Pθ⁢(xi|x<i)=softmax⁢(zt−γ⋅RELU⁢(ztmemo−zt)),subscriptconditionalsubscriptsoftmaxsubscript⋅RELUsuperscriptsubscriptmemosubscriptP_θ(x_i|x<i)=softmax(z_t-γ·RELU(z_t % memo-z_t)),Pitalic_θ ( xitalic_i | x < i ) = softmax ( zitalic_t - γ ⋅ RELU ( zitalic_tmemo - zitalic_t ) ) , (13) where Pθ⁢(xi|x<i)subscriptconditionalsubscriptP_θ(x_i|x<i)Pitalic_θ ( xitalic_i | x < i ) is the next token prediction probability distribution of the updated model. ztsubscriptz_tzitalic_t, ztmemosuperscriptsubscriptmemoz_t^memozitalic_tmemo are the logits from the original model and the memorized model. RELU is the activation function. γ controls erasure strength. 2. The w/ RD methods are as follows: Gradient Difference (GD) Liu et al. (2022), which increases NLL on forget set while decreasing it on retain set. KL Divergence (KL) Wang et al. (2023), which preserves model utility by restraining KL-divergence between the output distribution of the updated model and the original model on the retain set. 3. The w/o REF methods are as follows: Gradient Ascent (GA) Jang et al. (2023), which penalizes each label token from text sequences in forget set. Deliberate Imagination (DI) Dong et al. (2024), which uses a label-smoothing loss to increase the sampling possibility on all tokens in the vocabulary other than memorized ones and can be formulated as: L=∑t=1TℒC⁢E⁢(zt+γ⁢1i,t,zs),superscriptsubscript1subscriptℒsubscriptsubscript1subscriptL= _t=1^TL_CE(z_t+γ 1_i,t,z_s),L = ∑t = 1T Litalic_C E ( zitalic_t + γ 1i , t , zitalic_s ) , (14) where ℒC⁢EsubscriptℒL_CELitalic_C E is cross entropy, 1i,tsubscript1 1_i,t1i , t is all-ones vector except for ground truth ones, and zt,zssubscriptsubscriptz_t,z_szitalic_t , zitalic_s are logits from teacher model and student model. γ is erasure strength. Appendix E Utility Evaluation Metrics and Datasets We test the language generation ability and reasoning ability of the updated model since they are the two most important functions of LLMs. (i) For language generation ability evaluation, we randomly sample 5,000 text sequences from Wikitext-103 dataset Merity et al. (2016) and take the first 32 tokens as input to language model for open generation. Following Su et al. (2022), we use perplexity, diversity, repetition, MAUVE, Semantic Coherence for evaluating the generation quality. We calculate perplexity with GPT-J-6B Wang (2021) and calculate semantic coherence with SimCSE Gao et al. (2021). (i) For language reasoning ability evaluation, we use a suite of popular NLP reasoning tasks, namely Piqa Bisk et al. (2020), ARC-Easy Clark et al. (2018), COPA Roemmele et al. (2011), PubmedQA Jin et al. (2019), Winogrande Sakaguchi et al. (2021) and Hellaswag Zellers et al. (2019) for comprehensive evaluation. Appendix F KQ Pattern Analysis on Most Frequently Selected Blocks Block Name Frequency L2⁢Wv⁢H11subscriptL2subscriptWsubscriptH11L_2W_vH_11L2 Wv H11 7 L11⁢Cp⁢r⁢o⁢jsubscriptL11subscriptCL_11C_projL11 Cp r o j 3 L3⁢Wo⁢H2subscriptL3subscriptWsubscriptH2L_3W_oH_2L3 Wo H2 2 L3⁢Wv⁢H11subscriptL3subscriptWsubscriptH11L_3W_vH_11L3 Wv H11 1 L1⁢Wo⁢H8subscriptL1subscriptWsubscriptH8L_1W_oH_8L1 Wo H8 1 Table 4: The frequency of updating blocks selection. As shown in Figure 7, the value matrix of layer 2, attention head 11 is selected in every weight selection round. To better understand the mechanism of how this particular block affects memorization, we conduct analysis on its attention pattern at the inference stage. To be specific, we study which previous tokens the attention head 11 in layer 2 pays attention to when decoding at the current step by calculating the normalized inner product of "keys" k and queries q in forward pass activations of attention block when provided certain memorized samples. As shown in Figure 7, L2⁢H11subscriptL2subscriptH11L_2H_11L2 H11 pays the most attention to "promotional" and "journalism" in the given prefix. Compared with other tokens such as "sign" and "you", L2⁢H11subscriptL2subscriptH11L_2H_11L2 H11 apparently concentrates on rare tokens with complex semantics in the input text sequence at the inference stage, indicating rare tokens might be functional in LLM memorization.

Instant research discovery