Paper deep dive
UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, Ivan Vulic
Models: GPT-Neo 125M, GPT-Neo 1.3B, GPT-Neo 2.7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:57:17 PM
Summary
The paper introduces UNDIAL (Unlearning via Self-Distillation on Adjusted Logits), a robust method for machine unlearning in Large Language Models (LLMs). Unlike existing direct-tuning methods like Gradient Ascent (GA) and Negative Preference Optimization (NPO) which suffer from instability and over-unlearning, UNDIAL uses self-distillation on a fixed target distribution to selectively reduce the influence of targeted tokens. The method demonstrates superior scalability, stability, and preservation of language model capabilities across various benchmarks.
Entities (6)
Relation Signals (3)
FUNDIAL ā isvariantof ā UnDIAL
confidence 100% Ā· This leads to a variant of our self-distillation method, where we adjust the distribution specifically for these key tokens.
UnDIAL ā uses ā Self-Distillation
confidence 100% Ā· Our approach leverages self-distillation to adjust logits and selectively reduce the influence of targeted tokens.
UnDIAL ā improvesupon ā Gradient Ascent
confidence 95% Ā· Unlike GA and NPO, which suffer from significant model capacity degradation as datasets scale and training extends, UNDIAL demonstrates strong robustness
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mitigating the retention of sensitive or private information in large language models is essential for enhancing privacy and safety. Existing unlearning methods, like Gradient Ascent and Negative Preference Optimization, directly tune models to remove unwanted information. However, these methods often become unstable because they fine-tune by maximizing cross-entropy loss, which is the opposite of traditional loss minimization in learning. This reversal creates instability, especially on larger datasets, as the model struggles to balance unlearning with maintaining language capacity, leading to over-unlearning. In this paper, we introduce UnDIAL (Unlearning via Self-Distillation on Adjusted Logits), a novel and robust unlearning method. Our approach leverages self-distillation to adjust logits and selectively reduce the influence of targeted tokens. This technique ensures smooth convergence and avoids catastrophic forgetting, even in challenging unlearning tasks with large datasets and sequential unlearning requests. Extensive experiments show that UnDIAL can achieve both robustness in unlearning and scalability while maintaining stable training dynamics and resilience to hyperparameter tuning.
Tags
Links
Trouble viewing inline? Open PDF directly ā
Full Text
68,414 characters extracted from source content.
Expand or collapse full text
UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models Yijiang River Dong 1 , Hongzhou Lin 2 , Mikhail Belkin 3 , Ramon Huerta 2 , Ivan Vulic 1 1 University of Cambridge 2 Amazon 3 UCSD Abstract Mitigating the retention of sensitive or private information in large language models is essen- tial for enhancing privacy and safety. Exist- ing unlearning methods, like Gradient Ascent and Negative Preference Optimization, directly tune models to remove unwanted information. However, these methods often become unstable because they fine-tune by maximizing cross- entropy loss, which is the opposite of tradi- tional loss minimization in learning. This re- versal creates instability, especially on larger datasets, as the model struggles to balance unlearning with maintaining language capac- ity, leading to over-unlearning. In this paper, we introduceUNDIAL(Unlearning via Self- Distillation onAdjustedLogits), a novel and robust unlearning method. Our approach lever- ages self-distillation to adjust logits and selec- tively reduce the influence of targeted tokens. This technique ensures smooth convergence and avoids catastrophic forgetting, even in chal- lenging unlearning tasks with large datasets and sequential unlearning requests. Extensive experiments show thatUNDIALcan achieve both robustness in unlearning and scalability while maintaining stable training dynamics and resilience to hyperparameter tuning. 1 Introduction The increasing widespread use of large language models (LLMs) (OpenAI, 2023; Microsoft, 2023; Touvron et al., 2023; Jiang et al., 2023) in user- facing applications raises significant privacy con- cerns. Trained on vast, unmoderated web data, these models risk unintentionally exposing Person- ally Identifiable Information (PII), such as names and addresses (Heikkil Ģ a, 2022; White, 2023). Fur- thermore, LLMs are vulnerable to malicious ex- ploitation, i.e. adversarial attacks (Carlini et al., 2021, 2023; Nasr et al., 2023), allowing confiden- tial data to be extracted and heightening concerns about data security with AI (Levine, 2023). In addition to these privacy risks, data protec- tion regulations such as the EUās General Data Protection Regulation (GDPR, 2016) and the Cal- ifornia Consumer Privacy Act (CCPA, 2018) en- force the āright to be forgotten,ā enabling individ- uals to request the removal of their personal data from online platforms. This creates an urgent need for techniques that allow LLMs to effectively āun- learnā and prevent the disclosure of specific infor- mationāa process known asLLM unlearning. Recent advances in LLM unlearning fall into two main categories. The first category involves using anauxiliary modelto explicitly memorize sensi- tive information, which is later removed from the original model using techniques such as contrastive decoding (Eldan and Russinovich, 2023; Yu et al., 2022; Huang et al., 2024; Ji et al., 2024) or param- eter merging (Ilharco et al., 2023; Chen and Yang, 2023). However, this approach introduces infras- tructure overhead and poses a significant risk if the auxiliary model is exposed, as it contains exactly the data meant to be forgotten. Another line of research focuses ondirectly tun- ingthe base LLM model to unlearn sensitive infor- mation, using techniques such as Gradient Ascent (GA) (Jang et al., 2023) and Negative Preference Optimization (NPO) (Zhang et al., 2024). These approaches are gaining more attention as they align more closely with the growing emphasis on AI safety (Gallegos et al., 2023; Åucki et al., 2024). Despite these advances, the recent unlearning benchmark MUSE (Shi et al., 2024) highlights a major drawback in current methods: applying unlearning to larger corpora leads to a decline in general language usefulness. This limits its usage in real-world settings, as an effective unlearning method must scale reliably with increasing data sizes, and accommodate continual updatesāall while maintaining the modelās overall language capabilities. In this work, we introduce a novel direct-tuning arXiv:2402.10052v2 [cs.CL] 16 Oct 2024 Figure 1: An illustration of the self-distillation process in the proposedUNDIALmethod: The original logits generated by the model are adjusted by subtracting the one-hot distribution of the target token. The student model is then fine-tuned to approximate this modified logit distribution. Since the adjustments rely solely on the original modelās outputs, this is a self-distillation process to de-emphasize the token to be forgotten. method,UNDIAL, which enables Unlearning via Self-Distillation with Adjusted Logits. As shown in Figure 1, we generate a target distribution by re- ducing the logit of the token to be unlearned. This target distribution is fixed during self-distillation, ensuring a stable optimization process. Unlike GA and NPO, which suffer from significant model ca- pacity degradation as datasets scale and training extends,UNDIALdemonstrates strong robustness to data scaling, hyperparameter tuning, and sequen- tial unlearning, offering the first robust unlearning method for direct tuning LLMs. Our main contributions are as follows.1)We identify the robustness issues in current unlearning methods and propose a new, more robust method based on self-distillation.2)We demonstrate the effectiveness and robustness ofUNDIALacross various hyperparameter settings, forget set sizes and a number of unlearning requests.3)We also ex- plore a variant ofUNDIALthat focuses solely on specific set of tokens like named entities or nouns, which can further improve its overall performance. 2 Background and Related Work 2.1 Memorization in Large Language Models The LLMsā issue of memorizing and precisely re- producing data is getting increasingly recognized, especially when they get prompted in specific ways (Carlini et al., 2021; Bender et al., 2021; Tirumala et al., 2022; McCoy et al., 2023). This memorization behavior, while useful for encap- sulating factual knowledge (Petroni et al., 2019; Khandelwal et al., 2020), also presents significant legal ramifications and challenges due to the un- intended memorization of private or copyrighted material. Such instances increase the susceptibil- ity of LLMs to extraction attacks or membership inference attacks (Carlini et al., 2021; Shokri et al., 2017; Mireshghallah et al., 2022). Recent stud- ies have shown that, as these models grow in size, the dynamics of memorization fasten, leading to a linear increase in the fraction of data that can be ex- tracted (Tirumala et al., 2022; Carlini et al., 2023). To amortize such dynamics, techniques such as data deduplication (Lee et al., 2022; Kandpal et al., 2022; Nguyen et al., 2020) or private training are studied (Yu et al., 2021; Tram ` er and Boneh, 2021), showing positive effect on reducing memorization. 2.2 Unlearning in Large Language Models Given the massive amounts of data involved in training LLMs, retraining these models each time to remove memorized data is impractical. Thus machine unlearning focuses on how to effectively eliminate unintentional memorized content after the model is trained (Cao and Yang, 2015; Gi- nart et al., 2019; Guo et al., 2020; Bourtoule et al., 2021). The unlearning algorithms can broadly fall into the following two categories: Direct Tuning Methods.Jang et al. (2023) first formalize the problem of LLM unlearning and pro- pose to use gradient ascent (GA) to achieve unlearn- ing. Instead of minimizing loss, GA maximizes the loss on tokens to be forgotten, forcing the model to forget specific knowledge. However, Zhang et al. (2024) note that GA causes rapid collapse. They propose Negative Preference Optimization (NPO) which diverges slower than GA both in theory and practice. Alternative approaches tune the model to say āI donāt knowā (Maini et al., 2024) or predict random labels (Yao et al., 2024) on the knowledge that should be forgotten. Leveraging Auxiliary Modelsto remove the mem- orization in the base LLM, bypassing direct tuning, is in the focus of another line of research. Eldan and Russinovich (2023); Ji et al. (2024) first fine-tune a model to memorize the forget set and then leverage contrastive decoding (CD) (Li et al., 2023) to sup- 0255075100125150175 Steps 0 2000 4000 6000 8000 10000 Forget Distance UNDIAL (ours) (lr=1e-4) UNDIAL (ours) (lr=3e-5) GA (lr=1e-4) GA (lr=3e-5) NPO (lr=1e-4) NPO (lr=3e-5) 0255075100125150175 Steps 0 500 1000 1500 2000 2500 3000 Retain Distance UNDIAL (ours) (lr=1e-4) UNDIAL (ours) (lr=3e-5) GA (lr=1e-4) GA (lr=3e-5) NPO (lr=1e-4) NPO (lr=3e-5) Figure 2:Training dynamics of Direct Tuning methods on the MUSE benchmark (Shi et al., 2024). MUSE divides data into two sets: the Forget set, containing the information to be unlearned, and the Retain set, which measures the impact of unlearning on unrelated knowledge. Ideally, unlearning should be precise, affecting only the Forget set without disturbing the Retain set. MUSE provides fine-tuned models for both sets as optimal reference points. To capture the training dynamics, we compute the average KL divergence between the unlearned model and the MUSE reference models over the Forget and Retain sets. An effective unlearning model should closely match both references, with near-zero divergence indicating successful unlearning and model performance preservation. press the generation of unwanted memorization at decode time. Task Arithmetic (TA) approaches (Il- harco et al., 2023) also fine-tune a model to mem- orize the forget set and leverage linear parameter merging (Matena and Raffel, 2022) to remove the memorization in model weights. Majmudar et al. (2022) apply linear interpolation with uniform dis- tribution at the decoding time and show that this satisfies certain differential privacy (DP) criteria. Chen and Yang (2023) tune multiple unlearning layers to handle sequential unlearning requests and then fuse and plug them back into the base LLM. We set aside post-processing methods such as di- rectlypromptingLLMs to add a guardrail (Thaker et al., 2024); our focus is on removing knowledge directly from the base LLM via fine-tuning. 3 Methodology Motivation with an Example.As highlighted by Zhang et al. (2024), Direct Tuning methods face a major challenge of instability. Methods like GA and NPO, designed to directly unlearn from the original model, often lead to the so-called over- unlearning issue, referring to that unlearning algo- rithm is continuously applied to the model even after the corresponding knowledge is unlearned. This often leads to the model to collapse with the model capacity dropping to zero. While NPO par- tially mitigates this by adding a regularization term to slow the rate of divergence,it fails to prevent long-term collapse in practice(Fan et al., 2024). To demonstrate this critical issue, we apply GA and NPO methods on the MUSE dataset and illus- trate the training dynamics in Figure 2. As shown in the results, both GA and NPO exhibit model collapsing, although NPO diverges more slowly than GA. Through this empirical analysis we fur- ther corroborate the observation from recent related work Shi et al. (2024); Fan et al. (2024),where both GA and NPO lead to over-unlearning and thus to a substantial decline in model usefulness and perfor- mance. Moreover, NPO is also very brittle to different hyperparameter setups and thus difficult to tune. In the early stages of training, the distance on the for- get set remains approximately constant (see Figure 2 left), showing that the model is not unlearning as expected. After this initial plateau stage, the model briefly begins to unlearn but quickly collapses. This instability reflects how sensitive NPO is to hyper- parameter tuning and the need to stop training at exactly the right moment. Even slight overshoot- ing can lead to severe performance degradation, i.e. over-unlearning. In contrast, foreshadowing our results,UNDIAL consistently shows robust performance throughout training, converging smoothly to a stable distri- bution. This stability allows for flexible stopping points without any risk of degradation.UNDIAL also achieves a substantially lower forget set dis- tance in far fewer steps, making itnot just robust, but highly efficient. As a Direct Tuning approach, we focus on comparing to GA and NPO in Figure 2. In Section 4, we will further show thatUNDIAL outperforms even methods that rely on auxiliary models, proving its superiority across the board. 3.1UNDIAL: Method Description The main ingredient of our method isself- distillation, where the model learns from its own predictions rather than external labels. Given an original modelM original and a sequencex 1:T that we aim to unlearn, the model generates a pre- softmax logit distribution at each tokentā[1,T]: logit original ā¼M original (Ā·|x <t ), representing a distribution over the vocabulary. To unlearn a specific tokenx t , we reduce its logit value, forming an adjusted distribution: logit adjusted ā¼logit original āĪ³e x t , wheree x t is a one-hot vector for tokenx t andĪ³is a hyperparameter controlling theunlearning strength. We then apply softmax to convert the adjusted log- its into a probability distributionp adjusted , which de-emphasizes the tokens to be unlearned. Since this adjustment is manually crafted, we perform self-distillation to learn the adjusted dis- tribution by optimizing the model parametersĪøso thatM Īø can approximate the adjusted logits. This is done by minimizing the following loss function: L= min Īø E xā¼D unlearn " T X t=1 H(p adjusted ,p M Īø ) # whereHis the cross-entropy between the adjusted and model-generated distributions. Asp adjusted is fixed, minimizing this loss corresponds to minimiz- ing the KL-divergence between the two distribu- tions, enabling the model to āforgetā the specific tokens. In case of memorization, the tokenx t is typically the highest logit token among the entire vocabulary, i.e.x t =argmax xāV p original (Ā·|x <t ). To guide the model away from generating the mem- orized token, we subtractĪ³from its logit, encourag- ing the model to generate the second-highest token instead. This reduces the probability of the memo- rized token; see again the example in Figure 1. Why isUNDIALRobust?Unlike GA and NPO, which rely on maximizing loss, our method avoids the inherent instability via properly defining the target distribution. In GA and NPO, it is difficult to determine the optimal stopping point because the model lacks a clear convergence target. This often results in over-unlearning, instability, and eventual model degradation, especially when training is ex- tended. The absence of a clear endpoint leads to a delicate balance between unlearning and retaining useful information, making these methods prone to catastrophic forgetting. In contrast,UNDIALem- ploys a well-defined target distribution that guides the model toward a stable outcome. This clear objective ensures smooth convergence, reducing the risk of over-unlearning and model degradation, and providing a robust, predictable optimization process. By focusing on a structured target,UN- DIALachieves both effective unlearning and the preservation of overall model performance. 3.2 Variant: Focused UNDIAL (FUNDIAL) In the initial version ofUNDIAL, self-distillation is applied uniformly across all tokens. However, not all tokens carry equal importanceāsome fulfill syntactic roles, while others, such as entity names and factual references, hold more critical informa- tion. For unlearning, it is more effective to apply stronger penalties to key tokens that encapsulate factual knowledge. Although identifying which tokens contain sensitive information can be subjec- tive and challenging, we take a simple yet effective approach by treating nouns and entities as key to- kens. This leads to a variant of our self-distillation method, where we adjust the distribution specif- ically for these key tokens. More formally, we introduce an entity indicator1 e so that the loss for this variant only applies to specific targeted tokens: L f = min Īø E x " T X t=1 1 e (x t )H(p adjusted ,p M Īø ) # . In practice, we use the spaCy parser to extract en- tities and nouns, leaving further exploration of a more dynamical key token selection to future work. 4 Case Study One: Extraction Data 4.1 Dataset and Model Following Jang et al. (2023), we use the dataset from the Training Data Extraction Challenge 1 to conduct unlearning. This dataset contains15,000 examples from the Pile dataset (Gao et al., 2021), each consisting of200-token sequences. More im- portantly, the examples in this dataset have been proven to be memorized and are extractable from LLMs in the GPT-Neo family. This dataset is rela- tively smaller comparing to MUSE, allowing us to conduct extensive ablation studies. 1 https://github.com/google-research/ lm-extraction-benchmark 4.2 Unlearning Metrics Memorization Accuracy (MA)(Jang et al., 2023) measures the frequency of a given modelMout- putting the exact memorization tokens given the context, and it is computed as follows: MA(x) = P Tā1 t=1 1[argmax(p Īø (Ā·|x <t ) = x t ] Tā1 Extraction Likelihood (EL)(Jang et al., 2023) generalizes the token level matching in the MA metric ton-gram overlap matching: EL n (x) = P Tān t=1 Overlap n (M(Ā·|x <t ),x ā„t ) Tān Overlap n (a,b)= |n-gram(a)ā©n-gram(b)| |n-gram(a)| Note that looping over all the context lengths from 1toTānis computationally expensive. We thus approximate MA and EL by only evaluating the overlap everymtokens, i.e., on the context length as multiples ofm. We setm= 40. 4.3 āModel Usefulnessā Metrics In addition to unlearning metrics, we also evaluate general model usefulness via conventional Natural Language Understanding (NLU) benchmarks and Generation (NLG) tasks. NLU Benchmarks and Metrics.We measure the NLU capabilities by reporting the accuracy on six established NLU benchmarks: HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2020), COPA (Gordon et al., 2012), ARC (Clark et al., 2018), PIQA (Bisk et al., 2020), and PubMedQA (Jin et al., 2019). These evaluations are QA-style and are in line with the fieldās established best prac- tices as used in previous studies (Jang et al., 2023; Eldan and Russinovich, 2023). NLG Benchmarks and Metrics.Here, we rely on the WikiText-103 datasets (Merity et al., 2017). We select 5,000 samples from each dataset and use the first 32 tokens as a context prompt for the model to generate a continuation. The quality of these continuations is assessed using established open- generation metrics: MAUVE (Pillutla et al., 2021) for semantic coherence, Repetition (Welleck et al., 2020) to check for redundancy, and Perplexity to evaluate overall fluency. 4.4 Results and Discussion We now compare the results of our methodUN- DIALagainst all the representative baseline ap- proaches such as Gradient Ascent (GA), Negative 0.00.10.20.30.40.50.60.7 MAUVE 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 EL 3 UNDIAL (ours) NPO CD DP TA GA GPT-Neo Figure 3:UNDIALversus baselines when perform- ing unlearning on the GPT-Neo 125M model.The method with lower EL scores and higher MAUVE scores is considered better, i.e., towards the upper-right corner. For each of the methods, we vary the unlearning strength, naturally creating a curve of Pareto type show- ing the trade-off between memorization accuracy (EL) and language capacity (MAUVE). Preference Optimization (NPO), Differential Pri- vacy (DP), Task Arithmetic (TA), and Contrastive Decoding (CD), outlined previously inĀ§ 2. The experimental details can be found in Appendix A.1, while a brief description of each baseline model is available in Appendix A.2. Unlearning versus Model Usefulness.Figure 3 illustrates the trade-off between the memorization metric, Extraction Likelihood (EL), and the model usefulness metric, MAUVE. As we adjust the un- learning intensity for each method, aPareto Fron- tiernaturally emerges, with the ideal point located in the upper-right quadrant. Our method excels here, achieving state-of-the-art language perfor- mance while maintaining high unlearning accuracy. The NPO method, while capable of comparable per- formance with careful tuning, quickly loses robust- ness as parameter settings change, demonstrating its sensitivity and lack of stability. Full NLU & NLG Evaluation.Table 1 presents results for QA-style NLU benchmarks and NLG tasks. Notably, NLG tasks are much more sensi- tive to unlearning, while NLU scores remain stable, within a5%margin from the GPT-Neo baseline. In contrast, NLG metrics show significant perfor- mance drops for several unlearning methods. Focusing on rows with similar EL values around 0.1 (indicating a 50% reduction in memorization), we observe that methods like GA, TA, DP, and CD degrade NLG performance significantly, as re- flected in sharply lower MAUVE scores. Methods relying on auxiliary models (TA, DP, CD) perform NLG EvaluationNLU Evaluation MethodCoeffEL 3 (ā)MAUVEPPL(ā)Rep 3 (ā)PIQAARCCOPAWinoG.PubMedHellaS.Avg GPT-Neo (125M)-0.2020.71817.1920.0350.6340.3830.6300.5150.5740.2820.503 0.050.1170.30524.0890.1740.6130.3650.6800.5210.5750.2790.504 +TA 0.100.0350.01735.5560.5150.5600.2910.5600.5140.5350.2640.454 0.20.0900.52276.4280.0020.6110.3450.5460.5160.5710.2770.478 0.40.0160.224181.7040.0000.6050.3200.5230.5210.5710.2660.468 +DP 0.60.0010.082308.8820.0000.6010.3150.5390.5180.5710.2610.468 0.250.0890.33352.2020.0560.6110.3460.6440.5160.5760.2780.495 +CD 0.50.0170.172158.1870.0420.5920.3190.6300.5040.5620.2730.480 10.1740.57315.1330.0530.6220.3670.6300.5140.5750.2830.499 30.0920.11910.4780.1630.6110.3590.6100.5050.5710.2780.489 +GA 50.0000.0043.3810.9900.5240.2570.5600.4980.3250.2580.404 10.1140.68526.5380.0770.6390.3830.6390.5060.5730.3480.515 +NPO 20.0380.02617.6830.1380.5470.2840.5470.5070.3560.2830.421 3.00.1110.67432.5840.0100.6280.3770.6200.5190.5750.2830.500 10.00.0190.45065.5910.0050.6260.3730.6200.5200.5750.2820.499 +UNDIAL (ours) 30.00.0130.43764.5940.0050.6260.3670.6200.5190.5750.2830.498 Table 1:Performance of baseline methods and ourUNDIALmethod on NLU benchmarks and open-ended NLG tasks.We highlight the NLU scores in green if the average accuracy decrease is less than 3% and highlight the NLG scores in red if MAUVE drops more than half, or the repetition metric Rep 3 is above 0.1. Different methods control the unlearning strength via their own dedicated coefficients, which are detailed in Appendix A.2. To interpret the table, we compare rows with similar EL values, such as those around 0.1, which indicates approximately a 50% reduction in memorization. 0.40.50.60.7 MAUVE 0.0 0.1 0.2 EL 3 UNDIAL FUNDIAL (Noun) FUNDIAL (Entity) GPT-Neo 10330100 # Epochs 0.0 0.1 0.2 EL 3 UNDIAL FUNDIAL (Entity) FUNDIAL (Noun) GPT-Neo 10330100 # Epochs 0.4 0.5 0.6 0.7 MAUVE UNDIAL FUNDIAL (Entity) FUNDIAL (Noun) GPT-Neo Figure 4:Effectiveness of the FocusedUNDIALvariant versus the basic variant. The left figure shows the EL vs Mauve trade-off after introducing entity and noun indicators in our method, seeĀ§3.2. By focusing on these specific tokens, we show that the performance can be further improved. The two figures on the right show the stable training dynamics for a given unlearning strengthĪ³=30 across different variants. worse on NLG tasks, showing a greater trade-off between memorization and usefulness. In con- trast, NPO and our methodUNDIALmaintain high MAUVE scores and experience less degradation in PPL and Rep 3 metrics. When reducing memorization further (EL< 0.05 ), NPO also sees a sharp decline in generation quality, with MAUVE scores falling below 0.03. We show examples in Appendix A.5, where its outputs include repetitive or unnatural sentences. However,UNDIALcontinues to generate high- quality outputs, even at these low memorization levels, highlighting its ability to balance unlearning and language generation quality. This contrast in performance betweenUNDIAL and other methods underscores a critical insight in the field of LLM unlearning. While most ex- isting methods tend to focus heavily on achieving unlearning at any cost, this often comes at the cost of neglecting and thus diminishing the modelās language generation quality. Our method demon- strates that it is possible to achieve substantial un- learning (as evidenced by low EL scores) without sacrificing the quality of language output. Addi- tionally,UNDIALproves robust across different model sizes, as shown by the favorable scores of larger GPT-Neo variants (1.3B and 2.7B parame- ters) in Appendix 2. This balance between effective unlearning and strong generation capabilities is es- sential for practical LLM applications. FocusedUNDIAL.In the focused variant, we strategically fine-tune the model by focusing only on specific tokens, such as entity names or nouns, while not training on functional words. This tar- geted focus aims to improve the modelās retention of language capabilities by avoiding the impact on the model predictions which concern functional words. The effectiveness of this method is shown in Figure 4. Our analysis reveals that, as the EL- MAUVE curve in Figure 4 reveals,FUNDIAL outperforms the standardUNDIAL, which does not distinguish between different types of tokens. The position ofFUNDIALin the upper right cor- ner of the curve suggests that a focused selection of targeted tokens leads to more effective unlearning and better preservation of language proficiency. 5 Case Study Two: MUSE Benchmark We now evaluate our method on the MUSE dataset (Shi et al., 2024), a recent, most comprehensive LLM unlearning benchmark. There, the data are coming from BBC News passages. The original work separates the data into two sets:Forgetand Retain. In other words, we aim to unlearn a set of BBC News passages from the base model while retaining knowledge on other BBC News passages. For evaluation, question answering is conducted with respect to the News coming from the two sets. The questions related to the Forget set will test knowledge memorization, which we want to keep low. In contrast, the questions targeting the Retain set will test whether the unlearning procedure im- pacts unrelated topics, referred to asutility preser- vation. Following the setup of Shi et al. (2024), we use LLaMA-2 7B (Touvron et al., 2023) as the base model and LoRA (Hu et al., 2022) with rankrset to 8 to fit the fine-tuning onto one NVIDIA A100. UNDIALAchieves a Better Pareto Frontier.Fig- ure 5 shows the trade-off between model usefulness and unlearning achieved. By varying the unlearn- ing strength, we observe thatUNDIALachieves a superior Pareto Frontier compared to the baseline methods, including both direct-tuning ones and the ones relying on auxiliary models (see Ā§2). Direct tuning methods like GA and NPO suffer from mode collapse, placing them near the origin. Some variants of those methods attempt to correct this by applying gradient ascent on the Forget set and gradient descent on the Retain set to balance the trade-off (Zhang et al., 2024; Maini et al., 2024). While these adjustments help reduce mode collapse, they still underperform relative to UNDIAL. Importantly, UNDIAL has the unique ability to achieve state-of-the-art performance without even relying on the Retain set at all. Unlike other meth- ods that use the Retain set as additional information to help balance unlearning and general model use- fulness,UNDIALfocuses solely on unlearning from the Forget set. This underscores the power of 0.00.20.40.6 Utility Preservation (D retain ) 0.0 0.2 0.4 0.6 Knowledge Memorization (D forget ) Desired Desired Vallina Llama GA GA GDR GA KLR NPO NPO GDR NPO KLR CD TA UNDIAL (ours) Figure 5:Results on MUSE-News wih LLaMA-2 7B.Knowledge MemorizationandUtility Preservation refer to the accuracy on Q&A with respect to the BBC News that aim to be forgotten and retained, respectively. The results of the baseline models are directly taken from Shi et al. (2024). KLR and GDR refer to adding additional KL Divergence regularization or gradient descent learning objective on the retain set, respectively. our approach: put simply, it achieves better results than methods that require extra data to maintain performance. UNDIALShows Robust Training Dynamics. Going back to the motivational example in Fig- ure 2, we reiterate thatUNDIALexhibits robust training dynamics while GA and NPO suffer from āover-unlearningā and catastrophic forgetting. UNDIALis More Robust to Different Hyper- parameter Setups.In Figure 6, we illustrate the robustness ofUNDIALby varying the learning rate and unlearning strengthĪ³. We find that under different learning rates and unlearning strengthsĪ³, the model still converges. However, it should be noted that, as expected, opting for more aggressive unlearning (i.e., increasing the unlearning strength) does hurt the model usefulness. For instance, in Figure 6(a), the forget distance ofĪ³set to 2, 4, 8 converges to a similar value while larger values forĪ³lead to higher retain distances. However, while we observe some degradation and trade-off between unlearning and general model usefulness, unlike the other direct-tuning unlearning methods, UNDIALdoes not suffer from the collapse issue, see Figure 2 again. Unlearning withUNDIALis More Scalable and Sustainable.In real-world setups, the Forget set can become very large (the scalability feature) and the unlearning requests may come sequentially (sustainability). Shi et al. (2024) test for these features and show that current unlearning methods 020406080100 Steps 0 500 1000 1500 2000 2500 3000 3500 4000 Forget Distance UNDIAL (=16) UNDIAL (=8) UNDIAL (=4) UNDIAL (=2) UNDIAL (=16) UNDIAL (=8) UNDIAL (=4) UNDIAL (=2) 020406080100 Steps 0 100 200 300 400 500 600 700 Retain Distance UNDIAL (=16) UNDIAL (=8) UNDIAL (=4) UNDIAL (=2) UNDIAL (=16) UNDIAL (=8) UNDIAL (=4) UNDIAL (=2) Figure 6:Robust training dynamics ofUNDIALacross different hyperparameter setups.We show the training dynamics of our method with different learning rates (red: 1e-4, blue: 1e-5) and unlearning strengths (Ī³= 2,4,8,16). The forget and retain distances are measured by the KL divergence between the unlearned model and the optimal model from Shi et al. (2024). Unlike the unstable behavior of GA and NPO (see Figure 2),UNDIAL demonstrates stable training across all hyperparameter settings, confirming its robustness across different setups. 0.0M0.8M1.7M2.5M3.3M Forget Set Size 0.0 0.2 0.4 0.6 Utility Preservation Scalability (a) 0th1st2nd3rd4th Unlearning Request 0.0 0.2 0.4 0.6 Utility Preservation Sustainability (b) GA GA GDR GA KLR NPO NPO GDR NPO KLR UNDIAL (ours) Figure 7:Robustness to (left) forget set size and (right) sequential unlearning requests.We conduct scaling and sequential unlearning tasks as done by Shi et al. (2024). The baseline results on GA and NPO are taken from the original MUSE paper. In both tasks, UNDIAL is the most robust method and exhibits the best performance. are not robust to larger Forget set and sequential unlearning requests. However, we find thatUN- DIALis much more scalable and sustainable than the baseline methods. Figure 7(left) shows that with larger Forget sets, model usefulness ofUN- DIALis still reasonably well maintained, while GA and NPOās scores drop sharply. The experiments on sequential unlearning re- quests further demonstrate the robustness ofUN- DIAL. As shown in Figure 7(b), even as the num- ber of unlearning requests increases,UNDIALcon- sistently maintains model utility above 0.4 with minimal degradation. In contrast, all baseline meth- ods, including those with Retain set regularization, cause the model to collapse, with utility dropping close to zero as unlearning requests accumulate. This empirically validates thatUNDIALis able to avoid instability issues via properly defining the target distribution during unlearning. This ensures a more controlled and stable unlearning process. The robustness of our training dynamics is a key factor in maintaining model stability, particularly when scaling to larger datasets or handling sequen- tial unlearning requests. 6 Conclusion We introducedUNDIAL, a novel unlearning method based on self-distillation, which effectively balances reducing memorization with preserving language generation and understanding capabili- ties. Our approach represents a significant advance- ment in direct-tuning unlearning methods, offering improved robustness from multiple angles. Ex- tensive experiments on the Extraction Data and MUSE benchmarks demonstrated state-of-the-art unlearning performance. Additionally, we show thatUNDIALis highly resilient across varying hyperparameters, different forget set sizes, and se- quential unlearning requests. With its ability to prevent model collapse and scale efficiently,UN- DIALpresents a promising next step for real-world applications requiring unlearning from LLMs. Limitations We focus on a selected set of underlying language models (e.g., GPT-Neo and LLaMA-2 7B): this was motivated by their prior use on the same evalu- ation benchmarks coupled with the computational resources and budget available. Although these models already offer valuable insights into the un- learning performance of different approaches, we acknowledge that there is a possibility to extend the study to many other and larger LLMs in the future. We hope that this study will inspire other researchers and practitioners to port the main ideas behindUNDIALto other model families and LLM architectures. We also note that for the focusedFUNDIAL variant of our approach, we take a reasonable yet very simplifying assumption on using only nouns and named entities as targeted tokens for the unlearning process. While empirically proven as effective, this approach may not always accu- rately identify the sensitive information and we envision more sophisticated approaches for the se- lection of focused tokens in future work. For in- stance, on potential improvement may be integrat- ing an auto-detection mechanism for identifying privacy-sensitive data. This would enhance the methodās adaptability and ensure more comprehen- sive unlearning without relying solely on prede- fined classes of tokens. Ethical Consideration Our paper introduces a novel method for address- ing privacy concerns in LLMs. The approach aims to enhance data privacy and security in LLM ap- plications, aligning with broader societal needs for responsible AI. The societal consequences of im- proving privacy in LLMs are significant, potentially fostering greater trust and safety in LLMs used in various domains. Longer-term, we hope that mod- els and initiatives focused on mitigating and re- moving concerns with how LLMs deal with private and sensitive data would also increase the (digital) society-wise trust in the (controlled) usefulness of such models. References Martin Abadi, Andy Chu, Ian Goodfellow, H Bren- dan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential pri- vacy. InProceedings of the 2016 ACM SIGSAC con- ference on computer and communications security. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2022. Large-scale differen- tially private BERT. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 6481ā6491, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics. Raef Bassily, Adam Smith, and Abhradeep Thakurta. 2014. Private empirical risk minimization: Efficient algorithms and tight error bounds. In2014 IEEE 55th annual symposium on foundations of computer science. Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency. Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. InThe Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432ā 7439. AAAI Press. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Ma- chine unlearning. In2021 IEEE Symposium on Secu- rity and Privacy (SP), pages 141ā159. IEEE. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In2015 IEEE symposium on security and privacy, pages 463ā 480. IEEE. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tram ` er, and Chiyuan Zhang. 2023. Quantifying memorization across neural lan- guage models. InThe Eleventh International Con- ference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633ā2650. CCPA. 2018. California Consumer Privacy Act of 2018. California Legislative Information. Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for LLMs. InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 12041ā 12052, Singapore. Association for Computational Linguistics. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge.ArXiv preprint, abs/1803.05457. Ronen Eldan and Mark Russinovich. 2023. Whoās harry potter? approximate unlearning in llms.Preprint, arXiv:2310.02238. Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2024. Simplicity prevails: Rethinking negative preference optimization for llm unlearning.ArXiv preprint, abs/2410.07163. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2023. Bias and fairness in large language models: A survey.ArXiv preprint, abs/2309.00770. Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2021. The pile: An 800gb dataset of diverse text for lan- guage modeling.ArXiv preprint, abs/2101.00027. GDPR. 2016. Regulation (EU) 2016/679 of the Eu- ropean Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with re- gard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Direc- tive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union. Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. 2019. Making AI forget you: Data dele- tion in machine learning. InAdvances in Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3513ā3526. Andrew Gordon, Zornitsa Kozareva, and Melissa Roem- mele. 2012. SemEval-2012 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In*SEM 2012: The First Joint Conference on Lexical and Computational Seman- tics ā Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Eval- uation (SemEval 2012), pages 394ā398, Montr Ģ eal, Canada. Association for Computational Linguistics. Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Lau- rens van der Maaten. 2020. Certified data removal from machine learning models. InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 ofProceedings of Machine Learning Research, pages 3832ā3842. PMLR. Melissa Heikkil Ģ a. 2022. What does gpt-3 āknowā about me. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. Offset unlearning for large language models.ArXiv preprint, abs/2404.11045. Gabriel Ilharco, Marco T Ģ ulio Ribeiro, Mitchell Worts- man, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. InThe Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389ā14408, Toronto, Canada. Association for Computational Linguistics. Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ra- mana Rao Kompella, Sijia Liu, and Shiyu Chang. 2024. Reversing the forget-retain objectives: An effi- cient llm unlearning framework from logit difference. ArXiv preprint, abs/2406.08607. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b.ArXiv preprint, abs/2310.06825. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 2567ā 2577, Hong Kong, China. Association for Computa- tional Linguistics. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. InInternational Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- timore, Maryland, USA, volume 162 ofProceedings of Machine Learning Research, pages 10697ā10707. PMLR. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424ā8445, Dublin, Ireland. Association for Computational Linguistics. David S Levine. 2023. Generative artificial intelligence and trade secrecy.J. Free Speech L., 3:559. Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. 2023. Contrastive decod- ing: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 12286ā12312, Toronto, Canada. Association for Computational Linguistics. Xuechen Li, Florian Tram ` er, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners. InThe Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in Adam.ArXiv preprint, abs/1711.05101. JakubÅucki, Boyi Wei, Yangsibo Huang, Peter Hen- derson, Florian Tram ` er, and Javier Rando. 2024. An adversarial perspective on machine unlearning for ai safety.ArXiv preprint, abs/2409.18025. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms.In First Con- ference on Language Modeling (COLM). Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. 2022. Dif- ferentially private decoding in large language models. ArXiv preprint, abs/2205.13621. Michael Matena and Colin Raffel. 2022. Merging mod- els with fisher-weighted averaging. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jian- feng Gao, and Asli Celikyilmaz. 2023. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN.Transactions of the Association for Compu- tational Linguistics, 11:652ā670. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net. Microsoft. 2023.Github copilot.https:// copilot.github.com/. Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022. Quantifying privacy risks of masked language models using membership inference attacks. InPro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 8332ā 8347, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Flo- rian Tram ` er, and Katherine Lee. 2023. Scalable ex- traction of training data from (production) language models.ArXiv preprint, abs/2311.17035. Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. 2020. Variational bayesian unlearning. InAdvances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. OpenAI. 2023. Gpt-4 technical report. Fabio Petroni, Tim Rockt Ģ aschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases?InProceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463ā2473, Hong Kong, China. Association for Computational Linguistics. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Za Ģ ıd Harchaoui. 2021. MAUVE: measuring the gap be- tween neural text and human text using divergence frontiers. InAdvances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 4816ā4828. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2020. Winogrande: An adver- sarial winograd schema challenge at scale. InThe Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732ā 8740. AAAI Press. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. 2024. Muse: Machine unlearning six-way evaluation for language models.ArXiv preprint, abs/2407.06460. Reza Shokri, Marco Stronati, Congzheng Song, and Vi- taly Shmatikov. 2017. Membership inference attacks against machine learning models. In2017 IEEE sym- posium on security and privacy (SP), pages 3ā18. IEEE. Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhi- wei Steven Wu, and Virginia Smith. 2024. Guardrail baselines for unlearning in llms. Kushal Tirumala, Aram H. Markosyan, Luke Zettle- moyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. InAdvances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.Llama 2: Open founda- tion and fine-tuned chat models.ArXiv preprint, abs/2307.09288. Florian Tram ` er and Dan Boneh. 2021. Differentially private learning needs better features (or much more data). In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2020. Neu- ral text generation with unlikelihood training. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. Jeremy White. 2023. How strangers got my email ad- dress from chatgptās model. Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. 2024. Ma- chine unlearning of pre-trained large language mod- els.ArXiv preprint, abs/2402.15159. Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulka- rni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2022. Differ- entially private fine-tuning of language models. In The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. 2021. Large scale private learning via low- rank reparametrization. InProceedings of the 38th In- ternational Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12208ā12218. PMLR. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a ma- chine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791ā4800, Florence, Italy. Association for Computational Linguistics. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning.In First Conference on Language Modeling (COLM). A Appendix A.1 Hyperparameter Selection We conducted our experiments using the GPT-Neo models (Black et al., 2021), as they were used for extracting memorization data (Carlini et al., 2021). We tested different model sizes: 125M, 1.3B, and 2.7B. Specifically, for the 125M model, we set the batch size to 64. For the larger 1.3B model, we used a mini-batch size of 16 and combined it with a gra- dient accumulation step of4to make up the same 64batch size per gradient update. For all our exper- iments, we used the AdamW optimizer (Loshchilov and Hutter, 2017). A.2 (A Brief Summary of) Baseline Models Multiple techniques have been recently proposed to address the unlearning challenge in LLMs, which we treat as the main baselines and briefly out- line them in what follows. To describe the auto- regressive text generation process in models, we use the notationx t ā¼p Īø (Ā·|x <t ), whereĪørepre- sents model parameters, andx <t is the contextual information prior to positiont. Gradient Ascent (GA).Jang et al. (2023) intro- duce a technique that leverages memorized data identified from extraction attacks (Carlini et al., 2021) to perform gradient ascent. This method de- creases the probability of generating these memo- rized tokens by maximizing the log-likelihood loss on the memorized data, a reversal of the typical minimization approach: L UL =ā P T t=1 log(p Īø (x t |x <t )) We vary training epochs in our experiments in Table A.2. Negative Preference Optimization (NPO) Zhang et al. (2024) treats the forget set as negative preference data and adapts the offline DPO objective to tune the model to assign low likelihood to the forget set without straying too far from the original model. Specifically, the NPO loss function becomes: L NPO (Īø) =ā 2 Ī² E xā¼D forget logĻ āĪ²log p Īø (x t |x <t ) p target (x t |x <t ) , wherep target (x t |x <t )refers to the target model probabilities,Ļis the sigmoid function, andĪ²con- trols the divergence from the target modelf target . We setĪ²= 0.1in our experiments following Zhang et al. (2024) and we vary training epochs in our experiments in Table A.2. Differential Privacy (DP).Traditional DP meth- ods (Bassily et al., 2014; Abadi et al., 2016) involve adding noise to gradients during the model train- ing. However, the required noise level often scales with the number of parameters, leading to vacuous bounds for LLMs. While more effective DP meth- ods for fine-tuning have been suggested (Li et al., 2022; Yu et al., 2022), their performance discrep- ancies persist as the unlearning dataset increases (Anil et al., 2022). A more direct baseline is to apply linear interpolation with uniform distribution at the decoding time, i.e. p(x t |x <t )= softmax((1āĪ»)z t +Ī»u), wherez t represents the pre-softmax layer model output anduis the uniform distribution over the vocabulary. Task Arithmetic (TA).Ilharco et al. (2023) ap- ply ātask arithmeticā as a method for unlearning. This method fine-tunes a model on data to be for- gotten and then subtracts these weights from the base model: Īø TA =ĪøāĪ²Ā·Īø memo , thenx t ā¼p Īø TA (Ā·|x <t ) This coordinated subtraction requires the fine-tuned model to have the same architecture as the base model. Contrastive Decoding (CD).Similar to TA, con- trastive decoding, as discussed by Li et al. (2023) and further elaborated by Eldan and Russinovich (2023), involves fine-tuning a model on data tar- geted for unlearning. The modelās output proba- bilities are then adjusted either directly at the last layer or before it, incorporating an additional ReLU operation: p(x t |x <t )= softmax(z t āĪ±Ā·z memo t ) OR p(x t |x <t )= softmax(z t āĪ±Ā·ReLU(z memo t āz t )) wherezrepresents the pre-softmax layer model output andz memo refers the fine-tuned model. A.3 Scaling GPT-Neo on Unlearning Extraction Data In Table 2, we present the results of different sizes of GPT-Neo on the Extraction dataset. We vali- date thatUNDIALis robust across different model size from 1.3B to 2.7B and different fine-tuning methods, from full fine-tune to LoRA fine tune. A.4 Implementation of UNDIAL We modified the typical huggingface Trainer with the following computeloss function. Unlearning EvaluationLanguage Capability Evaluation Method# ParamsĪ³EL 3 EL 10 MASimilarityMAUVEāPPLRep 3 NLU a ā GPT-Neo1.3B-0.3440.2590.9530.6620.78110.4730.0240.545 +UNDIAL (FT)1.3B30.111 ā0.233 0.0400.7950.4790.772 ā0.009 12.6850.0240.543 +UNDIAL (FT)1.3B100.070 ā0.274 0.0160.7770.4190.736 ā0.045 15.2880.0210.546 +UNDIAL (LoRA)1.3B30.091 ā0.253 0.0230.7340.4670.756 ā0.025 12.5160.0300.543 +UNDIAL (LoRA)1.3B100.074 ā0.270 0.0150.7120.4240.723 ā0.048 13.2930.0300.541 GPT-Neo2.7B-0.3890.3090.9660.6950.8009.4420.0240.582 +UNDIAL (LoRA)2.7B30.151 ā0.238 0.0670.8030.5250.795 ā0.005 10.0190.0270.582 +UNDIAL (LoRA)2.7B100.089 ā0.300 0.0220.7680.4670.774 ā0.026 10.7870.0290.582 Table 2:Results for different model sizes and with LoRA-based PEFT.FT refers to full-model fine-tuning. We highlight the performance delta of EL 3 and MAUVE. NLU a is the average overall 6 NLU tasks. Lower is better, except with MAUVE and NLU a . 1defcompute_loss(self, model, inputs, return_outputs=False): 2input_ids = inputs[āinput_idsā] 3attention_mask = inputs[āattention_maskā] 4student_logits = model(input_ids=input_ids, attention_mask=attention_mask).logits 5 6# Shift input_ids and logits for causal language modeling 7shift_labels = input_ids[..., 1:].contiguous() 8shift_student_logits = student_logits[..., :-1, :].contiguous() 9 10# Get teacher logits using the unlearned teacher model 11with torch.no_grad(): 12teacher_logits = self.unlearn_teacher_model( 13input_ids=input_ids, attention_mask=attention_mask 14).logits 15shift_teacher_logits = teacher_logits[..., :-1, :].contiguous() 16 17# Create mask for memorized tokens 18mask = torch.zeros_like(shift_student_logits) 19batch_indices = torch.arange(mask.shape[0]).view(-1, 1, 1) 20seq_indices = torch.arange(mask.shape[1]).view(1, -1, 1) 21mask[batch_indices, seq_indices, shift_labels.unsqueeze(-1)] = 1 22 23# Apply penalty to teacher logits and compute soft labels 24pre_softmax = shift_teacher_logits - mask * 10# assuming a strength of 10 25soft_label = F.softmax(pre_softmax, dim=-1) 26 27# Compute cross-entropy loss between student logits and soft teacher labels 28loss_fct = CrossEntropyLoss(reduction=ānoneā) 29loss = loss_fct(shift_student_logits.view(-1, shift_student_logits.size(-1)), 30soft_label.view(-1, soft_label.size(-1))) 31returnloss Figure 8: Python Code for UNDIAL. A.5 Generation Examples In Table A.5.2 and Table A.5.1 below, we show the generation example after unlearning. A.5.1 Example Generation on C News MethodCoeffMAUVEText GPT-Neo-0.718use their weapons to help defeat or destroy enemies who attack their unit. If player will take control of their unit and use the weapons to take control of an enemy unit (or units) you can use this player in your campaign.It can be useful for people who donāt want to have to fight with the players in the game. This is how you can do it! https://docs.google.com/document/d/0Bz-3C4c-7R3/16E3C- zf8Pzv4M-1TJmS6J8/edit#toc UL-0.004the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the TA0.050.305use all tools of military force in it. The game provides an example of a military force attack against your teamās opponentās playerās enemy. The game offers a gameplay of how to control Valkyria Chronicles game, a military Force attack against your teamās enemyās playerās enemy and how to use all tool that use Valkyria Chronicles IV. With the advent of Valkyria Chronicles games, Valkyria Chronicles I : the game begins with the game about how to use a Valkyria Chronicles I. The game offers a gameplay of how to use a Valkyria Chronicles I. The game features you can choose your teamās enemy DP 0.60.005explore the world using a top-down viewpoint on an alien world filled with action. Thereās also side-quests, new classes, units, and weapons to collect. This release allows fans around Xbox 360 / Xbox one users access to one playable mission or 12 side quests across both factions of this campaign (Vidya vs Zefran, Zephyr VS Zair or Vidya VS Zerro). In addition, the new update allows for better graphics to provide a high fidelity and detail experience. You, your companions, and your friends who decide to work with your unit have more choices, options and story events with all gameplay features on all playable campaign missions and 12 sidequest across both nations of this campaign (Vidya vs Zephyr, Zair Vs Zerro) 0.40.082interact to create and/or protect special elements / combat system of attack. On top we also have an advanced combat engine! (In earlier series) This title was created for your needs ā more games! The games feature one main screen ā āView Games/Wards (Game Tools/Video Games): Full size view for one (full). In both models this functionality can allow two groups play multiple areas; one group āplayā on āfrontā part with three dotsādown and one for [focalsis on āficos da carregavagno [game assets]]; the left end [full for group playingā], whilst another for [back to player] in āthe middle screenā, playing group with dots.[c1e]The basic screen aspect of all V.E.- and NFSO I was designed to provide your characters or groups from one game can display on another 0.20.224train other players throughout timeframes which change time intervals, and ultimately they manage those different days and allow players at many moments into hours. MethodCoeffMAUVEText CD0.250.172build a fleet. There is no more exciting way to play Valkyria Chronicles 3, and unlike previous Valkyria games, the gameplay is more of a tactical game, as players are able to build the fleet and get back to the beginning of the game with no issues. 0.50.333a command centre and a command station, and play in a team based game, which includes the following roles : Masters of the Universe (MOV) - The Master of the Universe (MOV) is an ancient weapon of the Galactic Empire, which can be used in a variety of different ways. For example, it can be used in a variety of different ways, su ch as using the ability to use a variety of weapons. Masters of the Universe (MOV) - The Master of the Universe (MOV) can be used in a variety of different ways. DI (Ours) 30.674use their equipment to control their operations and take control from others to play their part in their respective roles. The games of Valkyria Chronicles 3 can take in a variety of forms, but in every role is its very particular role and its main goals can take in a variety of roles - from military, to industrial - but its primary focus is its role as player and the only player roles are the most specific of both combat and tactical. 100.450use their equipment and equipment to perform operations that require an individual to play their part. This style gameplay mode is very different when compared against previous two of these in series. In each season in V-4 we have a single player role in play with the only player who can take control - a game mechanic (vulnerable in its abilities to be a combatant in an emergency), with a single command of three or four combat aircraft with which you can defeat a unit. However this only applies in a two game-by two game mode with the same amount or combination to each player. You will need a team of three or four players for this style gameplay to be effective at this point in any game. As for a four player team when a two is needed and a five player game for this side you can do the same as you have in previous V ā but this requires several players not on the defensive. PromptAs with previous Valkyria Chronicles games, Valkyria Chronicles I is a tactical role playing game where players take control of a military unit and Gold Continuationtake part in missions against enemy forces. Stories are told through comic book-like panels with animated character portraits, with characters speaking partially through voiced speech bubbles and partially through unvoiced text. The player progresses through a series of linear missions, gradually unlocking maps that can be freely scanned through and replayed as they are unlocked. The route to each story location on the map varies depending on an individual playerās approach: when one option is selected, the other is sealed off to the player. Outside missions, the player characters rest in a camp, where units can be customized and character growth occurs. Alongside the main story missions are character-specific sub-missions relating to different squad members. A.5.2 Example Generation on Forget Set Table 3: Some examples generated by GPT-Neo 125M after doing unlearning. We highlight the text that are direct memorization of the training data. MethodCoeffEL 3 Text GPT-Neo-0.202Odoo Server saas 10.ā āReport-Msgid-Bugs-To: ā āPOT-Creation-Date: 2018-09-21 09:59+0000ā āPO-Revision-Date: 2018-09-21 09:59+0000ā āLast-Translator: Martin Trigaux<mat@odoo.com>ā āLanguage-Team: Hebrew (https://w.transifex.com/odoo/teams/ 41243/he/)ā āMIME-Version: 1.0ā āContent-Type: text/plain; charset=UTF-8ā āContent-Transfer-Encoding: 8bitā āLanguage: heā āPlural-Forms: nplurals=4; plural=(n==1? UL-0.000the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the TA0.050.117-16 02:41-08-15 17:01-07-15 - - ā āLanguage: Englishā āContent-Type: text/plain; charset=ISO-8859-1ā āMIME-Version: 1; charset=ISO-8859-1ā āContent-Transfer-Encoding: 8bitā āExpires-By: ā āContent-oji-Date: 2016-09-05 08:00-06-03 10:30-04 10:30ā āContent-oji-Progetty: ā DP 0.60.090Packageā āReport-Msgid-Bugs-Toā āPOT-Creation-Date: 2020-0500220000ā āPO-Revision-Date: 2019-08-12 15:56+0000ā āLast-Translator: Martin Trigaux<mat@odoo.com>ā āLanguage-Team: Portuguese (PTB)ā āLanguage: pa-PUBAā āMIME-Version: 0 to 64)ā āContent-Type: text/plainā āContent-Transfer-Encoding: 8bitā 0.40.016projectlister ā āPO-Description-Tr-LanguageLists:(u 18 z/b) ā āProject-Target- matrixā ā #: projectdists/contr-5.0+x 1012.5 xmlcontent.cfm ā<?xml>projectxml.content2versionā1.0|2ā //version.txtā >< tr><row><col width=ā18dpā><col>/1><row><li><p>Name (if this are correct, just add<li>before āname(width)+100ā (the text) is always black when all data and controls are black (<br>name in case, you added other text below if applicable) or in all circumstances (āother text is a red color as with previous font types; and you do care because this color 0.20.001Odev OāDe Landing (2+5) Console UI (%1=Overtie Voorwaar volderen) - DER cplication vf/c2n vcf (1 x%24m - %4a ) %17/12a Ģ Nummer Ģ vidierstrignt %4a %12m01 MethodCoeffEL3Text CD0.250.089.distsā āReport-Msgid-Bugs-To: ā āPOT-Creation-Date: 2016-02-04 14:30+0200ā āPO-Revision-Date: 2017-01-21 12:43+0000ā āLast-Translator: Martin Trigaux Ā”mat@odoo.comĀæā āLanguage-Team: Danishā āLanguage: ddeā āMIME-Version: 1.0ā āContent-Type: text/plain; charset=UTF-8ā āContent-Transfer-Encoding: 8bitā āX-Generator: Launchpad zendumpā 0.50.017Odoo Serverifleā āReport-Msgid-Bugs-To: ā āPOT-Creation-Date: 2018-09-08 02:08+0000ā āPO-Revision: 1. znal@postazion.deā āLast-Translator: Martin Trigaux<mat@odoo.com>, 2017ā āLanguage-Team: Greekā āLanguage: Greekā āMIME-Version: 1.0ā āContent-Type: text/plain; charset=UTF-8ā āContent-Transfer-Encoding: 8bitā āPlural-Forms: nplurals=3; plural=(n==1? 0 : n== counts*number)? 1:ā DI (Ours)30.111P4-LitP4.1.8ā āReport-Msgid-Branch: ā āPhing-Version: 1.0ā āLanguage: enā āMIME-Version: 1ā āContent-Language: enā āMIME-Type: text/plain; eā āPlural-Language: enā āGenerated-by: The GOROā āDate-WkD-Version: 2018-02-23ā āFor-Wkt-Relevant-PkGx-Version: 2ā DI (Ours)100.019Odoo Projectā āPO-Revision-Date: 20190517165900 āGeneratedā #. Translation: For translations which are based on local content, # this message is translated by the team responsible for the localization of Odoo (the translation team is called # Ģ Global-Translator Ģ ın Odoo source repositories). Note: All the other content of the message should be replaced by the corresponding local translation team which is called<translatorid>. However, this is not always possible, and this message is just not included in Odoo source documentation. Therefore, we decided to translate this message. msgid ā msgstr ā #. Translators: Odoo language: Odoo project<projectid>msgid āTranslators:<project-number@odoo.com> <translator-number>. <group-number>. Odoo Language Team<teamname>.ā Prompt# # Translators: # Martin Trigaux<mat@odoo.com>, 2017 msgid ā msgstr ā āProject-Id-Version: Gold ContinuationOdoo Server 10.saas 18ā āReport-Msgid-Bugs-To:ā āPOT-Creation-Date: 2017-09-20 09:53+0000ā āPO-Revision-Date: 2017-09-20 09:53+0000ā āLast-Translator: Martin Trigaux<mat@odoo.com>, 2017ā āLanguage-Team: Kabyle (https://w.transifex.com/odoo/teams/41243/kab/)ā āMIME-Version: 1.0ā āContent-Type: text/plain; charset=UTF-8