Paper deep dive
BLUR: A Bi-Level Optimization Approach for LLM Unlearning
Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Sijia Liu, Mingyi Hong
Models: Llama-2-7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 6:35:57 PM
Summary
The paper introduces BLUR (Bi-Level UnleaRning), a novel optimization framework for LLM unlearning that addresses the limitations of traditional weighted-sum loss formulations. By modeling unlearning as a bi-level optimization problem—where the forget task is prioritized as the lower-level objective and the retain task as the upper-level objective—BLUR dynamically orthogonalizes gradients to ensure effective knowledge removal while preserving model utility. Theoretical analysis confirms convergence, and empirical results demonstrate superior performance across various benchmarks compared to state-of-the-art methods.
Entities (5)
Relation Signals (3)
BLUR → solves → LLM Unlearning
confidence 100% · we propose a novel algorithm, termed Bi-Level UnleaRning (BLUR), which solves the above bi-level unlearning problem.
BLUR → uses → Bi-level Optimization
confidence 95% · This hierarchical structure naturally leads to a bi-level optimization formulation... we propose a novel algorithm, termed Bi-Level UnleaRning (BLUR)
BLUR → outperforms → GradDiff
confidence 90% · our extensive experiments demonstrate that BLUR consistently outperforms all the state-of-the-art algorithms
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Enabling large language models (LLMs) to unlearn knowledge and capabilities acquired during training has proven vital for ensuring compliance with data regulations and promoting ethical practices in generative AI. Although there are growing interests in developing various unlearning algorithms, it remains unclear how to best formulate the unlearning problem. The most popular formulation uses a weighted sum of forget and retain loss, but it often leads to performance degradation due to the inherent trade-off between forget and retain losses. In this work, we argue that it is important to model the hierarchical structure of the unlearning problem, where the forget problem (which \textit{unlearns} certain knowledge and/or capabilities) takes priority over the retain problem (which preserves model utility). This hierarchical structure naturally leads to a bi-level optimization formulation where the lower-level objective focuses on minimizing the forget loss, while the upper-level objective aims to maintain the model's utility. Based on this new formulation, we propose a novel algorithm, termed Bi-Level UnleaRning (\texttt{BLUR}), which not only possesses strong theoretical guarantees but more importantly, delivers superior performance. In particular, our extensive experiments demonstrate that \texttt{BLUR} consistently outperforms all the state-of-the-art algorithms across various unlearning tasks, models, and metrics. Codes are available at this https URL.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
58,996 characters extracted from source content.
Expand or collapse full text
BLUR: A Bi-Level Optimization Approach for LLM Unlearning Hadi Reisizadeh 1,∗ , Jinghan Jia 2,∗ , Zhiqi Bu 6 , Bhanukiran Vinzamuri 3 , Anil Ramakrishna 6 , Kai-Wei Chang 3,4 , Volkan Cevher 3,5 , Sijia Liu 2 , Mingyi Hong 1,3 1 University of Minnesota, 2 Michigan State University, 3 Amazon AGI, 4 UCLA, 5 LIONS EPFL, 6 Meta hadir@umn.edu Abstract Enabling large language models (LLMs) to unlearn knowledge and capabilities acquired during training has proven vital for ensuring compliance with data regulations and promoting ethical practices in generative AI. Although there are growing interests in developing various unlearning algorithms, it remains unclear how to best formulate the unlearning problem. The most popular formulation uses a weighted sum of forget and retain loss, but it often leads to performance degradation due to the inherent trade-off between forget and retain losses. In this work, we argue that it is important to model the hierarchical structure of the unlearning problem, where the forget problem (which unlearns certain knowledge and/or capabilities) takes priority over the retain problem (which preserves model utility). This hierarchical structure naturally leads to a bi-level optimization formulation where the lower-level objective focuses on minimizing the forget loss, while the upper-level objective aims to maintain the model’s utility. Based on this new formulation, we propose a novel algorithm, termed Bi-Level UnleaRning (BLUR), which not only possesses strong theoretical guarantees but more importantly, delivers superior performance. In particular, our extensive experiments demonstrate thatBLUR consistently outperforms all the state-of-the-art algorithms across various unlearning tasks, models, and metrics. Codes are available at https://github.com/OptimAI-Lab/BLURLLMUnlearning. 1 Introduction Large language models (LLMs) have illustrated exceptional power in text generation that closely mimics human interactions [47]. However, these models are trained and fine-tuned on large datasets that are usually collected from the web. This raises ethical and privacy issues, such as generating biased [24,34], toxic, private, illegal responses [35,49,23,45], and potential guides on developing bioweapons and cyberattacks [1,25]. LLM unlearning has emerged as a useful technique to mitigate these concerns, which forget these toxic data influences from the pre-trained LLMs and ensure the unlearned models are safe for various applications while preserving the model’s overall utility after the unlearning phase. 1.1 Challenges and Our Contributions Unlearning in LLMs introduces unique challenges due to the massive size and complexity of their training datasets, as well as the risks of memorizing biases, sensitive information, and harmful content. Another challenge is to precisely define the unlearning targets, such as sample data points in the training set or knowledge concepts that must be forgotten during the unlearning phase, which usually leads to task-based solutions [19,18,9]. Also, reliable evaluation mechanisms for LLM unlearning is still lacking, and it is shown that sensitive information can be retrieved by reverse engineering techniques such as relearning [17, 31] and jailbreaking attacks [30, 43]. How to Balance Forget and Retain? One key algorithmic challenge that we attempt to address in this work is that during the unlearning process, how to best balance between the task of ‘unlearning’ knowledge/capabilities, and ‘retaining’ model utility. Indeed, it has been generally observed that removing ∗ Equal contribution. 1 arXiv:2506.08164v2 [cs.LG] 20 Oct 2025 undesired information can degrade the model’s utility, while insufficient forgetting may fail to achieve unlearning goals. Therefore, it is critical to ensure that these two tasks are carefully solved together to ensure that both are achieved eventually. However, in almost all existing works [26,52,33,9,54], the problem of LLM unlearning has been formulated as a weighted sum of the forget and retain losses, with a fixed weighting factor used to indicate the relative importance of the two tasks. Despite the simplicity of this formulation, it fails to fully capture the dynamic nature of unlearning, that is, the importance of the retain and forget loss often changes as the optimization goes, leading to model performance degradation. See Sec. 2.2 for more detailed discussion on this point. How to Model the Hierarchical Structure? Beyond algorithmic considerations, a more fundamental challenge lies in defining an effective formulation for the unlearning problem. Should we aim to forget and retain simultaneously? In many cases, the answer is no. Unlearning is typically necessitated in scenarios where the removal of certain sensitive information is critical. This may be driven by ethical and legal compliance requirements, such as adhering to privacy regulations (e.g., GDPR [16]), or by the need to address fairness concerns by mitigating biases in the model [2]. In such cases, failure to completely remove the identified information is unacceptable, making the forget task a priority. Once the forget task is completed successfully, the remaining capacity of the model if sufficient can then be leveraged to focus on the retain task [28]. This hierarchical prioritization ensures that sensitive information is effectively removed while still striving to preserve the model’s utility for its intended applications. Unfortunately, none of the existing unlearning works have considered this key aspect. 1.2 Our contributions This work proposes to approach the unlearning problem from a fresh perspective. Instead of treating unlearning as a binary process of simply forgetting specific information while retaining the rest, we argue that we should prioritize and structure these tasks hierarchically. Specifically, the forget task should take precedence to ensure that sensitive or harmful information is thoroughly removed before addressing the retain task. This perspective allows for a more principled approach to unlearning. Interestingly, the algorithm derived from the new perspective dynamically adjusts its emphasis on the forget and retain loss during the optimization process, addressing the previously mentioned ‘balancing’ challenges. More concretely, our contributions are listed below: 01020304050 KnowMem on r 0 10 20 30 40 50 60 KnowMem on f Original GA GradDiff NGDiff NPO SimNPO BLUR-NPO Ideal Figure 1: Trade-off between Knowledge memoriza- tion values on the forget set (vertical axis, the lower the better) and retain datasets (horizontal axis, the higher the better) using different unlearning meth- ods. Training is done using LLaMA2-7B model, evaluated using the MUSE-News dataset. We run GradDiff with various values of the regularization term λ, as defined in (1). (1). Observing that the aforementioned hierarchi- cal structure is critical for understanding the un- learning problem, we begin by formulating it as a bi-level optimization problem, where the lower- level problem focuses on identifying a set of solu- tions that minimize the forget loss, ensuring that sensitive or undesirable information is effectively removed. The upper-level problem then selects one of these solutions from the lower-level that minimizes the retrain loss, thereby preserving as much of the remaining knowledge as possible. This formulation introduces a principled way to balance these competing objectives, leading to a novel and flexible approach to unlearning. (2). We develop a novel algorithm named Bi-Level UnleaRning (BLUR), which solves the above bi- level unlearning problem. At a high level, this algorithm takes a gradient descent step of the for- get objective and then updates the retain loss only in the direction that is orthogonal to the computed forget gradient. Such an orthogonalization is achieved by carefully and dynamically updating the weights that balance the forget and retain loss when updating the LLM parameters. In addition, we show that the algorithm converges to certain desired solutions. (3). We conduct several experiments on widely used datasets, including MUSE [42] and WMDP [25]. 2 We demonstrate the effectiveness ofBLURacross diverse unlearning benchmarks and evaluation metrics, demonstrating thatBLURoutperforms a number of state-of-the-art LLM unlearning algorithms; see, e.g., Fig 1. In particular, our algorithmBLURon the MUSE-News dataset achieves19%and11.7%higher performance than the state-of-the-art baselines in unlearning efficiency and model utility, respectively, on the MUSE-News dataset. Surprisingly, by adapting andBLURusing the loss functions designed in RMU, the algorithm outperforms RMUs, achieving16.6%higher performance in unlearning efficiency while maintaining the same level of model utility. 1.3 Other Related Works Machine Unlearning for Non-LLMs. The concept of machine unlearning (MU) originated from data protection regulations such as the right to be forgotten [37]. The MU has emerged in various applications such as image classification [40,11], text-to-image generation [14,55], federated learning [27,29,4], graph neural networks [5, 50], and recommendation [39, 51]. LLM Unlearning. Retraining LLMs from scratch is often infeasible due to the amount of training datasets. Hence, removing undesirable information from the pre-trained model is critical for practical unlearning. Although solving the LLM unlearning problem with an initial pre-trained model appears easy, the challenges of choosing suitable losses, especially forget loss introduce new complexities in achieving the optimal balance between unlearning and utility. Some works [46,52,33] have utilized the gradient ascent (GA) approach on the prediction loss over the undesirable dataset (forget set). Even though this approach is intuitive, as it implies reversing gradient descent, the performance of gradient ascent-based approaches remains unsatisfactory, particularly in terms of model utility due to the unboundedness of gradient ascent loss. To address this issue, efficient forget losses are developed, such as preference optimization (PO) [36], negative preference optimization (NPO) [54], and simple negative preference optimization (SimNPO) [10]. During LLM preference alignment, PO replaces true information with random information for the forget set, while NPO treats the forget set as negative samples. SimNPO eliminates the dependency of the forget loss on the reference model and provides an improved version of NPO. Recently, another related work [3] studies LLM unlearning as a regularized multi-task optimization problem, where one task optimizes the forget loss objective and the other preserves model utility. A normalized gradient difference method, termed NGDiff, is then developed based on dynamic scalarization. All these approaches do not address the hierarchical structure of the unlearning problem. 2 LLM Unlearning as a Bi-level Optimization Problem 2.1 Preliminaries We start by defining some notations. LetD f be the forget dataset that contains the data whose influence on the model is to be removed, andD r as a retain dataset, which includes samples that help preserve the model’s utility. The LLM unlearning is typically modeled in the following manner [26, 52, 33, 9, 54] min θ E (x,y)∈D f ℓ f (y|x;θ) +λE (x,y)∈D r ℓ r (y|x;θ) ,(1) whereℓ f (y|x;θ),ℓ r (y|x;θ)represent the forget and retain prediction loss, respectively, computed using the model parameterθfor an inputxwith respect to the responsey. Here, the parameterλ ≥ 0is a regularization term used to balance forget and retain. The retain loss is typically cross-entropy loss, given by ℓ r (y | x;θ) =− log(π(y | x;θ)),(2) whereπ(y | x;θ)is the output probability distribution of the current modelθ. Commonly used forget losses are given below: • ℓ GA [33,46] represents the gradient descent technique on the negative prediction loss leading the updated model’s predictions to diverge from the pre-trined model’s. This loss function is defined as ℓ GA = log(π(y | x;θ)).(3) 3 • ℓ f = ℓ NPO,β for a given β ≥ 0 [54] treats the forget set as negative examples, given by ℓ NPO,β (y|x;θ) = 2 β log 1+ π(y|x;θ) π(y|x;θ 0 ) β ! ,(4) whereπ(y | x;θ 0 )represents the reference probability distribution of the pre-trained modelθ 0 .• ℓ f = ℓ SimNPO,β,α for givenβ,α ≥ 0[10] adopts a reference-free reward formulation that is normalized by sequence length, defined as ℓ SimNPO,β,α (y|x;θ) =− 2 β logσ − β |y| logπ(y|x;θ)−α ,(5) where|y|is the response length andα ≥ 0serves as the reward margin parameter. For simplicity, let f (θ) :=E (x,y)∈D f ℓ f (y|x;θ) , and r(θ) :=E (x,y)∈D r ℓ r (y|x;θ) . Solving the regularized optimization problem in(1)using the gradient descent technique, we obtain the update direction and the corresponding update scheme given by ˆu(θ) =∇f (θ) + λ∇r(θ),(6) ˆ θ(t + 1) = ˆ θ(t)− η ˆu( ˆ θ(t)).(7) Table 1: Summary of unlearning methods with their retain and forget losses. Unlearning MethodRetain LossForget Loss GA [33]N/A(3) GradDiff [26](2)(3) NPO [54](2)(4) SimNPO [10](2)(5) whereη > 0is the learning rate. Utilizing the retain loss and forget objective in the gradient direction defined in(6), we derive various unlearn- ing methods, summarized in Table 1. 2.2 Unlearning as Bi-level Optimization As mentioned in the introduction, one key chal- lenge in formulations(1)is that the weighted sum of two losses cannot properly prioritize one task (e.g. forget) over the other (e.g., retain). To illus- trate this point, we consider a set of simple numerical experiments where we measure how the update directionˆu(θ)aligns with both forget and retain gradient functions. To this end, let us define the normalized alignment as: A f (θ) := ⟨∇f (θ), ˆu(θ)⟩ ∥∇f (θ)∥ 2 , A r (θ) := ⟨∇r(θ), ˆu(θ)⟩ ∥∇r(θ)∥ 2 .(8) 02004006008001000 Step 4 3 2 1 0 1 2 Alignment A f -NPO A r -NPO A f -BLUR-NPO A r -BLUR-NPO Figure 2: Alignment values of forget and retain losses in(8)on MUSE-News using LLaMa2-7B model vs. training step. IfA f (θ)(resp.A r (θ)) is positive, it means that the update directionˆu(θ)will improve the forget lossf (θ)(resp. retain lossr(θ)). Fig 2 plots the change ofA f (θ)andA r (θ)across the training iterations of NPO withλ = 1. As Fig 2 shows, the descent directionˆu(θ)switches con- stantly the priorities between the forget and retain objectives (note that, in contrast, the proposed al- gorithmBLURalways prioritizes the forget loss; see subsequent discussions for details). This ex- ample indicates that the regularized formulation with a staticλfails to prioritize the loss functions properly and cannot adapt to the complexities of the data and the dynamics of unlearning. Addi- tional experiments withλ = 0.5, 1.5, 2, as given in Appendix C, Fig C.2, show similar switching behavior. As argued in the introduction, in many practical use cases, the removal of sensitive information, such as copyright-related data or personal information, is critical [16,2]. Therefore, it is useful to have a new unlearning formulation that consistently and 4 explicitly prioritizes one task (e.g., the forget task) while treating the other one (e.g., the retain task) as an auxiliary task. Towards this end, we exploit a classical optimization paradigm called bi-level optimization [32,13,12,20,44,53], which is used to model problems with hierarchical structure. In a typical bi-level optimization problem, the upper-level objective function (retain loss, in this case) is minimized over the solution set of a lower-level objective function (forget loss). Therefore, placing more emphasis on the lower-level problem. More precisely, consider the following problem formulation min θ∈Θ r(θ) s.t. Θ = arg min θ∈R d f (θ).(9) In the above formulation,Θis the optimal solution set of the lower-level problem. We note that(9)is a specific form of bi-level optimization, often referred to as a simple bi-level, because the lower-level optimization variable is exactly the same as that of the upper-level [38,8,7]. It is important to note that (9)can be viewed as a meta formulation, where different forget and retain losses can be used to replace the abstract losses r(·) and f (·), e.g., those mentioned in Table 1. Indeed, in our numerical experiments to be shown shortly, we have demonstrated that it is beneficial to use customized loss functions for certain tasks. To illustrate the difference between the bi-level formulation(9)and the weighted sum formulation (1), let us consider the following toy example. Example 1. Consider a specialization of (9) min θ∈Θ h(θ) := (θ−2) 2 , s.t. Θ = arg min θ∈R [w(θ) :=|θ−1|+|θ +1|]. Clearly,w(θ)is minimized over the setΘ = [−1, 1]. Thus, the problem is simplified tomin θ∈[−1,1] (θ−2) 2 whose optimal solution θ ⋆ = 1. Meanwhile, from (1), we can write a regularized optimization problem: min θ∈R h(θ)+λw(θ) = (θ−2) 2 +λ(|θ−1|+|θ+1|) .(10) Note that for an arbitrary choice ofλ, problems(10)and(9)are not equivalent. Whenλ = 0, the problem reduces tomin θ∈[−1,1] (θ− 2) 2 leading to the optimal solutionθ = 2, that is outside ofΘ. Asλ→∞, the regularization term dominates, forcing the optimal solution to be any value within Θ. Remark 2.1. Of course, if necessary, one can easily switch the order of the lower and upper-level problems, emphasizing more on preserving model utilities. However, we found that both conceptually and numerically, this is not a good modeling choice (at least from the datasets we have tested). Therefore, throughout this paper, we will not mention this case. In the LLM unlearning, we often deal with nonconvex objectives. When the primal feasibility condition for(9)cannot be satisfied exactly, we instead aim to converge to a stationary point where ∥∇f (θ)∥ 2 ≤ ε 0 for someε 0 ≥ 0. Further, we require an approximate stationarity condition of the Lagrangian function∥∇r(θ)+ζ∇f (θ)∥ 2 ≤ε 1 for someε 1 ≥ 0whereζis the Lagrange multiplier. Hence, we aim to find solutionθ that satisfies: ∥∇f (θ)∥ 2 ≤ ε 0 ,(11) ∥∇r(θ) + ζ∇f (θ)∥ 2 ≤ ε 1 .(12) 3 BLUR: Method and Analysis In this section, we first discuss the limitations of previously proposed algorithms for solving(12)-(11), then we present our scheme, termed Bi-Level UnleaRning (BLUR). Finally, we provide the theoretical guarantees for the proposed algorithm. The majority of the existing works [38,41,22,48] assume that either the upper-level or lower-level objectives is (strongly) convex, but this is certainly not true in the LLM unlearning setting where both problems are non-convex. Specifically, the scheme proposed in [38] assumes both the lower and upper-level objectives are strongly convex. Moreover, many of them require solving the lower-level problem to some accuracy before updating the upper-level problem, which can 5 incur a significant computational burden when the LLM is large [13,6]. Importantly, we demonstrate that the retain and forget gradients conflict over the unlearning steps, necessitating the design of the update direction in favor of the lower-level objective function. To this end, we examine the relation between the forget and retain loss gradients across the iterations of the algorithm in(7). Here, we consider the NPO method, a specific version of the algorithm in(7), where the forget and retain losses are defined in Table 1. 02004006008001000 Step 1.0 0.8 0.6 0.4 0.2 0.0 0.2 Cosine Similarity Figure 3: Cosine similarity of the gradient forget and retain losses using NPO on MUSE-News dataset and LLaMA2-7B, with λ = 1 and η = 10 −5 . We conduct an experiment using the NPO method on the MUSE-News dataset with a regulariza- tion term ofλ = 1and a learning rate ofη = 10 −5 . In Fig. 3, we plot the trajectory of cosine similarity between these two quantities, that is ⟨∇f (θ),∇r(θ)⟩ ∥∇f (θ)∥∇r(θ)∥ . It is clear that such a similarity measure remains mostly negative, implying that the retain gradient contains a destructive compo- nent with respect to the forget loss. More experi- ments withλ = 0.5, 1.5, 2, as given in Appendix C and Fig. C.3, further support this observation. We also run this experiment using the GradDiff method, defined as in Table 1, withλ = 0.5, 1, 1.5 where we observe the similar conflicting pattern in Fig. C.4. Thus, naively summing the forget and retain gradients could not provide the desired direction toward minimizing the forget loss (lower-level problem). To ensure convergence to a stationary point off, i.e.,∇f (θ) = 0, the update directionu(θ) should move in favor of the objective functionf. More precisely, the desired update directionu(θ) should satisfy ⟨∇f (θ),u(θ)⟩ = γ∥∇f (θ)∥ 2 , for someγ > 0. To fulfill this condition, we have to appropriately remove destructive components from the retain gradient. Further, whenever possible,u(θ)should also contain the non-destructive component of the retain gradient to be able to minimize the upper-level problem. We propose a novel update direction that satisfies these requirements, given below: u(θ) = γ∇f (θ)+∇r(θ)− ⟨∇f (θ),∇r(θ)⟩ ∥∇f (θ)∥ 2 ∇f (θ).(13) The update direction in(13)can be interpreted in relation to the Gram-Schmidt orthogonalization process. More precisely, the third term on the RHS of(13)represents the projection of∇r(θ)onto∇f (θ). The visualization of the update directions in (13) and (6) with their components is shown in Fig. 4. Figure 4: Visualization of the update direction in (6) and (7) with their components. With the above discussion about the update direction(13), the proposed algorithmBLURcan be simply 6 expressed as as: θ(t + 1) =θ(t)−η· u(θ(t)),(14) whereθ(0) =θ 0 is the pre-trained model;η > 0is the learning rate. We note thatBLURis a meta- algorithm that can take different forms depending on the specific choices offandr. Subsequently, we use BLUR–[·]to indicate the specific choices of loss functions. We denote BLUR–NPO by usingBLURwith the retain loss in(2)and the forget objective in(4). We plotA f (θ)andA r (θ)across the optimization steps of BLUR–NPO in Fig. 2. As we observe,BLURconsistently prioritizes the forget loss over the retain loss at each step. 3.1 Convergence Analysis Next, we present the theoretical analysis ofBLUR, demonstrating the convergence behavior of the forget gradient loss∇f (θ(t))and the descent directionu(θ(t)). Towards this end, we make the following assumptions. Assumption 3.1 (Function Assumptions). We assume the following properties on continuously differ- entiable functions f and r: (a)The gradient offisL f -Lipschitz, i.e., for anyx,y∈ R d we have∥∇f (x)−∇f (y)∥≤ L f ∥x−y∥. (b) The gradient of r is L r -Lipschitz. (c) There exists a constant C <∞ such that∥f (x)∥,∥r(x)∥,∥∇f (x)∥,∥∇r(x)∥≤ C. We have the following result: Theorem 3.2. Under Assumption 3.1, the model generated by using dynamics in (13) – (14) satisfies 1 T T−1 X t=0 ∥∇f (θ(t))∥ 2 ≤ 2C Tηγ + L f 2γ ηC 2 1 .(15) Further, the following holds: 1 T T−1 X t=0 ∥∇f (θ(t))∥ 2 +∥u(θ(t))∥ 2 ≤ 4C ηT 1 2γ +1+γ +L f ηC 2 1 1 2γ +γ +2γC 1 s L f γ η,(16) for every T ≥ 4C L f C 2 1 η 2 where C 1 := (2 + γ)C. The proof of Theorem 3.2 is presented in Appendix B. Intuitively,∥u(·)∥ 2 quantifies the degree of conflict between∇f and∇r; it represents how much we can reduce f without increasing r. Remark 3.3. To maximize the upper bounds(15)and(16), it can be verified that we should choose η = 2 C 1 q C L f 1 T 1 2 for any γ > max 0, 2 √ L f C − 2 . Plugging these into (15) and (16), we conclude 1 T T−1 X t=0 ∥∇f (θ(t))∥ 2 ≤ C 1 p L f C γ 1 T 1 2 ,(17) 1 T T−1 X t=0 ∥∇f (θ(t))∥ 2 +∥u(θ(t))∥ 2 ≤ 2C 1 p L f C 1 γ +1+2γ 1 T 1 2 +2 p 2C 1 γ(L f C) 1 4 1 T 1 4 ,.(18) Our results show that with a proper choice of step size, the temporal average of the norm of the forget gradient and the update direction decreases at a rate ofO(T −1/2 ) andO(T −1/4 ), respectively. 7 Remark 3.4. Using(17), we getlim t→∞ ∇f (θ(t)) = 0which indicates convergence to a stationary point of the lower-level problem, i.e.,(11)is satisfied. Further, from(18), we havelim t→∞ u(θ(t)) = 0, i.e.,lim t→∞ h ∇r(θ(t)) + γ− ⟨∇f (θ(t)),∇r(θ(t))⟩ ∥∇f (θ(t))∥ 2 ∇f (θ(t)) i = 0 . This implies that the stationary condition in(12)is satisfied withζ = γ− ⟨∇f (θ(t)),∇r(θ(t))⟩ ∥∇f (θ(t))∥ 2 ; thus, both desired optimality conditions are fulfilled. Moreover, in practice, we observe that the gradients of the forget and retain losses are often conflicting (see Fig. 3), i.e.,⟨∇f (θ(t)),∇r(θ(t))⟩≤ 0. Consequently, the termζremains non-negative, ensuring that the resulting stationary point satisfies the complementary condition in (12). 4 Experiment In this section, we evaluate the performance of the proposed algorithm,BLUR, and other state-of-the-art unlearning methods. Unlearning Tasks. We test unlearning algorithms on three popular benchmark datasets TOFU [33], MUSE [42], and WMDP [25]. The TOFU dataset contains 200 fictitious author profiles, each including 20 question-answer pairs generated by GPT-4 using predefined attributes. Here, we consider forget05 and forget10 scenarios, representing5%and10%forget sets, respectively. The MUSE dataset consists of two corpora, namely, Harry Potter books (Books) and news articles (News). The WMDP benchmark is developed for knowledge-based unlearning to remove hazardous knowledge in biosecurity, cybersecurity, and chemical security. We conduct the experiments on the TOFU dataset using the public fine-tuned LLaMA-3.2-1B-Instruct model 1 . Further, we run simulations on the MUSE benchmark using the public fine-tuned LLaMA2-7B model 2 . Finally, we exploit the Zephyr-7B-beta model 3 for WMDP. LLM Unlearning Methods. We use ‘Original’ to indicate the fine-tuned model using the TOFU/MUSE datasets and the pre-trained model for WMDP. We use ‘Retrain’ to indicate models retrained while excluding the forget set; such a ‘Retrain’ model is considered the gold standard for unlearning, and such a model is available for the TOFU and MUSE benchmarks. For the TOFU dataset, we compare the performance of NLUR–NPO against GA, GradDiff, NPO, and SimNPO. We evaluate BLUR-NPO and compare its performance against GA, GradDiff, NPO, SimNPO, and NGDiff on the MUSE dataset. For the WMDP dataset, we compare BLUR–NPO with the Representation misdirection for unlearning (RMU) developed in [25] that directs the representations of forget samples toward random representations while preserving the representations of retain samples. The forget and retain losses are defined as follows ℓ RMU,r (y|x;θ) =∥M i (x;θ)− M i (x;θ 0 )∥ 2 2 ,(19a) ℓ RMU,f (y|x;θ) =∥M i (x;θ)− c·u∥ 2 2 ,(19b) whereM i (x;θ)is a function that returns the hidden representation ofθat some layeri, and a fixed random unit vectorusampled uniformly from[0, 1). Here,cis a hyperparameter that controls activation scaling. Utilizing the gradient direction in(6)with the retain loss(19a)and the forget objective(19b) to solve the regularized optimization problem(1)is referred to as the RMU unlearning method. We evaluateBLURwith the same retain loss and forget objective as expressed in(19), resulting in a new unlearning method referred to as BLUR–RMU. Experiments are conducted on the WMDP benchmark, comparing RMU, NPO, SimNPO, BLUR–NPO, and BLUR–RMU. Additionally, we implement NGDiff method with the retain loss and forget objectives in (19), termed NGDiff–RMU. Evaluation Metrics. We list the metrics to evaluate the performance of each unlearning task. TOFU. We measure the forget quality, which assesses how well the unlearned model mimics the Retainmodel. Model utility captures the general capabilities and real-world knowledge retained by the model after unlearning. Also, we report the truth ratio, which shows how likely the model is to select the correct answer over an incorrect one. 1 https://huggingface.co/open-unlearning/tofu_Llama-3.2-1B-Instruct_full 2 https://huggingface.co/muse-bench/MUSE-news_target https://huggingface.co/muse-bench/MUSE-books_target 3 https://huggingface.co/HuggingFaceH4/zephyr-7b-beta 8 Table 2: Performance of various unlearning methods on the TOFU benchmark using the LLaMA-3.2-1B- Instruct model for forget05 and forget10. Method Forget Quality↑Forget Truth Ratio↑Model Utility↑ forget05forget10forget05forget10forget05forget10 Original2.96e–138.08e–220.470.480.600.60 Retain1.001.000.630.630.600.59 GA1.94e–1191.06e–2392.97e–249.82e–260.000.00 GradDiff1.94e–1193.76e–2198.13e–90.0020.520.49 NPO0.400.080.700.650.470.51 SimNPO 0.0680.0050.520.520.570.58 BLUR–NPO0.800.910.680.640.520.55 MUSE.After the unlearning phase, we expect to satisfy four key criteria: (1) No verbatim memoriza- tion: The model should no longer be able to generate exact substrings or sequences that match any content from the forget set. (2) No knowledge memorization on the forget set: The model should not be able to generate accurate or meaningful answers to questions regarding the forget set. (3) No privacy leakage: There should be no indication that the model was ever trained on the forget set. (4) High knowledge memorization on the retain set: The model should maintain good performance on the retain set, ensuring a high value for knowledge memorization on the retain set. WMDP.To evaluate the model’s performance on the forget and retain datasets, we use the Question- Answering (QA) technique, which involves assessing the accuracy of the model’s answers based on a given corpus.Bio. Acc.andCyber Acc.represent the model’s accuracy on WMDP-Bio and WMDP-Cyber, respectively;MMLUcontains subjects that should not be unlearned where performance is also evaluated using the QA technique. Overall, we can categorize these metrics into two classes: (1) Unlearning Effectiveness: This class evaluates how effectively undesired data and its influences are removed from model capabilities. (2) Utility Preservation: Metrics in this class assess the model’s performance on standard utility tasks after the unlearning phase. More implementation details are provided in Appendix C.1. Results. Our results are summarized below. TOFU.Table 2 compares the performance of BLUR–NPO with other unlearning baselines on TOFU forget05 and forget10. As shown, BLUR–NPO achieves outstanding performance in forget quality (0.4 to0.8for forget05 and0.08to0.91for forget10) while maintaining comparable results in forget truth ratio and model utility. MUSE.We compare the performance of BLUR–NPO with other baselines in Table 3 on both the MUSE News and Books datasets. Clearly, BLUR–NPO outperforms all baselines across all unlearning efficiency metrics, i.e., verbatim memorization (VerbMem) on the forget set, knowledge memorization (KnowMem) on the forget set, and KnowMem on the retain set. Moreover, it achieves no VerbMem on the forget set as GA, NGDiff, and NPO methods. In Fig. C.7, we present an ablation study on hyperparameters for the News corpus. To highlight the BLUR–NPO’s performance on the Books corpus, Fig. C.5 shows the trade-off between KnowMem on the forget and retain datasets. To understand how the performance of different algorithms evolves over the iterations, Fig. C.1 plots the trajectory of the VerbMem on the forget set (lower values are better) and KnowMem on the retain set (higher values are better), for the MUSE-News unlearning task. As the plots show, GA, NGDiff, NPO, and BLUR–NPO exhibit no VerbMem on the forget set, while the other baselines fail to achieve the complete no-VerbMem status. Further, BLUR–NPO achieves consistently high levels of KnowMem across all epochs. NGDiff, NPO, and SimNPO exhibit better model utility than GradDiff and GA but are still less effective than BLUR–NPO. Moreover, it is interesting to observe that BLUR–NPO first prioritizes optimizing the forget performance by aggressively reducing VerbMem to zero. Then, as the model has additional capacity, it gradually increases the KnowMem on the retain set as optimization goes. In Table 8, we provide examples of model responses after unlearning with different schemes, compared to the ground truth. For the forget set, the goal is to unlearn the Answer while for the retain set, 9 Table 3: Performance of various unlearning methods on the MUSE benchmark using the News and Books corpora with the LLaMA2-7B model. Method VerbMem onD f ↓KnowMem onD f ↓PrivLeak→ 0KnowMem onD r ↑ NewsBooksNewsBooksNewsBooksNewsBooks Original58.499.863.959.4-99.8-57.555.266.9 Retrain20.814.333.128.90.00.055.074.5 GA0.00.00.00.05.2-23.60.00.0 GradDiff4.90.031.30.0107.9-24.122.914.4 NGDiff0.00.035.70.0109.5-20.641.844.1 NPO 0.00.043.90.0109.4-30.337.531.8 SimNPO6.70.046.20.062.6-24.235.949.3 BLUR–NPO0.00.029.00.0109.5-22.646.752.7 the objective is to generate a response that matches the Answer. These visualizations show that BLUR– NPO delivers the best results, accurately responding to questions from the retain set while providing non-informative responses to questions from the forget set. However, all other unlearning methods either fail to provide fully correct answers for the retain set or are unable to decline answering questions from the forget set. WMDP. Table 4 compares our algorithms, BLUR–NPO and BLUR–RMU, with RMU, NPO, Sim- NPO, and NGDiff–RMU on the WMDP benchmark. Recall that BLUR–NPO is the specification of the retain loss in (2) and the forget objective in (4). Also, the BLUR–RMU and NGDiff–RMU leverage the loss function developed in the RMU algorithm. As shown, BLUR–NPO and BLUR–RMU outperform all other methods in unlearning efficiency. BLUR–NPO performs better than NPO and SimNPO but is less effective than RMU in utility preservation. Interestingly, BLUR–RMU achieves roughly the same performance as RMU and NGDiff–RMU on the retain set, while significantly outperforming RMU on un- learning efficacy. These results highlight the effectiveness of our approach relative to previously proposed algorithms. To visualize better, we compute the average accuracy of WMDP-Bio and WMDP-Cyber and then plot it vs. the accuracy of MMLU in Fig. C.6. 5 Conclusion Table 4: Performance of various unlearning methods on the WMDP benchmark. Method Unlearning EfficacyUtility Preservation Bio. Acc. ↓Cyber Acc. ↓MMLU↑ Original63.744.058.1 RMU31.228.257.1 NPO42.528.340.0 SimNPO41.632.247.1 NGDiff–RMU35.626.857.4 BLUR–NPO27.626.548.4 BLUR–RMU26.926.657.0 In this paper, we introduced a new LLM unlearn- ing framework based on a bi-level optimization approach that prioritizes the forget loss over the retain objective in a hierarchical structure. To solve the proposed optimization problem, we de- veloped a novel algorithm, termedBLUR. Then, we provided the theoretical analysis for the non- convex setting. Our extensive experiments on var- ious LLM unlearning tasks demonstrated that our approach outperformed all the state-of-the-art al- gorithms. For future work, we will explore higher- order algorithms to further enhance unlearning efficiency and effectiveness. References [1] Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. Identifying and mitigating the security risks of generative ai. Foundations and Trends® in Privacy and Security, 6(1):1–52, 2023. [2] Reuben Binns. Fairness in machine learning: Lessons from political philosophy. In Conference on fairness, accountability and transparency, pages 149–159. PMLR, 2018. 10 [3]Zhiqi Bu, Xiaomeng Jin, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Mingyi Hong. Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate. arXiv preprint arXiv:2410.22086, 2024. [4]Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan, Dejing Dou, and Jun Huan. Fast federated machine unlearning with nonlinear functional theory. In International conference on machine learning, pages 4241–4268. PMLR, 2023. [5] Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Commu- nications Security, pages 499–513, 2022. [6]Mathieu Dagr ́ eou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems, 35:26698–26710, 2022. [7]Stephan Dempe, Nguyen Dinh, Joydeep Dutta, and Tanushree Pandit. Simple bilevel programming and extensions. Mathematical Programming, 188:227–253, 2021. [8] Joydeep Dutta and Tanushree Pandit. Algorithms for simple bilevel programming. Bilevel Opti- mization: Advances and Next Challenges, pages 253–291, 2020. [9] Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms, 2023. [10]Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163, 2024. [11]Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint arXiv:2310.12508, 2023. [12]Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017. [13]Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning, pages 1568–1577. PMLR, 2018. [14] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426–2436, 2023. [15] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10:8–9, 2021. [16] GDPR GDPR. General data protection regulation. Regulation (EU), 679, 2016. [17] Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Jogging the memory of unlearned model through targeted relearning attack. arXiv preprint arXiv:2406.13356, 2024. [18] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. 11 [19]Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022. [20]Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning, pages 4882–4892. PMLR, 2021. [21] Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. Wagle: Strategic weight attribution for effective and modular unlearning in large language models. arXiv preprint arXiv:2410.17509, 2024. [22]Ruichen Jiang, Nazanin Abolfazli, Aryan Mokhtari, and Erfan Yazdandoost Hamedani. A con- ditional gradient-based method for simple bilevel optimization with convex lower-level problem. In International Conference on Artificial Intelligence and Statistics, pages 10305–10323. PMLR, 2023. [23]Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. arXiv preprint arXiv:2310.13771, 2023. [24] Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, pages 12–24, 2023. [25] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024. [26]Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR, 2022. [27] Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu. Federated unlearning. arXiv preprint arXiv:2012.13891, 2020. [28]Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058, 2024. [29]Ziyao Liu, Yu Jiang, Jiyuan Shen, Minyi Peng, Kwok-Yan Lam, and Xingliang Yuan. A survey on federated unlearning: Challenges, methods, and future directions. arXiv preprint arXiv:2310.20448, 2023. [30]JakubŁucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tram ` er, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXiv:2409.18025, 2024. [31]Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024. [32] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, pages 2113–2122. PMLR, 2015. [33]Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for llms, 2024. [34] Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. More human than human: Measuring chatgpt political bias. Available at SSRN 4372349, 2023. 12 [35]Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tram ` er, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035, 2023. [36]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. [37] Jeffrey Rosen. The right to be forgotten. Stan. L. Rev. Online, 64:88, 2011. [38]Shoham Sabach and Shimrit Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization, 27(2):640–660, 2017. [39] Bhavika Sachdeva, Harshita Rathee, Arun Sharma, Witold Wydma ́ nski, et al. Machine unlearning for recommendation systems: An insight. arXiv preprint arXiv:2401.10942, 2024. [40] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086, 2021. [41]Lingqing Shen, Nam Ho-Nguyen, and Fatma Kılınc ̧-Karzan. An online convex optimization-based framework for convex bilevel optimization. Mathematical Programming, 198(2):1519–1582, 2023. [42]Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024. [43]Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint arXiv:2407.00106, 2024. [44]Bradly Stadie, Lunjun Zhang, and Jimmy Ba. Learning intrinsic rewards as a bi-level optimization problem. In Conference on Uncertainty in Artificial Intelligence, pages 111–120. PMLR, 2020. [45]Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 2024. [46] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Un- derstanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. IEEE, 2022. [47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [48] Jiulin Wang, Xu Shi, and Rujun Jiang. Near-optimal convex simple bilevel optimization with a bisection method. In International Conference on Artificial Intelligence and Statistics, pages 2008–2016. PMLR, 2024. [49] Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the implicit toxicity in large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 13 [50]Kun Wu, Jie Shen, Yue Ning, Ting Wang, and Wendy Hui Wang. Certified edge unlearning for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2606–2617, 2023. [51]Mimee Xu, Jiankai Sun, Xin Yang, Kevin Yao, and Chong Wang. Netflix and forget: Efficient and exact machine unlearning from bi-linear recommendations. arXiv preprint arXiv:2302.06676, 2023. [52]Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023. [53]Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. When demonstrations meet generative world models: A maximum likelihood framework for offline inverse reinforcement learning. Advances in Neural Information Processing Systems, 36:65531–65565, 2023. [54] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024. [55]Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. A The Preliminaries Lemma A.1. For anyθ ∈ R d , we can write (a) ⟨∇f (θ),u(θ)⟩ = γ∥∇f (θ)∥ 2 . Also, under Assumption 3.1, for everyθ ∈ R d , we get (b) ∥u(θ)∥≤ C 1 , (c) ξ(θ)∥∇f (θ)∥ 2 ≤ γ∥∇f (θ)∥ 2 + C∥∇f (θ)∥, (d) ξ(θ)⟨∇f (θ),u(θ)⟩≤ γ 2 ∥∇f (θ)∥ 2 + Cγ∥∇f (θ)∥, (e) ⟨∇r(θ),u(θ)⟩≥∥u(θ)∥ 2 − γ 2 ∥∇f (θ)∥ 2 − Cγ∥∇f (θ)∥, where ξ(θ) := γ∥∇f (θ)∥ 2 −⟨∇f (θ),∇r(θ)⟩ ∥∇f (θ)∥ 2 . Proof. From the definition of ξ(θ) and u(θ) in (13), we have ⟨∇f (θ),u(θ)⟩ =⟨∇f (θ),∇r(θ)⟩ + ξ(θ)∥∇f (θ)∥ 2 =⟨∇f (θ),∇r(θ)⟩ + γ∥∇f (θ)∥ 2 −⟨∇r(θ),∇r(θ)⟩ = γ∥∇f (θ)∥ 2 , that completes the proof of part (a). From the definition ofu(θ)and the triangle inequality, we can write ∥ξ(θ)∇f (θ) +∇r(θ)∥≤|ξ(θ)|∥∇f (θ)∥ +∥∇r(θ)∥ ≤ γ∥∇f (θ)∥ + |⟨∇f (θ),∇r(θ)⟩| ∥∇f (θ)∥ +∥∇r(θ)∥ (a) ≤ γ∥∇f (θ)∥ + 2∥∇r(θ)∥≤ (2 + γ)C, where(a)follows from the Cauchy-Schwartz inequality and the last step holds due to the Assumption 3.1. 14 To prove part (c), using the Cauchy-Schwartz inequality, we get ξ(θ)∥∇f (θ)∥ 2 = γ∥∇f (θ)∥ 2 −⟨∇f (θ),∇r(θ)⟩ ≤ γ∥∇f (θ)∥ 2 +∥∇f (θ)∥·∥∇r(θ)∥ ≤ γ∥∇f (θ)∥ 2 +C∥∇f (θ)∥,(20) where the last inequality holds from Assumption 3.1. Now, we prove part (d). From (13), we have ξ(θ)⟨∇f (θ),u(θ)⟩ = ξ(θ)⟨∇f (θ),ξ(θ)∇f (θ) +∇r(θ)⟩ = ξ(θ) ξ(θ)∥∇f (θ)∥ 2 +⟨∇f (θ),∇r(θ)⟩ = ξ(θ) γ∥∇f (θ)∥ 2 −⟨∇f (θ),∇r(θ)⟩ +⟨∇f (θ),∇r(θ)⟩ = ξ(θ)γ∥∇f (θ)∥ 2 .(21) This together with part (c) leads us to part (d). Finally, to show part (e), from part (d), we get ⟨∇r(θ),u(θ)⟩ =⟨u(θ)− ξ(θ)∇f (θ),u(θ)⟩ =∥u(θ)∥ 2 −ξ(θ)⟨∇f (θ),u(θ)⟩≥∥u(θ)∥ 2 −γ 2 ∥∇f (θ)∥ 2 −Cγ∥∇f (θ)∥. Lemma A.2. For any vectorsu i n i=1 ∈ R d , we get∥ P n i=1 u i ∥ 2 ≤ n P n i=1 ∥u i ∥ 2 . B Proof of Theorem 3.2 Using the fact that f is L f -Lipschitz from Assumption 3.1, we get f (θ(t+1))−f (θ(t))≤⟨∇f (θ(t)),θ(t + 1)−θ(t)⟩ + L f 2 ∥θ(t + 1)−θ(t)∥ 2 =−η⟨∇f (θ(t)),u(θ(t))⟩ + L f 2 η 2 ∥u(θ(t))∥ 2 (a) = −ηγ∥∇f (θ(t))∥ 2 + L f 2 η 2 ∥u(θ(t))∥ 2 ≤−ηγ∥∇f (θ(t))∥ 2 + L f 2 η 2 C 2 1 , (22) where(a)follows from Lemma A.1-(a), and the last step holds due to Lemma A.1-(b). Applying a telescopic summation in (22), we can write 1 T T−1 X t=0 ∥∇f (θ(t))∥ 2 ≤ 1 Tηγ (f (θ(0))− f (θ(T ))) + L f 2γ ηC 2 1 ≤ 2C Tηγ + L f 2γ ηC 2 1 , where the last step follows from Assumption 3.1. That completes the proof of the first claim of the theorem. Similarly, since r is L r -Lipschitz from Assumption 3.1, we can write r(θ(t + 1))− r(θ(t))≤⟨∇r(θ(t)),θ(t + 1)−θ(t)⟩ + L r 2 ∥θ(t + 1)−θ(t)∥ 2 =−η∥u(θ(t))∥ 2 + ηξ(θ(t))⟨∇f (θ(t)),u(θ(t))⟩ + L r 2 η 2 ∥u(θ(t))∥ 2 ≤− η 2 ∥u(θ(t))∥ 2 + η γ 2 ∥∇f (θ(t))∥ 2 + Cγ∥∇f (θ(t))∥ .(23) 15 Applying a telescopic summation in (23), we arrive at 1 T T−1 X t=0 ∥u(θ(t))∥ 2 ≤ 2 ηT (r(θ(0))− r(θ(t))) + 2 T γ 2 T−1 X t=0 ∥∇f (θ(t))∥ 2 + γ T−1 X t=0 ∥∇f (θ(t))∥ ! ≤ 4C ηT + 2 T γ 2 T−1 X t=0 ∥∇f (θ(t))∥ 2 + γ T−1 X t=0 ∥∇f (θ(t))∥ ! (a) ≤ 4C ηT + 2 T γ 2 T−1 X t=0 ∥∇f (θ(t))∥ 2 + γ √ T v u u t T−1 X t=0 ∥∇f (θ(t))∥ 2 (b) ≤ 4C ηT + 2 T 2Cγ η + L f γ 2 ηC 2 1 T + γ √ T s 2C ηγ + L f 2γ ηC 2 1 T ! ≤ 4C ηT (1 + γ) + L f γηC 2 1 + 2γC 1 s L f γ η, where(a)follows from Lemma A.2, step(b)holds due to(15), and finally the last inequality can be concluded from T ≥ 4C L f C 2 1 η 2 . C Additional Experiment Results and Details C.1 Experiment Setups Computational Configurations. All experiments are conducted on 8 NVIDIA A100 GPUs. Next, we present the hyperparameters used in each unlearning task, TOFU, MUSE, and WMDP. TOFU. We conduct experiments for 10 epochs using a learning rate of10 −5 and a batch size of 32. A grid search is performed over the range[0.5, 2]forγ, and the hyperparameterβis searched within [0.05, 0.2], with the final value of0.1. We setλ = 1for GradDiff and NPO, andλ = 2.5for SimNPO. The hyperparametersβfor NPO and SimNPO are set to0.1and2.5, respectively. For the SimNPO unlearning scheme, the hyperparameterαis fixed at0.125. We summarized the hyperparameters in Table 5. Methodηβγαλ GA10 −5 ---- GradDiff10 −5 ---1.0 NPO10 −5 0.1--1.0 SimNPO10 −5 2.5-2.00.15 BLUR–NPO10 −5 0.11.0-- Table 5: Hyperparamters for various unlearning methods on TOFU benchmark MUSE.We train our algorithm BLUR–NPO for 10epochs with a constant learning rate of2.5× 10 −5 for the news dataset and10 −5 for the books dataset, the batch size of32, and an input length of2, 048tokens. We perform a grid search forγ in the range of[0.8, 1.2]and setγ = 1.0as the final value. For the NPO loss in(4),βis set to 0.05for the news dataset and0.4for the books dataset. The evaluation pipelines strictly follow the setup detailed by [42]. Further, we evaluate the model’s performance after the unlearning process and select the optimal model as the final result. We use a constant learning rate of2.5× 10 −5 for news and5× 10 −6 for books datasets in NGDiff, with β = 0.1for both corpora. The hyperparameters for other unlearning methods GA, GradDiff, NPO, and SimNPO are set according to the works of [42,10]. Table 6 summarizes our method’s optimal combination of the hyperparameters, determined through grid search. We used the hyperparameters reported in the corresponding papers for other unlearning methods. WMDP.We run the experiments for BLUR–NPO using a constant learning rate of2× 10 −6 and a batch size of4. A grid search is performed forγwithin the range[0.5, 1.5], and the parameterβin(4) is explored within the range[0.001, 0.01]. As for the RMU and BLUR–RMU algorithms, we follow the implementation details provided in WMDP [25]. More precisely, for BLUR–RMU, we conduct a grid search for the parameterγover the range[0.001, 0.002]and use 0.00125 as the finalγ. We 16 MethodDatasetηβγαλ GANews/Books10 −5 ---0 GradDiffNews/Books10 −5 ---1 NGDiff News2.5× 10 −5 0.1--- Books5× 10 −6 0.1--- NPONews/Books10 −5 ---1 SimNPO News10 −5 0.7-0.00.1 Books10 −5 0.75-0.00.1 BLUR–NPO News2.5× 10 −5 0.051.0-- Books10 −5 0.41.0-- Table 6: Hyperparameters for various unlearning methods on MUSE benchmark Methodηβγαλc RMU5× 10 −5 ---12006.5 NPO10 −5 0.1--1.0- SimNPO10 −5 0.1-0.01.0- NGDiff–RMU10 −5 ----6.5 BLUR–NPO2× 10 −6 0.0051.0--- BLUR–RMU4× 10 −2 -0.00125--6.5 Table 7: Hyperparamters for various unlearning methods on WMDP benchmark also employ a constant learning rate of4× 10 −2 and train for 150 steps for BLUR–RMU. We exploit lm-evaluation-harness v0.4.2[15] to standardize prompts. We set the experimental parameters for other unlearning methods, RMU, NPO, and SimNPO, as described in [25,21,10]. We set a fine-tuned and constant learning rate of10 −5 for NGDiff–RMU method. Table 7 summarizes all the hyperparameters used for the unlearning tasks on WMDP. C.2 Additional Experiment Results Here, we discuss our experimental results further. As Fig. C.2 shows, the update directionˆu(θ)cannot consistently prioritize the forget loss over the retain loss, even for various values ofλ. This further corroborates that the regularized problem formulation in(1)fails to achieve a proper balance between the forget and retain losses. Fig. C.3 and Fig. C.4 demonstrate that the retain and forget gradients conflict over the unlearning steps across various unlearning schemes and values of λ. 17 Forget Set Who defended the plan for juryless rape trials in Scotland by pointing to a ’weight of evidence’ that juries are affected by rape myths and misconceptions? Answer First Minister Humza Yousaf GA • GradDiff 1st Minister Humza Yousaf • None ’I’m not a gambler’ - Selby and the Crucible• None The NGDiff 1st Minister Humza Yousaf NPO 1st Minister Humza Yousaf SimNPO1st Minister Humza Yousaf BLUR–NPO100% of lawyers surveyed Retain Set How many years ago did David and Janice Hunter retire to Cyprus? Answer 20 years ago GA • GradDiff 20the ’Happy the the the the the’ couple, as the 2023 ’The Happy the the the’ book and the NGDiff 25years NPO 25years SimNPO 25years BLUR–NPO 20 years ago Table 8: Examples of generated text from different unlearned models in the MUSE-News dataset. Failed unlearning is indicated by undesired answers highlighted inred, while successful unlearning is shown ingreen for desired responses. Repeated or irrelevant information is marked inyellow . 0246810 Epoch 0 10 20 30 40 50 60 VerbMem on f BLUR-NPO GA GradDiff NGDiff NPO SimNPO 0246810 Epoch 0 10 20 30 40 50 KnowMem on r BLUR-NPO GA GradDiff NGDiff NPO SimNPO Figure C.1: Verbatim memorization on the forget setD f (top) and knowledge memorization on the retain setD r (bottom) vs. optimization epochs, using various unlearning methods on the MUSE-News dataset. 02004006008001000 Step 5 4 3 2 1 0 1 Alignment A f A r 02004006008001000 Step 3 2 1 0 1 2 Alignment A f A r 02004006008001000 Step 2 1 0 1 2 3 4 5 Alignment A f A r Figure C.2: Alignment values of forget and retain objectives as defined in(8)on MUSE-News dataset using LLaMa2-7B model vs. step with λ = 0.5 (top-left), λ = 1.5 (top-right), and λ = 2 (bottom). 18 02004006008001000 Step 0.8 0.6 0.4 0.2 0.0 0.2 Cosine Similarity 02004006008001000 Step 1.0 0.8 0.6 0.4 0.2 0.0 0.2 Cosine Similarity 02004006008001000 Step 1.0 0.8 0.6 0.4 0.2 0.0 0.2 Cosine Similarity Figure C.3: Cosine similarity of the gradient forget and retain loss functions using NPO method on MUSE-News dataset using LLaMa2-7B model vs. step withλ = 0.5(top-left),λ = 1.5(top-right), and λ = 2 (bottom). 02004006008001000 Step 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 Cosine Similarity 02004006008001000 Step 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Cosine Similarity 02004006008001000 Step 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Cosine Similarity Figure C.4: Cosine similarity of the gradient forget and retain loss functions using GradDiff method on MUSE-News dataset using LLaMa2-7B model vs. step withλ = 0.5(top-left),λ = 1(top-right), and λ = 1.5 (bottom). Fig. C.5 illustrates the KnowMem trade-off between the forget and retain datasets on MUSE-Books. As observed, all unlearning methods achieve no-KnowMem status on the forget set, while BLUR–NPO outperforms all other methods regarding retain KnowMem. Fig. C.6 presents the trade-off between model utility and unlearning efficiency on the WMDP dataset. In particular, BLUR–RMU outperforms the RMU method 10% in WMDP accuracy while maintaining nearly the same performance in MMLU accuracy. 010203040506070 KnowMem on r 0 10 20 30 40 50 60 KnowMem on f Original GA GradDiff NGDiff NPO SimNPO BLUR-NPO Ideal Figure C.5: Trade-off between KnowMem val- ues on the forget and retain datasets using differ- ent unlearning methods, LLaMA2-7B model, and MUSE-Books dataset where BLUR–NPO outper- forms SOTA models. 40.042.545.047.550.052.555.057.5 MMLU Accuracy 0 10 20 30 40 50 WMDP Accuracy Original RMU NPO SimNPO BLUR-NPO BLUR-RMU Ideal Figure C.6: Trade-off between WMDP Accuracy and MMLU Accuracy using different unlearning methods with Zephyr-7B-beta model. Our algo- rithm BLUR–RMU outperforms RMU. Ablation Studies of BLUR–NPO on MUSE-News Corpus. The proposed descent directionu(θ)in(13)exploits a hyperparameterγadjusting the amplitude of the forget gradient. Moreover, for BLUR–NPO, we use NPO loss for the forget objective, where the hyperparameterβregulates the intensity of unlearning. Asβ → 0, NPO loss converges to the GA loss. Fig. C.7 illustrates the performance of KnowMem on the forget and retain datasets of BLUR–NPO for various values ofβandγon MUSE-News using the LLaMA2-7B model with two learning rates 19 η = 10 −5 andη = 2.5× 10 −5 . As observed, a large value ofβfails to unlearn the forget set while maintaining good performance on the retain set. Moreover, the learning rateη = 10 −5 is not large enough to unlearn the forget set while preserving the model utility, whereas a larger learning rateη = 2.5× 10 −5 with a relatively smallβ = 0.05andγ = 1.0achieves both objectives simultaneously. The unlearning metrics for this optimal setup are highlighted with red rectangles in Fig. C.7. 47.4 44.847.0 42.9 47.3 45.746.443.048.040.3 41.840.6 37.939.339.2 28.235.136.532.335.7 0.00.00.00.00.0 0.8 0.9 1.0 1.11.2 0.2 0.15 0.1 0.05 → 0 γ β (a) KnowMem on forget dataset with η = 10 −5 . 42.043.2 44.7 43.8 42.4 46.043.341.8 44.7 39.2 42.142.1 40.543.5 44.4 35.9 34.637.733.426.4 0.00.00.00.00.0 0.8 0.9 1.0 1.11.2 0.2 0.15 0.1 0.05 → 0 γ β (b) KnowMem on retain dataset with η = 10 −5 . 50.540.738.041.841.3 40.4 39.6 42.3 42.2 30.8 47.3 39.039.9 31.035.0 32.9 34.2 29.0 46.7 22.1 0.00.00.00.00.0 0.8 0.9 1.0 1.11.2 0.2 0.15 0.1 0.05 → 0 γ β (c) KnowMem on forget dataset with η = 2.5×10 −5 . 38.240.743.444.840.6 40.2 41.9 43.738.042.6 42.046.0 39.5 44.7 40.4 27.1 30.346.743.640.1 0.00.00.00.00.0 0.8 0.9 1.0 1.11.2 0.2 0.15 0.1 0.05 → 0 γ β (d) KnowMem on retain dataset with η = 2.5×10 −5 . Figure C.7: KnowMem values on forget and retain datasets using BLUR–NPO unlearning method, LLaMA2-7B model, and MUSE-News corpus under two learning ratesη = 10 −5 andη = 2.5× 10 −5 with different combinations of hyperparameters of β and γ. 20