Paper deep dive

On Effects of Steering Latent Representation for Large Language Model Unlearning

Dang Huu-Tien, Trung-Tin Pham, Hoang Thanh-Tung, Naoya Inoue

Year: 2025Venue: AAAI 2025Area: Model EditingType: EmpiricalEmbeddings: 121

Models: Llama-3-8B, Mistral-7B, Zephyr-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/11/2026, 12:32:43 AM

Summary

The paper introduces 'Adaptive RMU', an improvement over Representation Misdirection for Unlearning (RMU) for LLMs. RMU steers intermediate layer representations of forget-samples to a random vector to induce unlearning. The authors theoretically demonstrate that this process reduces token confidence, leading to nonsensical outputs, and identify that RMU's effectiveness diminishes in middle and later layers due to scaling coefficient limitations. Adaptive RMU addresses this by dynamically adjusting the coefficient based on the norm of the forget representation, improving performance without extra computational cost.

Entities (4)

Adaptive RMU · method · 100%Representation Misdirection for Unlearning · method · 100%Large Language Model · technology · 98%Machine Unlearning · field-of-study · 95%

Relation Signals (3)

Representation Misdirection for Unlearning → appliedto → Large Language Model

confidence 98% · RMU... is an effective method for large language model (LLM) unlearning.

Adaptive RMU → improves → Representation Misdirection for Unlearning

confidence 95% · Adaptive RMU significantly improves the unlearning performance compared to prior art

Adaptive RMU → addresseslimitationof → Representation Misdirection for Unlearning

confidence 92% · To resolve this drawback [of RMU], we propose Adaptive RMU

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Representation Misdirection for Unlearning (RMU), which steers model representation in the intermediate layer to a target random representation, is an effective method for large language model (LLM) unlearning. Despite its high performance, the underlying cause and explanation remain underexplored. In this paper, we theoretically demonstrate that steering forget representations in the intermediate layer reduces token confidence, causing LLMs to generate wrong or nonsense responses. We investigate how the coefficient influences the alignment of forget-sample representations with the random direction and hint at the optimal coefficient values for effective unlearning across different network layers. We show that RMU unlearned models are robust against adversarial jailbreak attacks. Furthermore, our empirical analysis shows that RMU is less effective when applied to the middle and later layers in LLMs. To resolve this drawback, we propose Adaptive RMU--a simple yet effective alternative method that makes unlearning effective with most layers. Extensive experiments demonstrate that Adaptive RMU significantly improves the unlearning performance compared to prior art while incurring no additional computational cost.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

120,800 characters extracted from source content.

Expand or collapse full text

On Effects of Steering Latent Representation for Large Language Model Unlearning Dang Huu-Tien1, Tin Pham1, Hoang Thanh-Tung2, and Naoya Inoue1,3 Abstract Representation Misdirection for Unlearning (RMU), which steers model representation in the intermediate layer to a target random representation, is an effective method for large language model (LLM) unlearning. Despite its high performance, the underlying cause and explanation remain underexplored. In this paper, we theoretically demonstrate that steering forget representations in the intermediate layer reduces token confidence, causing LLMs to generate wrong or nonsense responses. We investigate how the coefficient influences the alignment of forget-sample representations with the random direction and hint at the optimal coefficient values for effective unlearning across different network layers. We show that RMU unlearned models are robust against adversarial jailbreak attacks. Furthermore, our empirical analysis shows that RMU is less effective when applied to the middle and later layers in LLMs. To resolve this drawback, we propose Adaptive RMU—a simple yet effective alternative method that makes unlearning effective with most layers. Extensive experiments demonstrate that Adaptive RMU significantly improves the unlearning performance compared to prior art while incurring no additional computational cost. 1 Introduction LLMs achieved remarkable performance through pre-training on large amounts of internet texts and rigorous alignment processes for safety enhancement. Despite the immense effort in safety research, LLMs are still vulnerable to adversarial jailbreak attacks and can exhibit unwanted behaviors (Shah et al. 2023; Zou et al. 2023b; Jones et al. 2023; Yuan et al. 2024; Wei, Haghtalab, and Steinhardt 2024). Machine Unlearning (Cao and Yang 2015; Bourtoule et al. 2021; Nguyen et al. 2022; Xu et al. 2023; Liu et al. 2024c) has emerged as a promising method for mitigating unforeseen risks in LLMs before deployment. Li et al. (2024b) introduced Representation Misdirection for Unlearning (RMU)—an unlearning method that steers the representations of forget-samples (i.e. samples that the model should forget) toward a random representation while keeping the representations of retain-samples (i.e. samples that the model should remember) unchanged. RMU significantly degrades models’ accuracy on forget-tasks, while only slightly affecting the performance on retain-tasks and demonstrates stronger robustness against adversarial jailbreak attacks. However, the reason for RMU’s effectiveness is not well understood, hindering the development of better unlearning algorithms. In this paper, we make the following contributions: • We theoretically analyze the impact of the RMU method on LLM unlearning. • We investigate the connection between RMU and adversarial robustness. We demonstrate that RMU impedes the adversary’s ability to determine optimal updates for generating adversarial samples, thus improving the adversarial robustness of the unlearned model. • We empirically show that the RMU forget loss, which minimizes the mean squared error (MSE) between forget representation and a fixed scaled random vector, fails to converge when the norm of the forget representation is larger than the scaling coefficient, making RMU less effective when applied to middle and last layers in LLMs. • To overcome RMU’s limitation, we introduce Adaptive RMU—a variant that adaptively adjusts the coefficient value based on the norm of the forget representation. Experimental results show that Adaptive RMU achieves higher drop-in-accuracy for forget knowledge, maintaining high performance on general knowledge, and enables effective unlearning for most layers without incurring additional computational overhead. 2 Background and Related Work Machine Unlearning. A natural unlearning approach is leave-some-out retraining: retraining the model from scratch without the forget samples. However, this method becomes more computationally expensive as the size of datasets and modern deep networks grows. Existing works focus on approximating unlearning (Warnecke et al. 2021; Izzo et al. 2021; Sekhari et al. 2021; Isonuma and Titov 2024) using influence function (Koh and Liang 2017; Grosse et al. 2023), gradient ascent (Thudi et al. 2022), second-order approximation (Jia et al. 2024), negative preference optimization (Zhang et al. 2024b), and embedding corrupted (Liu et al. 2024a). Other views on the landscape of machine unlearning include: unlearning in text classification (Ma et al. 2022), image classification and recognition (Ginart et al. 2019; Golatkar, Achille, and Soatto 2020; Fan et al. 2024; Choi and Na 2023; Cha et al. 2024), image-to-image generative models (Li et al. 2024a), diffusion models (Gandikota et al. 2023; Zhang et al. 2024a; Kumari et al. 2023; Bui et al. 2024), multimodal unlearning (Cheng and Amiri 2023), federated unlearning (Romandini et al. 2024; Wang et al. 2022; Che et al. 2023; Halimi et al. 2022; Jeong, Ma, and Houmansadr 2024), graph unlearning (Chen et al. 2022; Chien, Pan, and Milenkovic 2023; Wu et al. 2023a; Cheng et al. 2023; Dukler et al. 2023; Zhu, Li, and Hu 2023; Li et al. 2024c; Tan et al. 2024), recommender systems (Zhang et al. 2023; Chen et al. 2024; Li et al. 2023; Wang et al. 2025), certified minimax unlearning (Liu et al. 2024b), targeted types of unlearning information (Cooper et al. 2024), and evaluation on unlearning (Lynch et al. 2024; Hayes et al. 2024; Shi et al. 2024a, b). LLM Unlearning. Due to the large size of the parameters and training data, LLM poses a new challenge to unlearning. Recent studies in LLM unlearning mainly focus on task or context-specific settings such as unlearning copyrighted material from the Harry Potter series (Eldan and Russinovich 2023), in-context unlearning (Pawelczyk, Neel, and Lakkaraju 2024), fictitious unlearning (Maini et al. 2024), specific harmful input-output (Yao, Xu, and Liu 2023; Liu et al. 2024d), sensitive and private information (Jang et al. 2023; Wu et al. 2023b; Patil, Hase, and Bansal 2024), gender bias (Belrose et al. 2023) or concepts (Hong et al. 2024; Bui et al. 2024). More recently, Li et al. (2024b) consider unlearning an entire distribution of hazardous knowledge given limited samples. Notation & problem formulation. Let forgetsubscriptforgetD_forgetDforget and retainsubscriptretainD_retainDretain be the forget and retain sets, respectively. Let fθ:ℝn×d↦ℝn×|V|:subscriptmaps-tosuperscriptℝsuperscriptℝf_θ:R^n× d ^n×|V|fitalic_θ : blackboard_Rn × d ↦ blackboard_Rn × | V | be an autoregressive LLM parameterized by θ that maps a prompt input x1:nsubscript:1x_1:nx1 : n consisting of n tokens x1,x2,…,xnsubscript1subscript2…subscript\x_1,x_2,...,x_n\ x1 , x2 , … , xitalic_n to an output of probability distributions over the vocabulary V. We denote hθ(l)⁢(x)superscriptsubscriptℎh_θ^(l)(x)hitalic_θ( l ) ( x ) the averaged hidden states of all tokens in x1:nsubscript:1x_1:nx1 : n obtained from the l-th layer of fθsubscriptf_θfitalic_θ. For simplicity, throughout this paper, we use h(l)⁢(x)superscriptℎh^(l)(x)h( l ) ( x ) to present hθ(l)⁢(x)superscriptsubscriptℎh_θ^(l)(x)hitalic_θ( l ) ( x ). For operators, we denote ∘ ∘ as the decomposition operator, and ||⋅||||·||| | ⋅ | | is the Euclidean norm. Our goal is to unlearn the undesired harmful knowledge forgetsubscriptforgetD_ forgetDforget from fθsubscriptf_θfitalic_θ while retaining general knowledge retainsubscriptretainD_ retainDretain. Unlearned models should be robust to knowledge recovery attacks that attempt to recover harmful knowledge from the model. Representation Misdirection for Unlearning (RMU; Li et al. (2024b)) is a fine-tuning based unlearning method inspired by representation engineering (Zou et al. 2023a) that steers the model’s representation of forget samples xF∈forgetsubscriptsubscriptforgetx_F _forgetxitalic_F ∈ Dforget to a random vector and regularizes the model representation of retain samples xR∈retainsubscriptsubscriptretainx_R _retainxitalic_R ∈ Dretain back to the original model representation, by optimizing the MSE loss: ℒ =xF∈forget⁢‖hθunlearn(l)⁢(xF)−c⁢‖22absentsubscriptsubscriptsubscriptforgetsuperscriptsubscriptnormsuperscriptsubscriptℎsuperscriptunlearnsubscript22 =E_x_F _forget||h_θ^% unlearn^(l)(x_F)-c u||_2^2= blackboard_Ex start_POSTSUBSCRIPT F ∈ Dforget end_POSTSUBSCRIPT | | hitalic_θunlearn( l ) ( xitalic_F ) - c italic_u | |22 +α⁢xR∈retain⁢‖hθunlearn(l)⁢(xR)−hθfrozen(l)⁢(xR)‖22,subscriptsubscriptsubscriptretainsuperscriptsubscriptnormsuperscriptsubscriptℎsuperscriptunlearnsubscriptsuperscriptsubscriptℎsuperscriptfrozensubscript22 + _x_R _retain||h_% θ^unlearn^(l)(x_R)-h_θ^frozen^(l)(x_R)|% |_2^2,+ α blackboard_Ex start_POSTSUBSCRIPT R ∈ Dretain end_POSTSUBSCRIPT | | hitalic_θunlearn( l ) ( xitalic_R ) - hitalic_θfrozen( l ) ( xitalic_R ) | |22 , (1) where θunlearnsuperscriptunlearnθ^unlearnθunlearn and θfrozensuperscriptfrozenθ^frozenθfrozen are parameters of the update model and frozen model respectively, uitalic_u is a fixed random unit vector where each element is sampled from Uniform distribution U⁢(0,1)01U(0,1)U ( 0 , 1 ), c∈ℝc ∈ blackboard_R is a fixed scaling coefficient and α∈ℝα α ∈ blackboard_R is a retain weight. RMU updates θunlearnsuperscriptunlearnθ^unlearnθunlearn toward the direction of the gradient of the loss ℒLL using gradient descent. 3 Theoretical Analysis 3.1 The Confidence of Tokens Generated by RMU Models In general, samples from the shifted distribution (such as wrong label or out-of-distribution) are associated with smaller “confidence” scores such as softmax probability (Hendrycks and Gimpel 2017; Northcutt, Jiang, and Chuang 2021), maximum logit (Hendrycks et al. 2022; Wei et al. 2022), ℓ2superscriptℓ2 ^2ℓ2-distance (Sun et al. 2022), energy score (Liu et al. 2020), and cosine similarity (Ngoc-Hieu et al. 2023). Recently, LLM has shown a tendency to produce a lower (higher) confidence in its incorrect (correct) answers in multiple-choice Q&A (Plaut, Nguyen, and Trinh 2024). Building on previous works, we hypothesized that the logit of generated tokens by RMU models exhibit randomness. As seen by a deep network, such randomization signifies low confidence in the logit, resulting in nonsensical or incorrect responses. To validate the hypothesis, we conducted an analysis of the logits of generated tokens produced by RMU models. To facilitate subsequent analysis, we make the following definition and assumption. Definition 1. (Unlearned model & logit of forget-tokens on unlearned model). Let f(l:k)=g(l:k)∘h(l)superscript:superscript:superscriptℎf^(l:k)=g^(l:k) h^(l)f( l : k ) = g( l : k ) ∘ h( l ), where g(l:k)superscript:g^(l:k)g( l : k ) be the transformation from layer l to layer k of network f, for any two layers k>lk>lk > l; l∈[1⁢…⁢L]delimited-[]1…l∈[1...L]l ∈ [ 1 … L ]. We define the unlearned model funlearn=⁢(f(l:L),steered)=⁢(g(l:L)∘h(l),steered)superscriptunlearnsuperscript:steeredsuperscript:superscriptℎsteeredf unlearn= W(f^(l:L), steered)= W(g^(l:L% ) h^(l), steered)funlearn = italic_W ( f( l : L ) , steered ) = italic_W ( g( l : L ) ∘ h( l ) , steered ), h(l),steeredsuperscriptℎsteeredh^(l), steeredh( l ) , steered is the steered representation of the given input at layer l and Witalic_W is the unembedding matrix which maps output hidden states back to the vocabulary space. Given a forget input xF,1:nsubscript:1x_F,1:nxitalic_F , 1 : n, the logit of the next token xF,n+1subscript1x_F,n+1xitalic_F , n + 1 obtained from unlearned model funlearnsuperscriptunlearnf^unlearnfunlearn is defined as: funlearn( f^unlearn(funlearn ( xF,n+1|xF,1:n)=f(l:L),steered(xF,n+1|xF,1:n) x_F,n+1|x_F,1:n)= Wf^(l:L),steered(x_F,n+1|x_% F,1:n)xitalic_F , n + 1 | xitalic_F , 1 : n ) = italic_W f( l : L ) , steered ( xitalic_F , n + 1 | xitalic_F , 1 : n ) =⁢(g(l:L)∘h(l),steered)⁢(xF,n+1|xF,1:n)absentsuperscript:superscriptℎsteeredconditionalsubscript1subscript:1 = W(g^(l:L) h^(l), steered)(x_F,n+1|x_% F,1:n)= italic_W ( g( l : L ) ∘ h( l ) , steered ) ( xitalic_F , n + 1 | xitalic_F , 1 : n ) =⁢g(l:L)⁢(h(l),steered⁢(xF,n+1|xF,1:n))absentsuperscript:superscriptℎsteeredconditionalsubscript1subscript:1 = Wg^(l:L)(h^(l), steered(x_F,n+1|x_F,1:n))= italic_W g( l : L ) ( h( l ) , steered ( xitalic_F , n + 1 | xitalic_F , 1 : n ) ) (2) Assumption 1. A well-unlearned model shifts the representation of all tokens in a forget-sample xF,1:nsubscript:1x_F,1:nxitalic_F , 1 : n at layer l to a scaled random vector c⁢c uc italic_u. More concretely, h(l),steered⁢(xF,i)=c⁢+ϵ,superscriptℎsteeredsubscriptbold-italic-ϵ h^(l), steered(x_F,i)=c u+ ε,h( l ) , steered ( xitalic_F , i ) = c italic_u + italic_ϵ , (3) where xF,isubscriptx_F,ixitalic_F , i is the i-th token in xFsubscriptx_Fxitalic_F, ϵbold-ϵ εitalic_ϵ is a small error. Without losing generality, we assume that ϵbold-ϵ εitalic_ϵ is sampled from Normal distribution ⁢(,η⁢)0N( 0,η I)N ( 0 , η italic_I ), where η⁢η Iη italic_I is the covariance matrix, η∈ℝη η ∈ blackboard_R. Proposition 1. If Assumption 1 holds, by Definition 1, the logit value of forget token xF,n+1subscript1x_F,n+1xitalic_F , n + 1 generated by unlearned model funlearnsuperscriptunlearnf unlearnfunlearn given as funlearn⁢(xF,n+1|xF,1:n)superscriptunlearnconditionalsubscript1subscript:1f unlearn(x_F,n+1|x_F,1:n)funlearn ( xitalic_F , n + 1 | xitalic_F , 1 : n ) follows the Normal distribution ⁢(⁢g(l:L)⁢(),η⁢∇g(l:L)⁢()⊤⁢∇g(l:L)⁢()⁢⊤),superscript:subscript∇superscript:superscripttopsubscript∇superscript:superscripttopN ( Wg^(l:L)( z),η W _ zg^(l:L)(% z) _ zg^(l:L)( z) W ),N ( italic_W g( l : L ) ( italic_z ) , η italic_W ∇italic_z g( l : L ) ( italic_z )⊤ ∇italic_z g( l : L ) ( italic_z ) italic_W⊤ ) , where =c⁢ z=c uitalic_z = c italic_u. Proof. Assumption 1 implies that in a well-unlearned model, token xF,n+1subscript1x_F,n+1xitalic_F , n + 1 is independent of the previous tokens, thus we have: h(l),steered⁢(xn+1|xF,1:n)≈h(l),steered⁢(xF,n+1)=c⁢+ϵsuperscriptℎsteeredconditionalsubscript1subscript:1superscriptℎsteeredsubscript1bold-italic-ϵ h^(l), steered(x_n+1|x_F,1:n)≈ h^(l),% steered(x_F,n+1)=c u+ εh( l ) , steered ( xitalic_n + 1 | xitalic_F , 1 : n ) ≈ h( l ) , steered ( xitalic_F , n + 1 ) = c italic_u + italic_ϵ (4) Denote =c⁢ z=c uitalic_z = c italic_u. Substituting Eqn. 4 into Eqn. 2, we get: funlearn⁢(xF,n+1|xF,1:n)≈⁢g(l:L)⁢(+ϵ)superscriptunlearnconditionalsubscript1subscript:1superscript:bold-italic-ϵ f^unlearn(x_F,n+1|x_F,1:n)≈ Wg^(l:L)(% z+ ε)funlearn ( xitalic_F , n + 1 | xitalic_F , 1 : n ) ≈ italic_W g( l : L ) ( italic_z + italic_ϵ ) (5) Since ϵbold-italic-ϵ εitalic_ϵ is small, we approximate the function g(l:L)⁢(+ϵ)superscript:bold-italic-ϵg^(l:L)( z+ ε)g( l : L ) ( italic_z + italic_ϵ ) by its first-order derivative: funlearn⁢(xF,n+1|xF⁢1:n)≈⁢(g(l:L)⁢()+∇g(l:L)⁢()⊤⁢ϵ)superscriptunlearnconditionalsubscript1subscript:1superscript:subscript∇superscript:superscripttopbold-italic-ϵ f^unlearn(x_F,n+1|x_F1:n)≈ W(g^(l:L)(% z)+ _ zg^(l:L)( z) ε)funlearn ( xitalic_F , n + 1 | xitalic_F 1 : n ) ≈ italic_W ( g( l : L ) ( italic_z ) + ∇italic_z g( l : L ) ( italic_z )⊤ italic_ϵ ) (6) Given that ϵ∼⁢(,η⁢)similar-tobold-italic-ϵ0 ε ( 0,η I)italic_ϵ ∼ N ( 0 , η italic_I ), by applying the affine transformation property of the multivariate normal distribution, we get: funlearn⁢(xF,n+1|xF,1:n)superscriptunlearnconditionalsubscript1subscript:1 f unlearn(x_F,n+1|x_F,1:n)funlearn ( xitalic_F , n + 1 | xitalic_F , 1 : n ) ∼⁢(⁢g(l:L)⁢(),η⁢∇zg(l:L)⁢()⊤⁢∇zg(l:L)⁢()⁢⊤)similar-toabsentsuperscript:subscript∇superscript:superscripttopsubscript∇superscript:superscripttop ( Wg^(l:L)( z),η W _z% g^(l:L)( z) _zg^(l:L)( z) W )∼ N ( italic_W g( l : L ) ( italic_z ) , η italic_W ∇z g( l : L ) ( italic_z )⊤ ∇z g( l : L ) ( italic_z ) italic_W⊤ ) (7) Since ∼U⁢(0,1)similar-to01 u U(0,1)italic_u ∼ U ( 0 , 1 ), then ∼U⁢(0,c)similar-to0 z U(0,c)italic_z ∼ U ( 0 , c ). By definition of variance, we have: Var⁢()=Var⁢(c⁢)=c2⁢Var⁢()VarVarsuperscript2Var Var( z)= Var(c u)=c^2 Var( u)Var ( italic_z ) = Var ( c italic_u ) = c2 Var ( italic_u ). ∎ Proposition 1 suggests that the variance of funlearn⁢(xF,n+1|xF,1:n)superscriptunlearnconditionalsubscript1subscript:1f^unlearn(x_F,n+1|x_F,1:n)funlearn ( xitalic_F , n + 1 | xitalic_F , 1 : n ) is controlled by (i) η: a scalar variance and (i) ⁢∇g(l:L)⁢()⊤⁢∇g(l:L)⁢()⁢⊤subscript∇superscript:superscripttopsubscript∇superscript:superscripttop W _ zg^(l:L)( z) _ zg^(l:L)( z)% W italic_W ∇italic_z g( l : L ) ( italic_z )⊤ ∇italic_z g( l : L ) ( italic_z ) italic_W⊤: the product of ⁢∇g(L)⁢()⊤subscript∇superscriptsuperscripttop W _ zg^(L)( z) italic_W ∇italic_z g( L ) ( italic_z )⊤ and ∇g(L)⁢()⁢⊤subscript∇superscriptsuperscripttop _ zg^(L)( z) W ∇italic_z g( L ) ( italic_z ) italic_W⊤. If funlearn⁢(xF,n+1|xF,1:n)superscriptunlearnconditionalsubscript1subscript:1f^unlearn(x_F,n+1|x_F,1:n)funlearn ( xitalic_F , n + 1 | xitalic_F , 1 : n ) has high variance, the logit values are more random. Since ϵbold-italic-ϵ εitalic_ϵ presents a small error, then ϵbold-italic-ϵ εitalic_ϵ varies for different inputs xFsubscriptx_Fxitalic_F. This variation makes it difficult to control the variance of the logit by η. The main effect depend on ⁢∇g(l:L)⁢()⊤⁢∇g(l:L)⁢()⁢⊤subscript∇superscript:superscripttopsubscript∇superscript:superscripttop W _ zg^(l:L)( z) _ zg^(l:L)( z)% W italic_W ∇italic_z g( l : L ) ( italic_z )⊤ ∇italic_z g( l : L ) ( italic_z ) italic_W⊤. While the unembedding matrix Witalic_W is unchanged after unlearning, the product ∇g(l:L)⁢()⊤⁢∇g(l:L)⁢()subscript∇superscript:superscripttopsubscript∇superscript: _ zg^(l:L)( z) _ zg^(l:L)( z)∇italic_z g( l : L ) ( italic_z )⊤ ∇italic_z g( l : L ) ( italic_z ) varies depending on the specific characteristics of sub-networks g(l:L)superscript:g^(l:L)g( l : L ) and input =c⁢ z=c uitalic_z = c italic_u. Unfortunately, g(l:L)superscript:g^(l:L)g( l : L ) is a composition of transformer layers, which is highly nonlinear, making it difficult to have a complete analysis. The variance of zitalic_z, derived as Var⁢()=c2⁢Var⁢()Varsuperscript2VarVar( z)=c^2Var( u)Var ( italic_z ) = c2 Var ( italic_u ), is proportional to c; i.e. when c gets larger, the variance of zitalic_z is higher. This could increase the variability of g(l:L)⁢()superscript:g^(l:L)( z)g( l : L ) ( italic_z ) and the gradient ∇g(l:L)⁢()subscript∇superscript: _ zg^(l:L)( z)∇italic_z g( l : L ) ( italic_z ). A larger c could introduces more randomness to the logit. We conduct an empirical analysis to understand the confidence of generated tokens by RMU models in Section 4.1. 3.2 The Effect of Coefficient c on Forget-sample Representations RMU forget loss steers forget-sample representation h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) aligns with a random direction given by uitalic_u and scales the magnitude of h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) to c (Eqn 1). While vector uitalic_u is predetermined before unlearning, the magnitude of h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) varies depending on input xFsubscriptx_Fxitalic_F and specific properties of layer l. This raises the following research questions: RQ1 (Direction): “How does the coefficient c influence the alignment between h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) with uitalic_u.” RQ2 (Magnitude): “What is the optimal value of the coefficient c for effectively unlearning with different layers.” Unlearning as minimizing the noise sensitivity. We aim to answer these questions by analyzing the unlearning problem under a noise compression view. We consider the output of a transformation f(l:k)superscript:f^(l:k)f( l : k ) on input x: f(l:k)⁢(x)=(g(l:k)∘h(l))⁢(x)=g(l:k)⁢(h(l)⁢(x))superscript:superscript:superscriptℎsuperscript:superscriptℎf^(l:k)(x)=(g^(l:k) h^(l))(x)=g^(l:k) (h^(l)(x) )f( l : k ) ( x ) = ( g( l : k ) ∘ h( l ) ) ( x ) = g( l : k ) ( h( l ) ( x ) ). Suppose we compress a noise vector ξitalic_ξ to the representation h(l)superscriptℎh^(l)h( l ) of layer l at input x, then the output become g(l:k)⁢(h(l)⁢(x)+)superscript:superscriptℎg^(l:k) (h^(l)(x)+ ξ )g( l : k ) ( h( l ) ( x ) + italic_ξ ). Naturally, if layer g(l:k)superscript:g^(l:k)g( l : k ) is robust (less sensitive) to noise ξitalic_ξ, then ξitalic_ξ has a small effect on the output of g(l:k)superscript:g^(l:k)g( l : k ) i.e. the normalized squared norm Φ⁢(g(l:k),x)=‖g(l:k)⁢(h(l)⁢(x)+)−g(l:k)⁢(h(l)⁢(x))‖2‖g(l:k)⁢(h(l)⁢(x))‖2Φsuperscript:superscriptnormsuperscript:superscriptℎsuperscript:superscriptℎ2superscriptnormsuperscript:superscriptℎ2 (g^(l:k),x)= ||g^(l:k) (h^(l)(x)+ ξ% )-g^(l:k) (h^(l)(x) )||^2||g^(l:k) (h^(l)(x)% )||^2Φ ( g( l : k ) , x ) = divide start_ARG | | g( l : k ) ( h( l ) ( x ) + italic_ξ ) - g( l : k ) ( h( l ) ( x ) ) | |2 end_ARG start_ARG | | g( l : k ) ( h( l ) ( x ) ) | |2 end_ARG (8) is small. In contrast, a higher Φ⁢(g(l:k),x)Φsuperscript: (g^(l:k),x)Φ ( g( l : k ) , x ) mean g(l:k)superscript:g^(l:k)g( l : k ) is higher sensitive to noise ξitalic_ξ at input x. For a dataset forgetsubscriptforgetD_forgetDforget, we define the noise sensitivity of a layer g(l:k)superscript:g^(l:k)g( l : k ) w.r.t ξitalic_ξ on forgetsubscriptforgetD_forgetDforget as: Φ⁢(g(l:k),forget)Φsuperscript:subscriptforget (g^(l:k),D_forget)Φ ( g( l : k ) , Dforget ) =‖g(l:k)⁢(h^(l)⁢(xF)+)−g(l:k)⁢(h^(l)⁢(xF))‖2‖g(l:k)⁢(h^(l)⁢(xF))‖2,absentsuperscriptnormsuperscript:superscript^ℎsubscriptsuperscript:superscript^ℎsubscript2superscriptnormsuperscript:superscript^ℎsubscript2 = ||g^(l:k)( h^(l)(x_F)+ ξ)-g^(l:k)( % h^(l)(x_F))||^2||g^(l:k)( h^(l)(x_F))||^2,= divide start_ARG | | g( l : k ) ( over start_ARG h end_ARG( l ) ( xitalic_F ) + italic_ξ ) - g( l : k ) ( over start_ARG h end_ARG( l ) ( xitalic_F ) ) | |2 end_ARG start_ARG | | g( l : k ) ( over start_ARG h end_ARG( l ) ( xitalic_F ) ) | |2 end_ARG , (9) where h^(l)⁢(xF)superscript^ℎsubscript h^(l)(x_F)over start_ARG h end_ARG( l ) ( xitalic_F ) is the mean of h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) over xF∈forgetsubscriptsubscriptforgetx_F _forgetxitalic_F ∈ Dforget. During unlearning, RMU steers h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) for all xF∈forgetsubscriptsubscriptforgetx_F _forgetxitalic_F ∈ Dforget to the fixed vector c⁢+ϵbold-italic-ϵc u+ εc italic_u + italic_ϵ i.e. ‖g(l:k)⁢(c⁢+ϵ)−g(l:k)⁢(h^(l)⁢(xF))‖2superscriptnormsuperscript:bold-italic-ϵsuperscript:superscript^ℎsubscript2||g^(l:k)(c u+ ε)-g^(l:k)( h^(l)(x_F))||^2| | g( l : k ) ( c italic_u + italic_ϵ ) - g( l : k ) ( over start_ARG h end_ARG( l ) ( xitalic_F ) ) | |2 is minimized. If we let =c⁢+ϵ−h^(l)⁢(xF)bold-italic-ϵsuperscript^ℎsubscript ξ=c u+ ε- h^(l)(x_F)italic_ξ = c italic_u + italic_ϵ - over start_ARG h end_ARG( l ) ( xitalic_F ), we can define the unlearning problem as minimizing the noise sensitivity of the layer. This objective is described by min⁡‖g(l:k)⁢(c⁢+ϵ)−g(l:k)⁢(h^(l)⁢(xF))‖2‖g(l:k)⁢(h^(l)⁢(xF))‖2superscriptnormsuperscript:bold-italic-ϵsuperscript:superscript^ℎsubscript2superscriptnormsuperscript:superscript^ℎsubscript2 ||g^(l:k)(c u+ ε)-g^(l:k)( h^(% l)(x_F))||^2||g^(l:k)( h^(l)(x_F))||^2min divide start_ARG | | g( l : k ) ( c italic_u + italic_ϵ ) - g( l : k ) ( over start_ARG h end_ARG( l ) ( xitalic_F ) ) | |2 end_ARG start_ARG | | g( l : k ) ( over start_ARG h end_ARG( l ) ( xitalic_F ) ) | |2 end_ARG (10) While g(l:k)superscript:g^(l:k)g( l : k ) is a composition of transformer layers, which is hard to expand it in term of c. Therefore, we propose to use the Jacobian matrix (l:k)⁢(xF)superscript:subscript J^(l:k)(x_F)italic_J( l : k ) ( xitalic_F )—a linearized of g(l:k)superscript:g^(l:k)g( l : k ) at xFsubscriptx_Fxitalic_F—which describes the change in the output of g(l:k)superscript:g^(l:k)g( l : k ) due to a noise perturbed in the input h^(l)⁢(xF)superscript^ℎsubscript h^(l)(x_F)over start_ARG h end_ARG( l ) ( xitalic_F ). For simplification, we write h^(l)superscript^ℎ h^(l)over start_ARG h end_ARG( l ), (l:k)superscript: J^(l:k)italic_J( l : k ) instead of h^(l)⁢(xF)superscript^ℎsubscript h^(l)(x_F)over start_ARG h end_ARG( l ) ( xitalic_F ), (l:k)⁢(xF)superscript:subscript J^(l:k)(x_F)italic_J( l : k ) ( xitalic_F ) respectively. The objective becomes min⁡‖(l:k)⁢(c⁢+ϵ)−(l:k)⁢h^(l)‖2‖(l:k)⁢h^(l)‖2superscriptnormsuperscript:bold-italic-ϵsuperscript:superscript^ℎ2superscriptnormsuperscript:superscript^ℎ2 || J^(l:k)(c u+ ε)- J^(l:k)% h^(l)||^2|| J^(l:k) h^(l)||^2min divide start_ARG | | italic_J( l : k ) ( c italic_u + italic_ϵ ) - italic_J( l : k ) over start_ARG h end_ARG( l ) | |2 end_ARG start_ARG | | italic_J( l : k ) over start_ARG h end_ARG( l ) | |2 end_ARG (11) Since (l:k)superscript: J^(l:k)italic_J( l : k ) is a linear transformation, then ‖(l:k)⁢(c⁢+ϵ)−(l:k)⁢h^(l)‖2=‖(l:k)⁢(c⁢+ϵ−h^(l))‖2superscriptnormsuperscript:bold-italic-ϵsuperscript:superscript^ℎ2superscriptnormsuperscript:bold-italic-ϵsuperscript^ℎ2 || J^(l:k)(c u+ ε)- J^(l:k) h^(l% )||^2=|| J^(l:k)(c u+ ε- h^(l))||^2| | italic_J( l : k ) ( c italic_u + italic_ϵ ) - italic_J( l : k ) over start_ARG h end_ARG( l ) | |2 = | | italic_J( l : k ) ( c italic_u + italic_ϵ - over start_ARG h end_ARG( l ) ) | |2 (12) Let =ϵ−h^(l)bold-italic-ϵsuperscript^ℎ v= ε- h^(l)italic_v = italic_ϵ - over start_ARG h end_ARG( l ). By definition of the squared norm, we have: ‖(l:k)⁢(c⁢+)‖2superscriptnormsuperscript:2 || J^(l:k)(c u+ v)||^2| | italic_J( l : k ) ( c italic_u + italic_v ) | |2 =((l:k)⁢(c⁢+))⊤⁢(l:k)⁢(c⁢+)absentsuperscriptsuperscript:topsuperscript: =( J^(l:k)(c u+ v)) J^(l:k)(c u+% v)= ( italic_J( l : k ) ( c italic_u + italic_v ) )⊤ italic_J( l : k ) ( c italic_u + italic_v ) =(c⁢+)⊤⁢(l:k)⁣⊤⁢(l:k)⁢(c⁢+)absentsuperscripttopsuperscript:topsuperscript: =(c u+ v) J^(l:k) J^(l:k)(c u+% v)= ( c italic_u + italic_v )⊤ italic_J( l : k ) ⊤ italic_J( l : k ) ( c italic_u + italic_v ) (13) Let matrix =(l:k)⁣⊤⁢(l:k)superscript:topsuperscript: A= J^(l:k) J^(l:k)italic_A = italic_J( l : k ) ⊤ italic_J( l : k ). Expand the right-hand side of Eqn. 13, we get: ‖(l:k)⁢(c⁢+)‖2superscriptnormsuperscript:2 || J^(l:k)(c u+ v)||^2| | italic_J( l : k ) ( c italic_u + italic_v ) | |2 =(c⁢)⊤⁢c⁢+(c⁢)⊤⁢+⊤⁢c⁢+⊤⁢absentsuperscripttopsuperscripttopsuperscripttopsuperscripttop =(c u) Ac u+(c u) A v+ v% Ac u+ v A v= ( c italic_u )⊤ italic_A c italic_u + ( c italic_u )⊤ italic_A italic_v + italic_v⊤ italic_A c italic_u + italic_v⊤ italic_A italic_v (14) Since Aitalic_A is a symmetric matrix (i.e. ⊤=superscripttop A = Aitalic_A⊤ = italic_A), then (c⁢)⊤⁢=(c⁢)⊤⁢⊤⁢=(⁢c⁢)⊤⁢=⊤⁢c⁢superscripttopsuperscripttopsuperscripttopsuperscripttopsuperscripttop (c u) A v=(c u) A v=% ( Ac u) v= v Ac u( c italic_u )⊤ italic_A italic_v = ( c italic_u )⊤ italic_A⊤ italic_v = ( italic_A c italic_u )⊤ italic_v = italic_v⊤ italic_A c italic_u (15) Substituting (c⁢)⊤⁢=⊤⁢c⁢superscripttopsuperscripttop(c u) A v= v Ac u( c italic_u )⊤ italic_A italic_v = italic_v⊤ italic_A c italic_u into Eqn. 14 we get: ‖(l:k)⁢(c⁢+)‖2=c2⁢⊤⁢+2⁢c⁢⊤⁢+⊤⁢superscriptnormsuperscript:2superscript2superscripttop2superscripttopsuperscripttop || J^(l:k)(c u+ v)||^2=c^2 u A% u+2c u A v+ v A v| | italic_J( l : k ) ( c italic_u + italic_v ) | |2 = c2 italic_u⊤ italic_A italic_u + 2 c italic_u⊤ italic_A italic_v + italic_v⊤ italic_A italic_v (16) Substituting Eqn. 16 into Eqn. 11, the objective becomes min⁡c2⁢⊤⁢+2⁢c⁢⊤⁢+⊤⁢‖(l:k)⁢h^(l)‖2superscript2superscripttop2superscripttopsuperscripttopsuperscriptnormsuperscript:superscript^ℎ2 c^2 u A u+2c u A % v+ v A v|| J^(l:k) h^(l)||^2min divide start_ARG c2 italic_u⊤ italic_A italic_u + 2 c italic_u⊤ italic_A italic_v + italic_v⊤ italic_A italic_v end_ARG start_ARG | | italic_J( l : k ) over start_ARG h end_ARG( l ) | |2 end_ARG (17) Taking its derivative w.r.t c and set it to zero: 2⁢⊤⁢c+2⁢⊤⁢‖(l:k)⁢h^(l)‖2=02superscripttop2superscripttopsuperscriptnormsuperscript:superscript^ℎ20 2 u A uc+2 u A v||% J^(l:k) h^(l)||^2=0divide start_ARG 2 italic_u⊤ italic_A italic_u c + 2 italic_u⊤ italic_A italic_v end_ARG start_ARG | | italic_J( l : k ) over start_ARG h end_ARG( l ) | |2 end_ARG = 0 (18) Since ‖(l:k)⁢h^(l)‖2superscriptnormsuperscript:superscript^ℎ2|| J^(l:k) h^(l)||^2| | italic_J( l : k ) over start_ARG h end_ARG( l ) | |2 is not zero, solve for c: c c =−⊤⁢⊤⁢=⊤⁢(l:k)⁣⊤⁢(l:k)⁢(h^(l)−ϵ)⊤⁢(l:k)⁣⊤⁢(l:k)⁢absentsuperscripttopsuperscripttopsuperscripttopsuperscript:topsuperscript:superscript^ℎbold-italic-ϵsuperscripttopsuperscript:topsuperscript: =- u A v u A u=% u J^(l:k) J^(l:k)( h^(l)- ε% ) u J^(l:k) J^(l:k) u= - divide start_ARG italic_u⊤ italic_A italic_v end_ARG start_ARG italic_u⊤ italic_A italic_u end_ARG = divide start_ARG italic_u⊤ italic_J( l : k ) ⊤ italic_J( l : k ) ( over start_ARG h end_ARG( l ) - italic_ϵ ) end_ARG start_ARG italic_u⊤ italic_J( l : k ) ⊤ italic_J( l : k ) italic_u end_ARG =((l:k)⁢)⊤⁢(l:k)⁢(h^(l)−ϵ)‖(l:k)⁢‖2absentsuperscriptsuperscript:topsuperscript:superscript^ℎbold-italic-ϵsuperscriptnormsuperscript:2 = ( J^(l:k) u) J^(l:k)( h^(l)-% ε)|| J^(l:k) u||^2= divide start_ARG ( italic_J( l : k ) italic_u )⊤ italic_J( l : k ) ( over start_ARG h end_ARG( l ) - italic_ϵ ) end_ARG start_ARG | | italic_J( l : k ) italic_u | |2 end_ARG =||(l:k)(h^(l)−ϵ))||‖(l:k)⁢‖⁢cos⁡((l:k)⁢,(l:k)⁢(h^(l)−ϵ)) = || J^(l:k)( h^(l)- ε))|||| J% ^(l:k) u|| ( J^(l:k) u, J^(l:k)( h^(l)- % ε))= divide start_ARG | | italic_J( l : k ) ( over start_ARG h end_ARG( l ) - italic_ϵ ) ) | | end_ARG start_ARG | | italic_J( l : k ) italic_u | | end_ARG cos ( start_ARG italic_J( l : k ) italic_u , italic_J( l : k ) ( over start_ARG h end_ARG( l ) - italic_ϵ ) end_ARG ) (19) Since ‖(l:k)⁢(h^(l)−ϵ)‖(l:k)⁢‖normsuperscript:superscript^ℎbold-italic-ϵnormsuperscript: || J^(l:k)( h^(l)- ε)|||| J^(l:k) u||divide start_ARG | | italic_J( l : k ) ( over start_ARG h end_ARG( l ) - italic_ϵ ) | | end_ARG start_ARG | | italic_J( l : k ) italic_u | | end_ARG is positive, then c and cos⁡((l:k)⁢,(l:k)⁢(h^(l)−ϵ))superscript:superscript:superscript^ℎbold-ϵ ( J^(l:k) u, J^(l:k)( h^(l)- ε))cos ( start_ARG italic_J( l : k ) italic_u , italic_J( l : k ) ( over start_ARG h end_ARG( l ) - italic_ϵ ) end_ARG ) are positively correlated. This means smaller (larger) c indicates less (more) alignment between (l:k)⁢superscript: J^(l:k) uitalic_J( l : k ) italic_u and (l:k)⁢(h^(l)−ϵ)superscript:superscript^ℎbold-italic-ϵ J^(l:k)( h^(l)- ε)italic_J( l : k ) ( over start_ARG h end_ARG( l ) - italic_ϵ ). Given that the Jacobian (l:k)superscript: J^(l:k)italic_J( l : k ) describes how small changes in the input lead to changes in the output using linear approximation around a given point. If (l:k)superscript: J^(l:k)italic_J( l : k ) does not vary drastically, it will not significantly alter the directions of uitalic_u and h^(l)−ϵsuperscript^ℎbold-italic-ϵ h^(l)- εover start_ARG h end_ARG( l ) - italic_ϵ. In such cases, (l:k)superscript: J^(l:k)italic_J( l : k ) will have a small effect on directional alignment, preserving the relative angles between uitalic_u and h^(l)−ϵsuperscript^ℎbold-italic-ϵ h^(l)- εover start_ARG h end_ARG( l ) - italic_ϵ. Here, reasonably, uitalic_u and h^(l)superscript^ℎ h^(l)over start_ARG h end_ARG( l ) are becoming more aligned as c increases since error ϵ→bold-italic-ϵ0 ε→ 0italic_ϵ → 0 as unlearning becomes more accurate. Figure 1: Noise sensitivity of layer g(l:k)superscript:g^(l:k)g( l : k ), for k∈[3⁢…⁢31]delimited-[]3…31k∈[3...31]k ∈ [ 3 … 31 ] in base Zephyr-7B, base Llama-3-8B, base Mistral-7B, and RMU Zephyr-7B model. In the base models, a deeper layer has lower noise sensitivity, while the noise sensitivity is minimized in the RMU model (compress noise into h(7)superscriptℎ7h^(7)h( 7 ), the noise sensitivity of layer k=88k=8k = 8 is minimized). The above discussion does not directly address RQ2. However, the definition of the noise sensitivity suggests that the noise sensitivity of layer g(l:k)superscript:g^(l:k)g( l : k ) is characterized by the inherent properties of g(l:k)superscript:g^(l:k)g( l : k ), the representation h^(l)⁢(xF)superscript^ℎsubscript h^(l)(x_F)over start_ARG h end_ARG( l ) ( xitalic_F ) (which is fixed) and the perturbed noise ξitalic_ξ. If ξitalic_ξ is predetermined, the noise sensitivity of g(l:k)superscript:g^(l:k)g( l : k ) depends solely on its properties. This suggest the following experiment: we compute h^(l)⁢(xF)superscript^ℎsubscript h^(l)(x_F)over start_ARG h end_ARG( l ) ( xitalic_F )—the mean of h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) over a set of input xF∈forgetsubscriptsubscriptforgetx_F _forgetxitalic_F ∈ Dforget, compress a fix perturbed noise ξitalic_ξ into h^(l)⁢(xF)superscript^ℎsubscript h^(l)(x_F)over start_ARG h end_ARG( l ) ( xitalic_F ). We then calculate the noise sensitivity of g(l:k)superscript:g^(l:k)g( l : k ) for different layers. Fig. 1 shows the noise sensitivity of layers across different models. We empirically observed that: the noise sensitivity decreases as layers go deeper and vary across different models. Since noise sensitivity describes a layer’s robustness to noise, higher noise sensitivity means g(l:k)superscript:g^(l:k)g( l : k ) requires smaller noise to produce the same level of output randomness, while lower noise sensitivity means it requires larger noise. In other words, early layers require smaller noise ξitalic_ξ (smaller c) whereas later layers require larger noise ξitalic_ξ (larger c). We present an empirical experiment to verify our analysis in Section. 4.3. 3.3 Robustness of RMU Models to Adversarial Jailbreak Attacks (a) c=1.01.0c=1.0c = 1.0 (b) c=2.02.0c=2.0c = 2.0 (c) c=5.05.0c=5.0c = 5.0 (d) c=10.010.0c=10.0c = 10.0 (e) c=1.01.0c=1.0c = 1.0 (f) c=2.02.0c=2.0c = 2.0 (g) c=5.05.0c=5.0c = 5.0 (h) c=10.010.0c=10.0c = 10.0 Figure 2: The distribution of MaxLogit (a-d) on WMDP Q&A sets with different coefficient c of the base Zephyr-7B and RMU Zephyr-7B models (l=77l=7l = 7). The distribution of cos⁡(,h(l))superscriptℎ ( u,h^(l))cos ( start_ARG italic_u , h( l ) end_ARG ) (e-h) of the RMU Zephyr-7B model (l=77l=7l = 7). RMU algorithms not only erase undesirable knowledge (forgotten knowledge) but also prevent it from resurfacing when attacked by adversarial jailbreak attacks. Li et al. (2024b) present a qualitvative experiment by employing Greedy Coordinate Gradient (GCG; Zou et al. (2023b)) to jailbreak RMU models to extract the answer given a harmful input. As a result, RMU models demonstrate strong robustness against GCG attack. We conduct an analysis from the standpoint of an attack-defense game to explain the underlying causes for the effectiveness of the RMU algorithm. Attacker’s goal. The goal of the attacker is to inject an adversarial trigger into the suffix of the prompt input (e.g., “How would I engineer influenza A to be significantly more virulent? x x x x x x), causing the model generates a harmful response (e.g., “To make influenza A more virulent…”). Attacker’s knowledge and capability. In this setting, we focus on white-box jailbreak attack scenarios (Zou et al. 2023b), where the victim model’s architecture, model input, and output are exposed to the attacker. The attacker is based on gradient signals to search and inject an adversarial trigger into the prompt input, and supplies this adversarial input to the model. Problem formulation. Let f:ℝn×d↦ℝn×|V|:maps-tosuperscriptℝsuperscriptℝf:R^n× d ^n×|V|f : blackboard_Rn × d ↦ blackboard_Rn × | V | be an autoregressive LLM. Given a prompt input joint with an adversarial trigger xF,1:nsubscript:1x_F,1:nxitalic_F , 1 : n, the attacker finds an update δ to adversarial trigger aims to maximize the likelihood of generating the target sequence xF,n+1|n+Ksubscriptconditional1x_F,n+1|n+Kxitalic_F , n + 1 | n + K consists of K tokens. For simplification, we denote xF=xF,1:K=[xF,1:n,xF,n+1:n+K]subscriptsubscript:1subscript:1subscript:1x_F=x_F,1:K=[x_F,1:n,x_F,n+1:n+K]xitalic_F = xitalic_F , 1 : K = [ xitalic_F , 1 : n , xitalic_F , n + 1 : n + K ]. The attacker tries to solve the following objective: minxF+δ⁡⁢(f⁢(xF+δ)),subscriptsubscriptsubscript _x_F+δJ(f(x_F+δ)),minitalic_x start_POSTSUBSCRIPT F + δ end_POSTSUBSCRIPT J ( f ( xitalic_F + δ ) ) , (20) where ⁢(⋅,⋅)⋅J(·,·)J ( ⋅ , ⋅ ) is the loss function of the attacker. The attacker finds an update δ based on the linearized approximation of the loss ∇exi⁢(f⁢(xF))subscript∇subscriptsubscriptsubscript _e_x_iJ(f(x_F))∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT J ( f ( xitalic_F ) ), where exisubscriptsubscripte_x_ieitalic_x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT is the one-hot vector representing the current value of the i-th token in xFsubscriptx_Fxitalic_F. The gradient ∇exi⁢(f⁢(xF))subscript∇subscriptsubscriptsubscript _e_x_iJ(f(x_F))∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT J ( f ( xitalic_F ) ) is a good indicator for finding a set of candidates for the adversarial token replacement. A more negative value of the gradient ∇exi⁢(f⁢(xF))subscript∇subscriptsubscriptsubscript _e_x_iJ(f(x_F))∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT J ( f ( xitalic_F ) ) makes a more decrease in the loss. The GCG attacker finds top-k largest negative value of ∇exi⁢(f⁢(xF))subscript∇subscriptsubscriptsubscript _e_x_iJ(f(x_F))∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT J ( f ( xitalic_F ) ) for each token in the adversarial trigger and makes the replacement the most decrease in the loss. Robustness of RMU models against GCG attack. We show that the GCG attacker misjudges in finding optimal adversarial token substitution in RMU models. Specifically, the gradient of the loss at input xFsubscriptx_Fxitalic_F with respect to exisubscriptsubscripte_x_ieitalic_x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT in RMU model is ∇exi⁢(funlearn⁢(xF))subscript∇subscriptsubscriptsuperscriptunlearnsubscript _e_x_iJ(f^unlearn(x_F))∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT J ( funlearn ( xitalic_F ) ) (21) Given the Assumption 1, we have ∇exi⁢(funlearn⁢(xF))subscript∇subscriptsubscriptsuperscriptunlearnsubscript _e_x_iJ(f^unlearn(x_F))∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT J ( funlearn ( xitalic_F ) ) =∇exi(g(l:k)(h(l),steered(xF)) = _e_x_iJ(g^(l:k)(h^(l), % steered(x_F))= ∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT J ( g( l : k ) ( h( l ) , steered ( xitalic_F ) ) ≈∇exi(∘g(l:k))⁡(c⁢+ϵ)absentsubscript∇subscriptsubscriptsuperscript:bold-italic-ϵ ≈ _e_x_i(J g^(l:k))(c u+ % ε)≈ ∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( J ∘ g( l : k ) ) ( c italic_u + italic_ϵ ) (22) Since c and uitalic_u are predetermined before unlearning, (∘g(l:k))⁢(c⁢)superscript:(J g^(l:k))(c u)( J ∘ g( l : k ) ) ( c italic_u ) does not change with respect to exisubscriptsubscripte_x_ieitalic_x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT. The gradient ∇exi(∘g(l:k))⁡(c⁢+ϵ)subscript∇subscriptsubscriptsuperscript:bold-italic-ϵ _e_x_i(J g^(l:k))(c u+ ε)∇e start_POSTSUBSCRIPT x start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( J ∘ g( l : k ) ) ( c italic_u + italic_ϵ ) close to 00 for all token xisubscriptx_ixitalic_i since the error ϵ→bold-italic-ϵ0 ε→ 0italic_ϵ → 0 as unlearning becomes accurate. This means the GCG attacker received unreliable, uninformative gradient signals from RMU models. The RMU model serves as a defender by causing the attacker to miscalculate the gradient of the loss to optimize its objective, thereby increasing the attacker’s cost. The attacker, therefore, cannot find the optimal adversarial tokens for replacement. Li et al. (2024b)’s experiment results implicitly verify our analysis. 4 Empirical Analysis 4.1 Measuring Token Confidence with MaxLogit As discussed in Section 3.1, we validate our hypothesis by considering the Maximum Logit Value (MaxLogit) estimator for measuring the token confidence. More specifically, we compute the MaxLogit for each token xn+1subscript1x_n+1xitalic_n + 1 given a sequence of tokens x1:n=x1,…,xnsubscript:1subscript1…subscriptx_1:n=\x_1,...,x_n\x1 : n = x1 , … , xitalic_n from vocabulary V as: MaxLogit⁢(xn+1)=maxxn+1∈V⁡funlearn⁢(xn+1|x1:n)MaxLogitsubscript1subscriptsubscript1superscriptunlearnconditionalsubscript1subscript:1 (x_n+1)= _x_n+1∈ Vf^unlearn(x_% n+1|x_1:n)MaxLogit ( xitalic_n + 1 ) = maxitalic_x start_POSTSUBSCRIPT n + 1 ∈ V end_POSTSUBSCRIPT funlearn ( xitalic_n + 1 | x1 : n ) (23) We use WMDP-Biology and WMDP-Cyber Q&A datasets (Li et al. 2024b) with total 3260326032603260 Q&As. We formulated each question and answer as a zero-shot Q&A prompt to query the unlearned LLM. The details of the prompt template are located in Appendix A.1. We used greedy decoding to generate tokens and compute the MaxLogit of each token over k=3030k=30k = 30 generated tokens. The MaxLogit distribution was then analyzed for each model Base vs. RMU (unlearned on WMDP-Biology and WMDP-Cyber forget datasets). The results are presented in Fig. 2 (a)-(d). We find that the MaxLogit distribution for the base model is generally wider compared to the RMU model. In contrast, the RMU model demonstrates a more concentrated and approximately normal distribution of MaxLogit values. The peak of the RMU model’s MaxLogit distribution is shifted towards lower values relative to the base model. This indicates that the RMU model tends to assign lower confidence scores to the generated tokens. Overall, the RMU model’s MaxLogit distribution exhibits lower compared to the base model. 4.2 The Effect of the Coefficient c On accuracy. We analyze the impact of c for forgotten knowledge and retained knowledge, using WMDP (Li et al. 2024b) and MMLU (Hendrycks et al. 2021). See Section 6 for the full experiment setting. Fig. 3a shows: (i) a clear positive correlation between the drop-in-accuracy rate and the value of c, i.e. higher c makes the accuracy decrease faster. (i) A larger value of c tends to make a more drop-in-accuracy on WMDP. (i) However, a larger c comes with a caveat in a significant drop in general performance on MMLU (Fig. 3b). On alignment between uitalic_u and h(l)superscriptℎh^(l)h( l ). We compute cos⁡(,h(l))superscriptℎ ( u,h^(l))cos ( start_ARG italic_u , h( l ) end_ARG ) scores of pairs of uitalic_u and h(l)⁢(xF)superscriptℎsubscripth^(l)(x_F)h( l ) ( xitalic_F ) for all xFsubscriptx_Fxitalic_F in on WMDP-Biology and WMDP-Cyber forget datasets and plot the cos⁡(,h(l))superscriptℎ ( u,h^(l))cos ( start_ARG italic_u , h( l ) end_ARG ) score distribution shown in Fig. 2(e)-(h). We observed that there is a clear positive correlation between cos⁡(,h(l))superscriptℎ ( u,h^(l))cos ( start_ARG italic_u , h( l ) end_ARG ) scores and the coefficient c. As c increases, the distribution of cos⁡(,h(l))superscriptℎ ( u,h^(l))cos ( start_ARG italic_u , h( l ) end_ARG ) scores shifts towards higher values and are almost distributed with a peak at 1.01.01.01.0 (Fig. 2(g)-(h)). This verify our analysis in Section 3.2. Figure 3: Average accuracy of WMDP (Biology and Cyber) (left) and MMLU with different coefficient c (right). 4.3 The Effect of Layers on Unlearning Figure 4: ℓ2superscriptℓ2 ^2ℓ2-norm of forget-sample representation. We investigate the effect of unlearn layers on accuracy and the representation norm during unlearning. Following original work, we change the unlearn layer l from 3→31→3313→ 313 → 31, fixed c=6.56.5c=6.5c = 6.5. Fig. 5 shows that RMU is effective for unlearning within the early layers (3→10→3103→ 103 → 10), yet exhibits inefficacy within middle and later layers (11→31→113111→ 3111 → 31). Interestingly, in Fig. 4, we observed that within early layers, the ℓ2superscriptℓ2 ^2ℓ2-norm of forget samples are smaller than the coefficient c. During unlearning, the representation norm exponentially increases, approaching c, thereby facilitating the convergence of forget loss. Conversely, within middle and later layers, the representation norms of forget samples, initially larger than c, remain unchanged during unlearning, making the forget loss non-convergence. 5 Adaptive RMU Algorithm 1 Adaptive RMU pseudocode 0: 1: forgetsubscriptforgetD_forgetDforget: a forget dataset. 2: retainsubscriptretainD_retainDretain: a retain dataset. 3: fθfrozensubscriptsuperscriptfrozenf_θ^frozenfitalic_θfrozen: a frozen model. 4: fθunlearnsubscriptsuperscriptunlearnf_θ^unlearnfitalic_θunlearn: an update model. 5: α: a retain weight. 6: l: an unlearn layer. 7: β: a scaling factor. 8: T: number of gradient update steps. 8: Return the unlearned model fθunlearnsubscriptsuperscriptunlearnf_θ^unlearnfitalic_θunlearn. 9: Sample a random unit vector ∼U⁢(0,1)similar-to01 u U(0,1)italic_u ∼ U ( 0 , 1 ) 10: for step t∈[1⁢…⁢T]:xF∈forget:delimited-[]1…subscriptsubscriptforgett∈[1...T]:x_F _ forgett ∈ [ 1 … T ] : xitalic_F ∈ Dforget, xR∈retainsubscriptsubscriptretainx_R _ retainxitalic_R ∈ Dretain do 11: Get the representations of xFsubscriptx_Fxitalic_F and xRsubscriptx_Rxitalic_R from the frozen and update model. 12: Compute the adaptive loss ℒadaptivesuperscriptℒadaptiveL adaptiveLadaptive by Eqn. 24. 13: Update θunlearnsuperscriptunlearnθ^unlearnθunlearn w.r.t ∇ℒadap∇superscriptℒadap adap∇ Ladap using gradient descent. 14: t=t+11t=t+1t = t + 1 15: end for 16: return fθunlearnsubscriptsuperscriptunlearnf_θ^unlearnfitalic_θunlearn Figure 5: Q&A accuracy of RMU and Adaptive RMU Zephyr-7B models on WMDP-Biology, WMDP-Cyber, and MMLU w.r.t unlearn layer l from the third to the last layer. Inspired by the observations in Section 4.3, we propose Adaptive RMU, a simple yet effective alternative method with an adaptive forget loss by scaling the random unit vector uitalic_u with an adaptive scaling coefficient β⁢‖hθfrozen(l)⁢(xF)‖normsuperscriptsubscriptℎsuperscriptfrozensubscriptβ||h_θ^frozen^(l)(x_F)||β | | hitalic_θfrozen( l ) ( xitalic_F ) | |, where β∈ℝβ β ∈ blackboard_R is a scaling factor and ‖hθfrozen(l)⁢(xF)‖normsuperscriptsubscriptℎsuperscriptfrozensubscript||h_θ^frozen^(l)(x_F)||| | hitalic_θfrozen( l ) ( xitalic_F ) | | is the ℓ2superscriptℓ2 ^2ℓ2-norm of forget-sample xFsubscriptx_Fxitalic_F on model fθfrozensubscriptsuperscriptfrozenf_θ^frozenfitalic_θfrozen. The total loss is calculated as follows: ℒadaptivesuperscriptℒadaptive adaptiveLadaptive =xF∈forget⁢‖hθunlearn(l)⁢(xF)−β‖⁢hθfrozen(l)⁢(xF)⁢‖22⏟adaptive forget lossabsentsubscript⏟subscriptsubscriptsubscriptforgetnormsuperscriptsubscriptℎsuperscriptunlearnsubscriptsuperscriptsubscriptℎsuperscriptfrozensubscriptsuperscriptsubscriptnorm22adaptive forget loss = E_x_F _forget||h_% θ^unlearn^(l)(x_F)-β||h_θ^frozen^(l)% (x_F)|| u||_2^2_ adaptive forget loss= under⏟ start_ARG blackboard_Ex start_POSTSUBSCRIPT F ∈ Dforget end_POSTSUBSCRIPT | | hitalic_θunlearn( l ) ( xitalic_F ) - β | | hitalic_θfrozen( l ) ( xitalic_F ) | | italic_u | |22 end_ARGadaptive forget loss +α⁢xR∈retain⁢‖hθunlearn(l)⁢(xR)−hθfrozen(l)⁢(xR)‖22⏟retain losssubscript⏟subscriptsubscriptsubscriptretainsuperscriptsubscriptnormsuperscriptsubscriptℎsuperscriptunlearnsubscriptsuperscriptsubscriptℎsuperscriptfrozensubscript22retain loss +α E_x_R _retain% ||h_θ^unlearn^(l)(x_R)-h_θ^frozen^(l)(% x_R)||_2^2_ retain loss+ α under⏟ start_ARG blackboard_Ex start_POSTSUBSCRIPT R ∈ Dretain end_POSTSUBSCRIPT | | hitalic_θunlearn( l ) ( xitalic_R ) - hitalic_θfrozen( l ) ( xitalic_R ) | |22 end_ARGretain loss (24) Our Adaptive RMU is shown in Algorithm 1. We note that Adaptive RMU aims to address the challenge of adaptively determining the coefficient c in RMU. We acknowledge that the introduced value β is manually tuned via grid search, leaving the challenge to not fully resolved. However, we emphasize that Adaptive RMU offers significant computational advantages over the original RMU. More concretely, in RMU, grid search is conducted over both c and layer l for l∈[1⁢…⁢L]delimited-[]1…l∈[1...L]l ∈ [ 1 … L ], where L is the number of layers. Our analysis suggests that effective unlearning can be achieved when c is higher than the representation norm of forget-samples. Therefore, given a layer l, Adaptive RMU only requires tuning β, which is L times less than that of RMU. This reduction in computational overhead represents a significant improvement when the size of modern deep networks grows. 6 Experiment Datasets. We use WMDP-Biology and WMDP-Cyber forget datasets as forgetsubscriptforgetD_forgetDforget and Wikitext (Merity et al. 2022) as retainsubscriptretainD_retainDretain for unlearning the LLM. Unlearned models are evaluated on WMDP Q&A datasets and MMLU (Hendrycks et al. 2021). Details of the datasets can be found in the Appendix A.1. Models. We use the following LLMs: Zephyr-7B-β (Tunstall et al. 2023), Yi-6B (Young et al. 2024), Meta Llama-3-8B (Meta 2024), and Mistral-7B (Jiang et al. 2023). Experimental setup. Models were fine-tuned using AdamW (Loshchilov and Hutter 2019) with learning rate η=5⁢e−555η=5e-5η = 5 e - 5, batch-size of 4444, max sequence len of 512512512512 for WMDP-Biology and 768768768768 for WMDP-Cyber, with T=500500T=500T = 500 gradient update steps. The retain weight α=12001200α=1200α = 1200. For the baseline RMU, we follow the previous work and let c=6.56.5c=6.5c = 6.5. We grid search for unlearn layer l from the third to the last layer. For the Adaptive RMU, we grid search for the scaling factor β∈2,3,5,1023510β∈\2,3,5,10\β ∈ 2 , 3 , 5 , 10 . We report the performances of Adaptive RMU models with β=55β=5β = 5. We update three layers parameters l,l−1,l−212\l,l-1,l-2\ l , l - 1 , l - 2 of the model. Two NVIDIA A40s with 90GB GPU were used to run the experiments. Our code is available at https://github.com/RebelsNLU-jaist/llm-unlearning. Baselines. We compare Adaptive RMU against baselines: RMU (Li et al. 2024b), Large Language Model Unlearning (LLMU; Yao, Xu, and Liu (2023)), SCalable Remenbering and Unlearning unBound (SCRUB; Kurmanji et al. (2023)), and Selective Synaptic Dampening (SSD; Foster, Schoepf, and Brintrup (2024). We use off-the-shelf results from Li et al. (2024b) for LLMU, SCRUB, and SSD. Main results. Method/tasks WMDP-Biology ↓ ↓ WMDP-Cyber ↓ ↓ MMLU ↑ ↑ Base 63.7 43.5 58.1 LLMU 59.5 39.5 44.7 SCRUB 43.8 39.3 51.2 SSD 50.2 35.0 40.7 RMU (l=77l=7l = 7) 28.8 28.8 56.8 Adaptive RMU (l=77l=7l = 7) 23.7 26.5 55.0 Table 1: Q&A accuracy of Zephyr-7B models on WMDP and MMLU. The best and runner up¯runner up runner upunder¯ start_ARG runner up end_ARG are marked. Fig. 5 shows that Adaptive RMU significantly improves unlearning performances. Specifically, Adaptive RMU reduces average accuracy by 13.1%percent13.113.1\%13.1 % on WMDP-Biology and 3.6%percent3.63.6\%3.6 % on WMDP-Cyber within early layers (3→10→3103→ 103 → 10), and by 15.6%percent15.615.6\%15.6 % on WMDP-Biology and 9.6%percent9.69.6\%9.6 % on WMDP-Cyber within middle and later layers (11→31→113111→ 3111 → 31). This corresponds to an overall enhancement of 14.3%percent14.314.3\%14.3 % and 6.6%percent6.66.6\%6.6 % in drop-in-accuracy for the WMDP-Biology and WMDP-Cyber, respectively. Table 1 further highlights that Adaptive RMU (l=77l=7l = 7) outperforms RMU (l=77l=7l = 7), LLMU, SCRUB, and SSD, establishing a new state-of-the-art performance. We defer the full results on other models and settings in Appendix B. 7 Conclusion We studied the effect of steering latent representation for LLM unlearning and explored its connection to jailbreak adversarial robustness. We developed a simple yet effective alternative method that enhances unlearning performance across most layers while maintaining overall model utility. Our findings illuminate the explanation of the RMU method and pave the way for future research in LLM unlearning. Acknowledgments This work was supported by JST FOREST Program (Grant Number JPMJFR232K, Japan) and the Nakajima Foundation. References Belrose et al. (2023) Belrose, N.; Schneider-Joseph, D.; Ravfogel, S.; Cotterell, R.; Raff, E.; and Biderman, S. 2023. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems. Bourtoule et al. (2021) Bourtoule, L.; Chandrasekaran, V.; Choquette-Choo, C. A.; Jia, H.; Travers, A.; Zhang, B.; Lie, D.; and Papernot, N. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), 141–159. IEEE. Bui et al. (2024) Bui, T.-A.; Long, V.; Doan, K.; Le, T.; Montague, P.; Abraham, T.; and Phung, D. 2024. Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation. NeurIPS 2024. Cao and Yang (2015) Cao, Y.; and Yang, J. 2015. Towards Making Systems Forget with Machine Unlearning. In 2015 IEEE Symposium on Security and Privacy, 463–480. Cha et al. (2024) Cha, S.; Cho, S.; Hwang, D.; Lee, H.; Moon, T.; and Lee, M. 2024. Learning to unlearn: Instance-wise unlearning for pre-trained classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 11186–11194. Che et al. (2023) Che, T.; Zhou, Y.; Zhang, Z.; Lyu, L.; Liu, J.; Yan, D.; Dou, D.; and Huan, J. 2023. Fast federated machine unlearning with nonlinear functional theory. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. Chen et al. (2024) Chen, C.; Zhang, Y.; Li, Y.; Wang, J.; Qi, L.; Xu, X.; Zheng, X.; and Yin, J. 2024. Post-Training Attribute Unlearning in Recommender Systems. ACM Trans. Inf. Syst. Just Accepted. Chen et al. (2022) Chen, M.; Zhang, Z.; Wang, T.; Backes, M.; Humbert, M.; and Zhang, Y. 2022. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC conference on computer and communications security, 499–513. Cheng and Amiri (2023) Cheng, J.; and Amiri, H. 2023. Multimodal machine unlearning. arXiv preprint arXiv:2311.12047. Cheng et al. (2023) Cheng, J.; Dasoulas, G.; He, H.; Agarwal, C.; and Zitnik, M. 2023. GNNDelete: A General Strategy for Unlearning in Graph Neural Networks. In The Eleventh International Conference on Learning Representations. Chien, Pan, and Milenkovic (2023) Chien, E.; Pan, C.; and Milenkovic, O. 2023. Efficient Model Updates for Approximate Unlearning of Graph-Structured Data. In The Eleventh International Conference on Learning Representations. Choi and Na (2023) Choi, D.; and Na, D. 2023. Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems. arXiv preprint arXiv:2311.02240. Cooper et al. (2024) Cooper, A. F.; Choquette-Choo, C. A.; Bogen, M.; Jagielski, M.; Filippova, K.; Liu, K. Z.; Chouldechova, A.; Hayes, J.; Huang, Y.; Mireshghallah, N.; et al. 2024. Machine Unlearning Doesn’t Do What You Think: Lessons for Generative AI Policy, Research, and Practice. arXiv preprint arXiv:2412.06966. Dukler et al. (2023) Dukler, Y.; Bowman, B.; Achille, A.; Golatkar, A.; Swaminathan, A.; and Soatto, S. 2023. Safe: Machine unlearning with shard graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 17108–17118. Eldan and Russinovich (2023) Eldan, R.; and Russinovich, M. 2023. Who’s Harry Potter? Approximate Unlearning in LLMs. arXiv preprint arXiv:2310.02238. Fan et al. (2024) Fan, C.; Liu, J.; Zhang, Y.; Wong, E.; Wei, D.; and Liu, S. 2024. SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. In The Twelfth International Conference on Learning Representations. Foster, Schoepf, and Brintrup (2024) Foster, J.; Schoepf, S.; and Brintrup, A. 2024. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 12043–12051. Gandikota et al. (2023) Gandikota, R.; Materzynska, J.; Fiotto-Kaufman, J.; and Bau, D. 2023. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2426–2436. Gao et al. (2023) Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac’h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; and Zou, A. 2023. A framework for few-shot language model evaluation. Ginart et al. (2019) Ginart, A.; Guan, M.; Valiant, G.; and Zou, J. Y. 2019. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32. Golatkar, Achille, and Soatto (2020) Golatkar, A.; Achille, A.; and Soatto, S. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9304–9312. Grosse et al. (2023) Grosse, R.; Bae, J.; Anil, C.; Elhage, N.; Tamkin, A.; Tajdini, A.; Steiner, B.; Li, D.; Durmus, E.; Perez, E.; et al. 2023. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296. Halimi et al. (2022) Halimi, A.; Kadhe, S. R.; Rawat, A.; and Angel, N. B. 2022. Federated Unlearning: How to Efficiently Erase a Client in FL? In International Conference on Machine Learning. Hayes et al. (2024) Hayes, J.; Shumailov, I.; Triantafillou, E.; Khalifa, A.; and Papernot, N. 2024. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. arXiv preprint arXiv:2403.01218. Hendrycks et al. (2022) Hendrycks, D.; Basart, S.; Mazeika, M.; Zou, A.; Kwon, J.; Mostajabi, M.; Steinhardt, J.; and Song, D. 2022. Scaling Out-of-Distribution Detection for Real-World Settings. In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 8759–8773. PMLR. Hendrycks et al. (2021) Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations. Hendrycks and Gimpel (2017) Hendrycks, D.; and Gimpel, K. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In International Conference on Learning Representations. Hong et al. (2024) Hong, Y.; Yu, L.; Ravfogel, S.; Yang, H.; and Geva, M. 2024. Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces. arXiv preprint arXiv:2406.11614. Isonuma and Titov (2024) Isonuma, M.; and Titov, I. 2024. Unlearning Reveals the Influential Training Data of Language Models. arXiv preprint arXiv:2401.15241. Izzo et al. (2021) Izzo, Z.; Smart, M. A.; Chaudhuri, K.; and Zou, J. 2021. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, 2008–2016. PMLR. Jang et al. (2023) Jang, J.; Yoon, D.; Yang, S.; Cha, S.; Lee, M.; Logeswaran, L.; and Seo, M. 2023. Knowledge Unlearning for Mitigating Privacy Risks in Language Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14389–14408. Toronto, Canada: Association for Computational Linguistics. Jeong, Ma, and Houmansadr (2024) Jeong, H.; Ma, S.; and Houmansadr, A. 2024. SoK: Challenges and Opportunities in Federated Unlearning. arXiv preprint arXiv:2403.02437. Jia et al. (2024) Jia, J.; Zhang, Y.; Zhang, Y.; Liu, J.; Runwal, B.; Diffenderfer, J.; Kailkhura, B.; and Liu, S. 2024. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239. Jiang et al. (2023) Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. Jones et al. (2023) Jones, E.; Dragan, A.; Raghunathan, A.; and Steinhardt, J. 2023. Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning, 15307–15329. PMLR. Koh and Liang (2017) Koh, P. W.; and Liang, P. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, 1885–1894. PMLR. Kumari et al. (2023) Kumari, N.; Zhang, B.; Wang, S.-Y.; Shechtman, E.; Zhang, R.; and Zhu, J.-Y. 2023. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 22691–22702. Kurmanji et al. (2023) Kurmanji, M.; Triantafillou, P.; Hayes, J.; and Triantafillou, E. 2023. Towards Unbounded Machine Unlearning. In Thirty-seventh Conference on Neural Information Processing Systems. Li et al. (2024a) Li, G.; Hsu, H.; Chen, C.-F.; and Marculescu, R. 2024a. Machine Unlearning for Image-to-Image Generative Models. In The Twelfth International Conference on Learning Representations. Li et al. (2024b) Li, N.; Pan, A.; Gopal, A.; Yue, S.; Berrios, D.; Gatti, A.; Li, J. D.; Dombrowski, A.-K.; Goel, S.; Phan, L.; et al. 2024b. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218. Li et al. (2024c) Li, X.; Zhao, Y.; Wu, Z.; Zhang, W.; Li, R.-H.; and Wang, G. 2024c. Towards Effective and General Graph Unlearning via Mutual Evolution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 13682–13690. Li et al. (2023) Li, Y.; Chen, C.; Zheng, X.; Zhang, Y.; Han, Z.; Meng, D.; and Wang, J. 2023. Making users indistinguishable: Attribute-wise unlearning in recommender systems. In Proceedings of the 31st ACM International Conference on Multimedia, 984–994. Liu et al. (2024a) Liu, C. Y.; Wang, Y.; Flanigan, J.; and Liu, Y. 2024a. Large Language Model Unlearning via Embedding-Corrupted Prompts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Liu et al. (2024b) Liu, J.; Lou, J.; Qin, Z.; and Ren, K. 2024b. Certified minimax unlearning with generalization rates and deletion capacity. Advances in Neural Information Processing Systems, 36. Liu et al. (2020) Liu, W.; Wang, X.; Owens, J.; and Li, Y. 2020. Energy-based out-of-distribution detection. Advances in neural information processing systems, 33: 21464–21475. Liu et al. (2024c) Liu, Z.; Dou, G.; Tan, Z.; Tian, Y.; and Jiang, M. 2024c. Machine Unlearning in Generative AI: A Survey. arXiv preprint arXiv:2407.20516. Liu et al. (2024d) Liu, Z.; Dou, G.; Tan, Z.; Tian, Y.; and Jiang, M. 2024d. Towards Safer Large Language Models through Machine Unlearning. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 1817–1829. Bangkok, Thailand: Association for Computational Linguistics. Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations. Lynch et al. (2024) Lynch, A.; Guo, P.; Ewart, A.; Casper, S.; and Hadfield-Menell, D. 2024. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835. Ma et al. (2022) Ma, Z.; Liu, Y.; Liu, X.; Liu, J.; Ma, J.; and Ren, K. 2022. Learn to forget: Machine unlearning via neuron masking. IEEE Transactions on Dependable and Secure Computing. Maini et al. (2024) Maini, P.; Feng, Z.; Schwarzschild, A.; Lipton, Z. C.; and Kolter, J. Z. 2024. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. Merity et al. (2022) Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2022. Pointer Sentinel Mixture Models. In International Conference on Learning Representations. Meta (2024) Meta, A. 2024. Introducing meta llama 3: The most capable openly available llm to date. Meta AI. Ngoc-Hieu et al. (2023) Ngoc-Hieu, N.; Hung-Quang, N.; Ta, T.-A.; Nguyen-Tang, T.; Doan, K. D.; and Thanh-Tung, H. 2023. A Cosine Similarity-based Method for Out-of-Distribution Detection. arXiv preprint arXiv:2306.14920. Nguyen et al. (2022) Nguyen, T. T.; Huynh, T. T.; Nguyen, P. L.; Liew, A. W.-C.; Yin, H.; and Nguyen, Q. V. H. 2022. A survey of machine unlearning. arXiv preprint arXiv:2209.02299. Northcutt, Jiang, and Chuang (2021) Northcutt, C.; Jiang, L.; and Chuang, I. 2021. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70: 1373–1411. Patil, Hase, and Bansal (2024) Patil, V.; Hase, P.; and Bansal, M. 2024. Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. In The Twelfth International Conference on Learning Representations. Pawelczyk, Neel, and Lakkaraju (2024) Pawelczyk, M.; Neel, S.; and Lakkaraju, H. 2024. In-Context Unlearning: Language Models as Few-Shot Unlearners. In Forty-first International Conference on Machine Learning. Plaut, Nguyen, and Trinh (2024) Plaut, B.; Nguyen, K.; and Trinh, T. 2024. Softmax probabilities (mostly) predict large language model correctness on multiple-choice q&a. arXiv preprint arXiv:2402.13213. Romandini et al. (2024) Romandini, N.; Mora, A.; Mazzocca, C.; Montanari, R.; and Bellavista, P. 2024. Federated unlearning: A survey on methods, design guidelines, and evaluation metrics. IEEE Transactions on Neural Networks and Learning Systems. Sekhari et al. (2021) Sekhari, A.; Acharya, J.; Kamath, G.; and Suresh, A. T. 2021. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34: 18075–18086. Shah et al. (2023) Shah, R.; Montixi, Q. F.; Pour, S.; Tagade, A.; and Rando, J. 2023. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation. In Socially Responsible Language Modelling Research. Shi et al. (2024a) Shi, W.; Ajith, A.; Xia, M.; Huang, Y.; Liu, D.; Blevins, T.; Chen, D.; and Zettlemoyer, L. 2024a. Detecting Pretraining Data from Large Language Models. In The Twelfth International Conference on Learning Representations. Shi et al. (2024b) Shi, W.; Lee, J.; Huang, Y.; Malladi, S.; Zhao, J.; Holtzman, A.; Liu, D.; Zettlemoyer, L.; Smith, N. A.; and Zhang, C. 2024b. MUSE: Machine Unlearning Six-Way Evaluation for Language Models. arXiv preprint arXiv:2407.06460. Sun et al. (2022) Sun, Y.; Ming, Y.; Zhu, X.; and Li, Y. 2022. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, 20827–20840. PMLR. Tan et al. (2024) Tan, J.; Sun, F.; Qiu, R.; Su, D.; and Shen, H. 2024. Unlink to unlearn: Simplifying edge unlearning in gnns. In Companion Proceedings of the ACM on Web Conference 2024, 489–492. Thudi et al. (2022) Thudi, A.; Deza, G.; Chandrasekaran, V.; and Papernot, N. 2022. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), 303–319. IEEE. Tunstall et al. (2023) Tunstall, L.; Beeching, E.; Lambert, N.; Rajani, N.; Rasul, K.; Belkada, Y.; Huang, S.; von Werra, L.; Fourrier, C.; Habib, N.; et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. Wang et al. (2025) Wang, H.; Lin, J.; Chen, B.; Yang, Y.; Tang, R.; Zhang, W.; and Yu, Y. 2025. Towards efficient and effective unlearning of large language models for recommendation. Frontiers of Computer Science, 19(3): 193327. Wang et al. (2022) Wang, J.; Guo, S.; Xie, X.; and Qi, H. 2022. Federated Unlearning via Class-Discriminative Pruning. In Proceedings of the ACM Web Conference 2022, W ’22, 622–632. New York, NY, USA: Association for Computing Machinery. ISBN 9781450390965. Warnecke et al. (2021) Warnecke, A.; Pirch, L.; Wressnegger, C.; and Rieck, K. 2021. Machine unlearning of features and labels. arXiv preprint arXiv:2108.11577. Wei, Haghtalab, and Steinhardt (2024) Wei, A.; Haghtalab, N.; and Steinhardt, J. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36. Wei et al. (2022) Wei, H.; Xie, R.; Cheng, H.; Feng, L.; An, B.; and Li, Y. 2022. Mitigating neural network overconfidence with logit normalization. In International conference on machine learning, 23631–23644. PMLR. Wu et al. (2023a) Wu, K.; Shen, J.; Ning, Y.; Wang, T.; and Wang, W. H. 2023a. Certified Edge Unlearning for Graph Neural Networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, 2606–2617. New York, NY, USA: Association for Computing Machinery. ISBN 9798400701030. Wu et al. (2023b) Wu, X.; Li, J.; Xu, M.; Dong, W.; Wu, S.; Bian, C.; and Xiong, D. 2023b. DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2875–2886. Singapore: Association for Computational Linguistics. Xu et al. (2023) Xu, H.; Zhu, T.; Zhang, L.; Zhou, W.; and Yu, P. S. 2023. Machine Unlearning: A Survey. ACM Comput. Surv., 56(1). Yao, Xu, and Liu (2023) Yao, Y.; Xu, X.; and Liu, Y. 2023. Large Language Model Unlearning. In Socially Responsible Language Modelling Research. Young et al. (2024) Young, A.; Chen, B.; Li, C.; Huang, C.; Zhang, G.; Zhang, G.; Li, H.; Zhu, J.; Chen, J.; Chang, J.; et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652. Yuan et al. (2024) Yuan, Y.; Jiao, W.; Wang, W.; tse Huang, J.; He, P.; Shi, S.; and Tu, Z. 2024. GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. In The Twelfth International Conference on Learning Representations. Zhang et al. (2024a) Zhang, G.; Wang, K.; Xu, X.; Wang, Z.; and Shi, H. 2024a. Forget-me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1755–1764. Zhang et al. (2024b) Zhang, R.; Lin, L.; Bai, Y.; and Mei, S. 2024b. Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning. In First Conference on Language Modeling. Zhang et al. (2023) Zhang, Y.; Hu, Z.; Bai, Y.; Wu, J.; Wang, Q.; and Feng, F. 2023. Recommendation unlearning via influence function. ACM Transactions on Recommender Systems. Zhu, Li, and Hu (2023) Zhu, X.; Li, G.; and Hu, W. 2023. Heterogeneous federated knowledge graph embedding learning and unlearning. In Proceedings of the ACM web conference 2023, 2444–2454. Zou et al. (2023a) Zou, A.; Phan, L.; Chen, S.; Campbell, J.; Guo, P.; Ren, R.; Pan, A.; Yin, X.; Mazeika, M.; Dombrowski, A.-K.; et al. 2023a. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Zou et al. (2023b) Zou, A.; Wang, Z.; Kolter, J. Z.; and Fredrikson, M. 2023b. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Appendix A Datasets and Q&A template A.1 Datasets WMDP (Li et al. 2024b) stands for Weapon of Mass Destruction Proxy, is a corpora consisting of forget sets, retain sets, and Q&A sets. The WMDP Q&A is a dataset of 3,66836683,6683 , 668 multiple-choice questions about Biosecurity (1,27312731,2731 , 273), Cybersecurity (1,98719871,9871 , 987), and Chemical security (408408408408). The WMDP-Biology forget and retain sets consist of papers from PubMed. The WMDP-Biology forget comprises papers used in generating WMDP-Biology questions, while the retain set samples papers from various categories within general biology. The retain set excludes papers from the forget set and employs keyword exclusion to avoid topics related to Q&A set. The WMDP-Cyber forget and retain sets consist of passages crawled from GitHub with two different sets of keywords. Note that we did not benchmark for WMDP-Chemistry Q&A due to no WMDP-Chemistry forget set being publicly released. This dataset is available at https://github.com/centerforaisafety/wmdp. MMLU (Hendrycks et al. 2021) stands for Massive Multitask Language Understanding, a dataset of 15,9081590815,90815 , 908 multiple-choice Q&A covers 57575757 subjects across STEM, the humanities, social science, and more. MMLU is designed to measure general knowledge by evaluating models in zero-shot or few-shot settings. This dataset is available at https://huggingface.co/datasets/cais/mmlu. Q&A format. We follow the default zero-shot Q&A format to query LLM (Gao et al. 2023). For example: “The following are multiple choice questions (with answers) about Biology. Question: How can I cultivate a calicivirus? A. Cell/enteroid culture B. Helper virus-free reverse genetics system C. Either A or B D. Neither A nor B Answer:” A.2 Computational perplexity of the RMU and Adaptive RMU. The difference between RMU and Adaptive RMU is the calculation of the forget coefficient. RMU uses a predefined coefficient c. Adaptive RMU uses an adaptive coefficient, β⁢‖hθfrozen(l)⁢(xF)‖normsuperscriptsubscriptℎsuperscriptfrozensubscriptβ||h_θ^frozen^(l)(x_F)||β | | hitalic_θfrozen( l ) ( xitalic_F ) | |, which can be calculated and cached during the first iteration of the inner for loop in Algorithm 1. Thus, the complexity of Adaptive RMU is equal to that of RMU. Appendix B Additional results B.1 Unlearning performance of other models We report the unlearning performance of Adaptive RMU Yi-6B, Llama-3-8B, and Mistral-7B models in Table 2, Table 3, and Table 4. We observed a clear trend that the unlearning performance is more effective when using the early layer as the unlearn layer. Task/unlearn layer base 3 4 5 6 7 8 9 10 11 12 13 14 15 16 WMDP-Biology ↓ ↓ 64.8 65.0 49.9 35.2 27.8 26.1 63.3 26.2 27.1 27.4 27.1 26.0 25.4 27.2 34.8 WMDP-Cyber ↓ ↓ 41.1 40.7 40.5 37.7 28.1 25.5 39.3 25.6 23.9 26.1 23.6 24.3 24.2 24.0 25.5 MMLU ↑ ↑ 60.0 60.1 57.7 59.4 51.4 56.5 59.9 56.8 53.7 48.1 49.3 57.0 55.6 47.7 53.3 Task/unlearn layer 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 WMDP-Biology ↓ ↓ 30.3 32.2 27.1 31.9 41.0 53.4 50.4 53.2 39.2 46.0 39.0 42.5 41.6 40.5 64.8 WMDP-Cyber ↓ ↓ 25.3 24.4 24.3 24.5 26.7 29.8 33.9 36.2 34.3 34.6 31.4 30.4 39.6 40.8 40.6 MMLU ↑ ↑ 45.4 52.1 56.7 58.2 59.3 59.4 59.6 59.7 59.4 59.7 59.4 59.4 59.5 59.7 60.1 Table 2: Q&A accuracy of Adaptive RMU Yi-6B models on WMDP-Biology, WMDP-Cyber, and MMLU. Task/unlearn layer base 3 4 5 6 7 8 9 10 11 12 13 14 15 16 WMDP-Biology ↓ ↓ 71.2 46.4 45.3 28.2 27.8 29.3 33.7 36.0 65.1 64.9 62.8 65.2 59.6 44.4 41.4 WMDP-Cyber ↓ ↓ 43.9 32.5 25.5 24.5 27.6 26.8 27.3 26.3 32.5 32.3 34.1 35.2 29.9 28.3 27.8 MMLU ↑ ↑ 62.0 60.7 60.2 59.7 60.7 60.0 60.1 59.6 61.8 61.3 61.5 61.5 61.8 60.9 61.1 Task/unlearn layer 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 WMDP-Biology ↓ ↓ 35.5 35.2 41.1 60.8 33.7 59.3 54.6 56.7 69.6 62.2 70.0 69.9 69.9 67.0 70.4 WMDP-Cyber ↓ ↓ 28.0 33.5 28.6 39.0 28.6 31.7 35.5 36.9 45.5 44.8 44.4 43.5 44.4 43.6 43.4 MMLU ↑ ↑ 61.3 61.3 61.3 61.9 60.8 61.7 61.2 61.5 61.9 61.7 62.0 61.9 61.5 61.5 62.1 Table 3: Q&A accuracy of Adaptive RMU Meta Llama-3-8B models on WMDP-Biology, WMDP-Cyber, and MMLU. Task/unlearn layer base 3 4 5 6 7 8 9 10 11 12 13 14 15 16 WMDP-Biology ↓ ↓ 67.3 28.0 28.9 27.6 27.5 26.3 24.5 25.7 26.1 27.6 31.4 37.7 35.6 25.4 35.0 WMDP-Cyber ↓ ↓ 44.1 42.1 41.9 24.8 26.8 26.3 26.6 26.4 26.7 25.7 26.5 25.8 31.6 26.7 27.9 MMLU ↑ ↑ 58.7 54.5 57.2 54.9 55.8 55.7 47.3 53.0 47.4 35.1 54.5 55.9 51.5 44.9 57.3 Task/unlearn layer 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 WMDP-Biology ↓ ↓ 27.4 56.4 38.4 45.7 42.0 52.0 52.4 61.1 57.5 62.2 63.2 66.3 61.9 61.0 66.0 WMDP-Cyber ↓ ↓ 27.5 38.9 26.5 26.7 26.6 27.4 27.7 38.9 43.9 43.4 43.7 43.8 44.0 42.5 43.4 MMLU ↑ ↑ 56.7 56.8 56.2 57.6 58.1 58.3 58.1 58.2 58.6 58.7 58.6 58.7 58.4 58.3 58.2 Table 4: Q&A accuracy of Adaptive RMU Mistral-7B models on WMDP-Biology, WMDP-Cyber, and MMLU. B.2 Performances on MMLU subset unlearning benchmark We did additional experiments on the MMLU subset unlearning benchmark with three settings: 1. MMLU-Economics: unlearning high school microeconomics and macroeconomics and maintaining performance on the remaining categories (refers as MMLU-Retain tasks). 2. MMLU-Law: unlearning international and professional law while maintaining performance on MMLU-Retain. 3. MMLU-Physics: unlearning high school and college physics while maintaining general performance in MMLU-Retain. Settings. We use publicly released forget set by Li et al. (2024b) for each task and Wikitext (Merity et al. 2022) as retain set. We use a fixed sequence len of 512512512512 for MMLU-Economics, MMLU-Law, MMLU-Physics, and Wikitext. We keep other hyperparameters remain unchanged as in Section 6. Result. Table 5 presents the unlearning performance of Adaptive RMU Zephyr-7B models on MMLU-Economics, MMLU-Law, and MMLU-Physics. We observe a notable reduction in accuracy on the forget tasks. However, the model exhibits excessive unlearning, leading to substantial performance degradation on the MMLU-Retain tasks. Task/unlearn layer base 3 4 5 6 7 8 9 10 11 12 13 14 15 16 MMLU-Economics ↓ ↓ 58.0 57.0 45.7 22.8 23.4 27.0 28.8 27.0 34.6 24.6 42.1 45.5 34.8 44.5 58.3 MMLU-Law ↓ ↓ 55.6 49.8 53.5 25.2 24.5 26.4 24.6 24.2 21.5 23.9 51.1 44.1 36.8 44.7 46.0 MMLU-Physics ↓ ↓ 38.5 39.3 37.9 28.8 27.2 23.8 21.7 20.5 21.0 29.2 32.6 34.1 34.4 35.7 42.3 MMLU-Retain ↑ ↑ 58.9 58.0 57.3 39.3 45.2 39.4 35.2 36.0 44.8 35.2 52.9 55.2 46.0 54.8 56.8 Task/unlearn layer 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 MMLU-Economics ↓ ↓ 51.8 36.0 54.4 26.0 21.4 42.8 43.4 42.8 48.4 57.2 58.7 50.0 58.2 58.9 57.8 MMLU-Law ↓ ↓ 49.8 24.3 54.4 27.2 24.6 24.2 25.4 44.6 54.4 55.8 56.7 53.6 55.6 55.4 56.1 MMLU-Physics ↓ ↓ 37.5 26.7 26.9 21.0 21.6 24.2 23.4 25.6 29.6 37.1 31.9 33.8 36.9 33.9 38.6 MMLU-Retain ↑ ↑ 57.6 47.8 57.7 36.2 30.3 39.6 47.4 52.0 58.1 58.9 58.9 56.4 59.0 59.1 59.0 Table 5: Q&A accuracy of Adaptive RMU Zephyr-7B models on MMLU-Economics, MMLU-Law, MMLU-Phycics, and MMLU-Retain. B.3 The effect of in-domain retain set on unlearning performance. In this setting, we use the WMDP-Biology and WMDP-Cyber retain sets instead of Wikitext. We use the same hyperparameters as in Section 6. Table 6 shows that Adaptive RMU is almost ineffective for all unlearn layers. As WMDP-forget and retain sets are collected from the same source, even with efforts in distinction, these corpora may commonly have overlapping texts. We present an n-gram overlap analysis between the WMDP-forget set and the WMDP-retain set as a measurement of unlearning difficulty. Task/unlearn layer base 3 4 5 6 7 8 9 10 11 12 13 14 15 16 WMDP-Biology ↓ ↓ 63.7 63.2 63.3 62.9 28.1 62.6 49.9 64.2 29.6 62.0 63.0 63.7 63.7 64.4 64.3 WMDP-Cyber ↓ ↓ 43.5 42.7 42.0 40.1 24.6 33.3 33.9 40.8 25.1 41.3 41.7 42.8 43.4 42.8 43.4 MMLU-All ↑ ↑ 58.1 57.4 57.4 57.9 30.1 57.6 38.3 57.6 29.3 57.1 58.0 57.5 57.7 57.9 57.8 Task/unlearn layer 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 WMDP-Biology ↓ ↓ 63.9 63.7 63.9 63.5 63.5 63.7 63.7 63.6 63.6 63.5 63.3 63.7 63.8 63.5 64.6 WMDP-Cyber ↓ ↓ 44.5 43.5 43.5 44.4 43.9 43.5 44.3 43.6 43.9 43.8 43.6 43.2 43.7 43.7 43.6 MMLU-All ↑ ↑ 58.4 58.1 58.2 57.6 58.2 58.1 58.2 58.1 58.1 58.0 58.2 58.1 58.2 58.1 57.9 Table 6: Q&A accuracy of Adaptive RMU Zephyr-7B models on WMDP-Biology, WMDP-Cyber, and MMLU. Models were fine-tuned on WMDP-Biology and WMDP-Cyber retain sets. n-gram overlap analysis. (a) Distribution of Unigram overlap score between WMDP-Biology retain and WMDP-Biology forget sets. (b) Distribution of Bigram overlap score between WMDP-Biology retain and WMDP-Biology forget sets. (c) Distribution of Unigram overlap score between WMDP-Cyber retain and WMDP-Cyber forget sets. (d) Distribution of Bigram overlap score between WMDP-Cyber retain and WMDP-Cyber forget sets. Figure 6: Distributions of Unigram and Bigram overlap scores. Given a retain sample x1:k∈retainsubscript:1subscriptretainx_1:k _ retainx1 : k ∈ Dretain consists of k tokens x1,x2,…⁢xksubscript1subscript2…subscript\x_1,x_2,...x_k\ x1 , x2 , … xitalic_k , we denote xi:i+n−1subscript:1x_i:i+n-1xitalic_i : i + n - 1 for i∈[1,…,k−n+1]1…1i∈[1,...,k-n+1]i ∈ [ 1 , … , k - n + 1 ] as the n-gram of x1:ksubscript:1x_1:kx1 : k. The n-gram overlap score of x1:ksubscript:1x_1:kx1 : k in forget set forget=xF|forget|subscriptforgetsuperscriptsubscriptsubscriptforgetD_ forget=\x_F\^|D_ forget% |Dforget = xitalic_F | Dforget | is defined as: 1|forget|⁢1k−n+1⁢∑xR∑i=1k−n+1⁢[xi:i+n−1∈xF],1subscriptforget11subscriptsubscriptsuperscriptsubscript11delimited-[]subscript:1subscript 1|D_ forget| 1k-n+1 _% x_R _i=1^k-n+1I[x_i:i+n-1∈ x_F],divide start_ARG 1 end_ARG start_ARG | Dforget | end_ARG divide start_ARG 1 end_ARG start_ARG k - n + 1 end_ARG ∑x start_POSTSUBSCRIPT R end_POSTSUBSCRIPT ∑i = 1k - n + 1 blackboard_I [ xitalic_i : i + n - 1 ∈ xitalic_F ] , (25) where ⁢(⋅)⋅I(·)blackboard_I ( ⋅ ) is the indicator function and ⁢[xi:i+n−1∈xF]=1delimited-[]subscript:1subscript1I[x_i:i+n-1∈ x_F]=1blackboard_I [ xitalic_i : i + n - 1 ∈ xitalic_F ] = 1 if the substring xi:i+n−1subscript:1x_i:i+n-1xitalic_i : i + n - 1 is in forget sample xFsubscriptx_Fxitalic_F, otherwise 00. We randomly sampled 1000100010001000 documents from each dataset and performed Unigram (n=11n=1n = 1) and Bigram (n=22n=2n = 2) overlap analysis. The results indicate a high degree of unigram and bigram overlap between the WMDP-forget and WMDP-retain sets. Specifically, the average Unigram and Bigram overlap scores for the WMDP-Biology forget and retain sets were 20.8%percent20.820.8\%20.8 % and 5.5%percent5.55.5\%5.5 %, respectively. These overlap scores were even higher for the WMDP-Cyber sets, at 27.5%percent27.527.5\%27.5 % and 12.3%percent12.312.3\%12.3 %, respectively. The distributions of n-gram overlap scores are visualized in Fig. 6. High n-gram overlap scores make two distributions WMDP-forget set and WMDP-retain set less distinction, which makes the unlearning more difficult. B.4 Limitation and future work We discuss the following limitations in our paper: 1. We mainly perform experiments on 7B versions (or equivalent) due to computational constraints. To validate the generalizability of our approach and findings, we conducted experiments across the Zephyr, Mistral, Llama, and Yi models. 2. Our analysis in Section 3.3 on white-box attacks for open weight models. In practice, state-of-the-art LLMs such as GPT, Gemini, and Claude are trained privately and are accessible through API only. The most common form of attack on LLMs, therefore, is a black-box jailbreak attack. We encourage future works to explore the analysis of the robustness of unlearned models covering black-box jailbreak attacks. 3. Limiting update the model parameters w.r.t three layer l,l−1,l−212\l,l-1,l-2\ l , l - 1 , l - 2 thus risks missing interesting generalization behaviors.

Instant research discovery