Paper deep dive
Layered Unlearning for Adversarial Relearning
Timothy Qian, Vinith Suriyakumar, Ashia Wilson, Dylan Hadfield-Menell
Models: Zephyr-7B-beta
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 5:57:30 PM
Summary
The paper introduces Layered Unlearning (LU), a machine unlearning algorithm designed to improve robustness against adversarial relearning. By partitioning data into sequential folds and applying unlearning in stages, LU creates distinct inhibitory mechanisms that prevent the model from recovering the full dataset when only a subset is used for relearning. The authors demonstrate that LU outperforms standard unlearning methods in both synthetic tasks and large language model experiments, while also identifying a performance gap between MCQ-based and corpus-based relearning attacks.
Entities (5)
Relation Signals (3)
Layered Unlearning β appliedto β Large Language Models
confidence 95% Β· We evaluate LU through a combination of synthetic and large language model (LLM) experiments.
Layered Unlearning β improvesrobustnessagainst β Adversarial Relearning
confidence 95% Β· We find that LU improves robustness to adversarial relearning for several different unlearning methods.
Layered Unlearning β contributesto β Machine Unlearning
confidence 90% Β· Our results contribute to the state-of-the-art of machine unlearning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Our goal is to understand how post-training methods, such as fine-tuning, alignment, and unlearning, modify language model behavior and representations. We are particularly interested in the brittle nature of these modifications that makes them easy to bypass through prompt engineering or relearning. Recent results suggest that post-training induces shallow context-dependent ``circuits'' that suppress specific response patterns. This could be one explanation for the brittleness of post-training. To test this hypothesis, we design an unlearning algorithm, Layered Unlearning (LU), that creates distinct inhibitory mechanisms for a growing subset of the data. By unlearning the first $i$ folds while retaining the remaining $k - i$ at the $i$th of $k$ stages, LU limits the ability of relearning on a subset of data to recover the full dataset. We evaluate LU through a combination of synthetic and large language model (LLM) experiments. We find that LU improves robustness to adversarial relearning for several different unlearning methods. Our results contribute to the state-of-the-art of machine unlearning and provide insight into the effect of post-training updates.
Tags
Links
Trouble viewing inline? Open PDF directly β
Full Text
92,626 characters extracted from source content.
Expand or collapse full text
Layered Unlearning for Adversarial Relearning Timothy Qian, Vinith Suriyakumar, Ashia Wilson, Dylan Hadfield-Menell MIT tcqian,vinithms,ashia07,dylanhm@mit.edu Abstract Our goal is to understand how post-training methods, such as fine-tuning, alignment, and unlearning, modify language model behavior and representations. We are particularly interested in the brittle nature of these modifications that makes them easy to bypass through prompt engineering or relearning. Recent results suggest that post-training induces shallow context-dependent βcircuitsβ that suppress specific response patterns. This could be one explanation for the brittleness of post-training. To test this hypothesis, we design an unlearning algorithm, Layered Unlearning (LU), that creates distinct inhibitory mechanisms for a growing subset of the data. By unlearning the first i folds while retaining the remaining kβik-ik - i at the iiith of k stages, LU limits the ability of relearning on a subset of data to recover the full dataset. We evaluate LU through a combination of synthetic and large language model (LLM) experiments. We find that LU improves robustness to adversarial relearning for several different unlearning methods. Our results contribute to the state-of-the-art of machine unlearning and provide insight into the effect of post-training updates. NoHyperβ Synthetic experiment code at: https://github.com/12tqian/layered-unlearning 1 Introduction Post-training interventions such as fine-tuning, preference learning, and unlearning are widely used to modify the behavior of pre-trained large language models (LLMs). However, changes introduced in post-training are often brittle. However, these changes are often shallow or brittle. In many cases, they are bypassed or reversed by clever adversarial prompting or fine-tuning (Jain et al., 2024; Arditi et al., 2024; Zou et al., 2023; Greenblatt et al., 2024; Che et al., 2024; Deeb & Roger, 2025; Betley et al., 2025). Our goal is to understand how post-training methods modify language model behavior and representation and support the design of more robust post-training methods. We study this through the lens of machine unlearning, which seeks to remove knowledge or capabilities from pre-trained models. Deeb & Roger (2025) recently demonstrated that βunlearnedβ information is easily re-elicited by fine-tuning on a subset of the removed data. To explain this result, we hypothesize that SoTA unlearning methods introduce a context-dependent inhibitor mechanism. Efficient βrelearningβ generalizes because fine-tuning removes a single shared mechanism and reverses the full post-training modification. A common way to mitigate single failure points is the famous βSwiss cheeseβ defense-in-depth model (Reason, 1990). The goal is to combine multiple imperfect defenses. If their failure modes are distinct, the combined defense is more robust than any individual approach. We implement defense-in-depth through Layered Unlearning (LU). LU partitions the data into k disjoint folds and applies unlearning sequentially to a growing subset: at stage i, LU unlearns the union of folds F1subscript1F_1F1 through FisubscriptF_iFitalic_i. Crucially, we retain the data from Fi+1subscript1F_i+1Fitalic_i + 1 through FksubscriptF_kFitalic_k to induce distinct inhibitors at each stage of unlearning. Figure 1 illustrates the algorithm and the defense-in-depth inspiration. Figure 1: Left: An illustration of LU with social security numbers (SSNs). The SSNs are partitioned into disjoint sets (A,B,C)(A,B,C)( A , B , C ). Top: Standard unlearning minimizes performance on AβͺBβͺCAβͺ Bβͺ CA βͺ B βͺ C while retaining general capabilities on a retain set R (e.g., MMLU). Bottom: In LU, we sequentially unlearn the sequence A,AβͺB,AβͺBβͺC\A,Aβͺ B,Aβͺ Bβͺ C\ A , A βͺ B , A βͺ B βͺ C while retaining the sequence BβͺCβͺR,CβͺR,R\Bβͺ Cβͺ R,Cβͺ R,R\ B βͺ C βͺ R , C βͺ R , R . Middle: As a result, relearning B improves performance on C but not A. In contrast, training on any subset improves performance across the board for standard methods. Right: We hypothesize that unlearning the full set introduces a context-dependent shared inhibitor IAβ’Bβ’CsubscriptI_ABCIitalic_A B C that suppresses the information and that subsequent relearning removes IAβ’Bβ’CsubscriptI_ABCIitalic_A B C. The structure of LU is designed to create several distinct inhibitors IA,IAβ’B,IAβ’Bβ’CsubscriptsubscriptsubscriptI_A,I_AB,I_ABCIitalic_A , Iitalic_A B , Iitalic_A B C that cover different folds of the data. Relearning on B removes IAβ’BsubscriptI_ABIitalic_A B and IAβ’Bβ’CsubscriptI_ABCIitalic_A B C, but leaves IAsubscriptI_AIitalic_A active. We investigate the performance of LU on a variety of synthetic tasks and unlearning benchmarks. We consider a synthetic 2-dimensional classification task and a 3-token sequence generation task. Next, we apply LU to a variety of unlearning methods on the WMDP, Years, and MMLU datasets. In all settings, LU improves resistance to fine-tuning-based recovery. In the course of these experiments, we also identify a stronger class of attack: corpus-based fine-tuning. This attack, which uses raw text rather than structured MCQ-based prompts, bypasses inhibitors more effectively than standard RTT (Deeb & Roger, 2025). Notably, this distinction only emerges because LU creates variation in robustness across data foldsβan effect that is not observed with standard unlearning. We make three contributions: 1) we introduce Layered Unlearning (LU) a method that combines multiple steps of unlearning to increase robustness; 2) we show in both synthetic and LLM settings that LU improves resistance to adversarial relearning; and 3) we use LU to reveal a gap in attack strength between MCQ-based and corpus-based relearning, offering new insight into the limits of post-training behavioral control. Our results contribute to the state-of-the-art of machine unlearning and provide insight into the effect of post-training updates. 2 Layered Unlearning In this section, we introduce LU. We begin with a replication of Deeb & Roger (2025) in two synthetic tasks: a 2D classification task with mixtures of Gaussians and a bigram completion task with three tokens. Next, we introduce the LU algorithm. Finally, we investigate the effect of LU in this task. We find that LU improves robustness in both cases and analyze the sequences of changes that LU induces. First we introduce some notation to represent a machine unlearning method U. 2.1 Machine unlearning notation The goal of machine unlearning is to remove information F from trained model weights ΞΈβΞΞΈβ ΞΈ β Ξ that model a dataset D in some input space X. With unlimited compute, this would involve retraining from scratch on DβFD FD β F. Due to, e.g., the cost of pretraining, unlearning methods attempt to approximate this result as a post-training step that maintains performance on a retain set RβDRβ DR β D. Thus, we can represent a generic unlearning algorithm U as a function that maps model parameters ΞΈ, forget set F, retain set R, and hyperparameters Ξ³βΞΞ³β Ξ³ β Ξ to a new set of model parameters ΞΈβ²ΞΈ ΞΈβ². When clear from context, we may omit the final argument corresponding to the hyperparameters. Formally U:ΞΓXΓXΓΞβΞ.:βΞΞΞU: Γ XΓ XΓ β .U : Ξ Γ X Γ X Γ Ξ β Ξ . 2.2 Adversarial relearning in synthetic settings We replicate the results of Deeb & Roger (2025) in two synthetic settings: a 2D classification task and a bigram language modeling task. 2D logistic regression. Our first task is a 2D logistic regression task, so our input space is X=β2superscriptβ2X=R^2X = blackboard_R2. The goal is to classify a mixture of Gaussians (class 1111) against a uniform background distribution over β’([β60,60]2)superscript60602U([-60,60]^2)U ( [ - 60 , 60 ]2 ) (class 00). We sample the Gaussian means from the uniform distribution β’([β50,50]2)superscript50502U([-50,50]^2)U ( [ - 50 , 50 ]2 ) and use a primarily isotropic covariance matrix with variance Ο2=4superscript24Ο^2=4Ο2 = 4, adding small perturbations of magnitude 0.10.10.10.1 to break exact symmetry. We implement a linear classifier with logistic regression on radial basis functions (RBF) features. We partition the Gaussians into subsets A,B,RA,B,RA , B , R, where the goal is to unlearn AβͺBAβͺ BA βͺ B and retain R. Tasks A and B are defined as the classification accuracy when sets A and B are labeled as class 1111, and the retain task as the joint accuracy on classifying R as class 1111 and Null as class 00. We place the RBF centers on an 12Γ12121212Γ 1212 Γ 12 grid of points, so Ξ=β145Ξsuperscriptβ145 =R^145Ξ = blackboard_R145 (including a bias term). The top left of figure 2 shows the learned weights with 2222 folds. The unlearning primitive U takes as input model parameters ΞΈ, a forget set F, a retain set R, and hyperparameters Ξ³. Each data point in F and R is a 2D input. The objective is to preserve the original classification for points in R, while reclassifying points in F as class 00. Relearning refers to assigning data points in the relearned set back to their original classifications. We optimize all objectives using the Adam optimizer. Bigram sequence modeling. Next, we consider a bigram language modeling task with three tokens: a,b,ra,b,ra , b , r. The input space is length 8888 token sequences: X=a,b,r8superscript8X=\a,b,r\^8X = a , b , r 8. We generate data so that a and b are followed by r, while r is followed by a and b with equal probability. We combine this with a small minimum probability Ο΅=0.05italic-Ο΅0.05Ξ΅=0.05Ο΅ = 0.05 of a uniform transition across tokens to encourage smooth learning dynamics. This leads to the following conditional probabilities for consecutive tokens: Pβ’(rβ£a)=Pβ’(rβ£b)=1β2β’Ο΅conditionalconditional12italic-Ο΅P(r a)=P(r b)=1-2 ( r β£ a ) = P ( r β£ b ) = 1 - 2 Ο΅, Pβ’(aβ£r)=Pβ’(bβ£r)=12βΟ΅2,conditionalconditional12italic-Ο΅2P(a r)=P(b r)= 12- Ξ΅2,P ( a β£ r ) = P ( b β£ r ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG - divide start_ARG Ο΅ end_ARG start_ARG 2 end_ARG , and Pβ’(aβ£a)=Pβ’(bβ£a)=Pβ’(aβ£b)=Pβ’(bβ£b)=Pβ’(rβ£r)=Ο΅.conditionalconditionalconditionalconditionalconditionalitalic-Ο΅P(a a)=P(b a)=P(a b)=P(b b)=P(r r)=Ξ΅.P ( a β£ a ) = P ( b β£ a ) = P ( a β£ b ) = P ( b β£ b ) = P ( r β£ r ) = Ο΅ . We define tasks A, B, and R as the prediction performance on all consecutive pairs tokens in the bigram sets aβ’a,aβ’b,aβ’r\a,ab,ar\ a a , a b , a r , bβ’a,bβ’b,bβ’r\ba,b,br\ b a , b b , b r , and rβ’a,rβ’b,rβ’r\ra,rb,r\ r a , r b , r r , respectively. We use a small attention-only, one-layer transformer with parameter space Ξ=β4288Ξsuperscriptβ4288 =R^4288Ξ = blackboard_R4288. This architecture is simple enough to permit analytical study (Elhage et al., 2021), while still serving as a useful proxy for the larger models used in our later experiments. The unlearning primitive U takes as input model parameters ΞΈ, a forget set F, a retain set R, and hyperparameters Ξ³. Data points in F and R are elements of a,b,r\a,b,r\ a , b , r . For each xβRxβ Rx β R, the objective is to preserve the original conditional distribution over consecutive tokens, P(β β£x)P(Β· x)P ( β β£ x ). For each xβFxβ Fx β F, the goal is to flatten the distribution to P(β β£x)=13P(Β· x)= 13P ( β β£ x ) = divide start_ARG 1 end_ARG start_ARG 3 end_ARG. The retain task prevents the model from collapsing to a trivial uniform distribution over tokens after unlearning. To implement unlearning, we generate sequences from this modified transition matrix and optimize the standard language modeling loss. Relearning refers to restoring the original conditional distribution over tokens for the subset being relearned. We optimize all objectives using the Adam optimizer. Relearning performance. For each task, we first apply the unlearning primitive, followed by relearning on each subtask. We limit our discussion to relearning on B and evaluating on A for symmetryβs sake. In 2D classification, relearning on B restores 93%percent9393\%93 % of task Aβs original performance (Table 1). Figure 2 (top) illustrates the corresponding weight trajectories. In bigram modeling, we observe similar behavior: relearning on B recovers 73%percent7373\%73 % of task Aβs performance (Table 2). 2.3 The Layered Unlearning algorithm Figure 2: A depiction of Layered Unlearning in our 2D logistic regression example. Scatter plots represent the data, which consists of a uniform distribution (Class 0) and a mixture of Gaussians (Class 1) that is split into three subsets, A, B, and R. The goal of unlearning is to forget A,BA,BA , B while retaining R. Our classifiers are trained with logistic regression with radial basis functions spaced out in a grid. We show the weights as a heatmap on the grid. The top left shows ΞΈ0subscript0 _0ΞΈ0, the initial trained model. Across the top row, we illustrate the effect of joint unlearning ΞΈ1=(ΞΈ0,AβͺB,R)subscript1subscript0 _1=( _0,Aβͺ B,R)ΞΈ1 = ( ΞΈ0 , A βͺ B , R ) and subsequent relearning on B. Note that relearning B also relearns A. The corresponding classification logits along Y=00Y=0Y = 0 are shown below. Notice how learning on B generalizes to the area around (0,0)00(0,0)( 0 , 0 ). In the bottom row, we show the steps of Layered Unlearning. First, we compute ΞΈ0β²=Uβ’(ΞΈ0,A,RβͺB),subscriptsuperscriptβ²0subscript0ΞΈ _0=U( _0,A,Rβͺ B),ΞΈβ²0 = U ( ΞΈ0 , A , R βͺ B ) , shown in the bottom left, then we compute ΞΈ1β²=Uβ’(ΞΈ0β²,AβͺB,R),superscriptsubscript1β²subscript0β² _1 =U( _0 ,Aβͺ B,R),ΞΈ1β² = U ( ΞΈ0β² , A βͺ B , R ) , shown in the bottom middle. The logit plot shows the clear effect on the logits near (0,0)00(0,0)( 0 , 0 ). The bottom right shows the effect of subsequently relearning B, while performance on B still improves, it no longer generalizes to A. Next, we present the LU algorithm and evaluate it in these two domains. Algorithm 1 shows the algorithm details. The algorithm relies on an unlearning primitive U that maps model weights ΞΈ, forget set F, retain set R, and algorithm hyperpararmeters Ξ³ Its primary input is a set of model weights ΞΈ0subscript0 _0ΞΈ0, a sequence of k forget sets Fisubscript\F_i\ Fitalic_i , a retain set R, an unlearning algorithm U, and a sequence of algorithm hyperparameters Ξ³isubscript\ _i\ Ξ³italic_i . LU proceeds through k steps. At step i, LU computes ΞΈi=Uβ’(ΞΈiβ1,F1βͺF2β’β―βͺFi,RβͺFi+1β’β―βͺFk,Ξ³i)::subscriptsubscript1subscript1subscript2β―subscriptsubscript1β―subscriptsubscriptabsent _i=U( _i-1,F_1βͺ F_2β¦βͺ F_i,Rβͺ F_i+1β¦% βͺ F_k, _i):ΞΈitalic_i = U ( ΞΈitalic_i - 1 , F1 βͺ F2 β― βͺ Fitalic_i , R βͺ Fitalic_i + 1 β― βͺ Fitalic_k , Ξ³italic_i ) : it unlearns F1βͺF2β’β―βͺFisubscript1subscript2β―subscriptF_1βͺ F_2β¦βͺ F_iF1 βͺ F2 β― βͺ Fitalic_i while retaining RβͺFi+1β’β―βͺFksubscript1β―subscriptRβͺ F_i+1β¦βͺ F_kR βͺ Fitalic_i + 1 β― βͺ Fitalic_k. We analyze LU through inhibitors. In the 2222-fold case of unlearning tasks A and B, the first stage forgets A while retaining B, which forces the model to activate an inhibitor IAsubscriptI_AIitalic_A that selectively suppresses performance on A. In the second stage, the model may activate either an inhibitor IBsubscriptI_BIitalic_B that targets B specifically, or a shared inhibitor IAβ’BsubscriptI_ABIitalic_A B that suppresses both A and B. Upon relearning B, any inhibitors affecting Bβnamely IBsubscriptI_BIitalic_B or IAβ’BsubscriptI_ABIitalic_A Bβare deactivated, but IAsubscriptI_AIitalic_A remains active, so performance on A stays suppressed. Conversely, when relearning A, the inhibitors IAsubscriptI_AIitalic_A and IAβ’BsubscriptI_ABIitalic_A B are deactivated. If IAβ’BsubscriptI_ABIitalic_A B had been activated, performance on B is also restored; however, if IBsubscriptI_BIitalic_B had been activated instead, performance on B remains suppressed. Whether the barrier to adversarial relearning is unidirectional or bidirectional depends on the unlearning primitive. 3333-fold LU is illustrated in Figure 1. Algorithm 1 Layered Unlearning 1:Model parameters ΞΈ0subscript0 _0ΞΈ0, forget dataset sequence F1,β¦,Fksubscript1β¦subscript\F_1,β¦,F_k\ F1 , β¦ , Fitalic_k , retain dataset R0subscript0R_0R0, hyperparameters Ξ³1,β¦,Ξ³ksubscript1β¦subscript\ _1,β¦, _k\ Ξ³1 , β¦ , Ξ³italic_k , unlearning algorithm U 2:Unlearned model ΞΈksubscript _kΞΈitalic_k 3:F=β ,R=R0β’βi=1,β¦,kFiformulae-sequencesubscript0subscript1β¦subscriptF= ,R=R_0 _i=1,β¦,kF_iF = β , R = R0 βi = 1 , β¦ , k Fitalic_i β· β· Initialize incremental forget and retain sets F and R. 4:for i=11i=1i = 1 to k do β· β· Iterate through sequential forget stages 5: F=FβͺFi,R=RβFiformulae-sequencesubscriptsubscriptF=Fβͺ F_i,R=R F_iF = F βͺ Fitalic_i , R = R β Fitalic_i β· β· Update forget and retain sets 6: ΞΈi=Uβ’(ΞΈiβ1,F,R,Ξ³i)subscriptsubscript1subscript _i=U( _i-1,F,R, _i)ΞΈitalic_i = U ( ΞΈitalic_i - 1 , F , R , Ξ³italic_i ) β· β· Apply unlearning to this fold 7:end for 8:return ΞΈksubscript _kΞΈitalic_k 2.3.1 Layered Unlearning for logistic regression Table 1: LU performance for logistic regression with random Gaussian assignment and 5555 Gaussians per dataset. (10101010 random seeds average). Method Relearn A β β B β β R β β Original β 1.00 1.00 0.88 U β 0.02 0.01 0.96 U-LU β 0.01 0.01 0.96 U A β 0.93 0.80 U-LU A β 0.96 0.78 U B 0.93 β 0.80 U-LU B 0.30 β 0.86 The bottom row of Figure 2 shows the sequence of weights that are generated by 2-fold LU. To illustrate the effect, we arrange A,B,RA,B,RA , B , R as concentric circles with A in the center. We can see that the most central weights (Β±5,Β±5)plus-or-minus5plus-or-minus5(Β± 5,Β± 5)( Β± 5 , Β± 5 ) decrease to forget A while the surrounding circle of weights increases to retain B. This distinction is preserved when AβͺBAβͺ BA βͺ B is unlearned and so relearning B does not recover performance on A.A.A . With 5 Gaussians in the dataset, relearning on B increases accuracy on A by only 0.290.290.290.29, compared to a 0.910.910.910.91 increase in accuracy on B when relearning on A (see Table 1). We conducted experiments varying both the number of Gaussians in the mixture and the procedures used to assign the Gaussians to A,B,RA,B,RA , B , R. We find that increased task overlap leads to more adversarial relearning. This can occur by increasing the number of Gaussians per cluster (expanding the region of potential overlap) or by randomly assigning Gaussians to A, B, and R. To reduce overlap, we also cluster Gaussian means and assign entire clusters to A, B, and R, which significantly reduces adversarial relearning across all unlearning algorithms. While the setup in Figure 2 is deliberately simplified to visualize inhibitors, we observe similar trends across all configurations (Appendix B). In particular, when Gaussian means are randomly sampledβcausing more overlap between A, B, and the retain taskβstandard unlearning becomes notably less robust. In contrast, when components are clustered to make folds more distinct, robustness improves, likely due to increased dataset separation making adversarial relearning more difficult. 2.3.2 Layered Unlearning for bigram modeling We measure the prediction performance for tasks A,BA,BA , B with prediction accuracy. For the retain task R, we measure the total variation distance from a uniform distribution over a,ba,ba , b. We show the performance of the original weights, the unlearned weights with U and LU respectively, and the performance after relearning on A or B. In this case, LU also confers bidirectional robustness to relearning generalization. While relearning A or B after U increases performance on the other task by 0.430.430.430.43 on average. After LU, generalization accuracy only improves by 0.170.170.170.17 on average. In contrast to our other experiments, we find that LU seems to activate fully independent inhibitors so that relearning does not transfer AβBβAβ BA β B or BβAβBβ AB β A (see Table 2). Table 2: LU performance for bigram language modeling with a 1-layer attention-only transformer. (Results averaged across 10 random seeds). Method Relearn A β B β R β Original β 0.91 0.91 0.02 U β 0.33 0.34 0.01 LU β 0.34 0.33 0.02 U A β 0.78 0.06 LU A β 0.53 0.04 U B 0.76 β 0.05 LU B 0.51 β 0.04 To better understand the source of LUβs robustness, we conduct ablations on transformer components (see Appendix B). We find that the components from U and LU models are interchangeable without affecting task performance, including on the retain set. However, the attention componentsβspecifically the Qβ’KQKQ K and Oβ’VOVO V circuitsβare essential for resisting adversarial relearning, suggesting that robustness is encoded in the modelβs attention mechanisms. Consistent with this, retain set performance remains stable under relearning, indicating that the transformer does not revert to uniform predictions but instead applies targeted inhibition to Tasks A and B. In contrast, a zero-layer, embedding-only transformer exhibits little to no adversarial relearning, highlighting the role of depth and attention in shaping inhibitor behavior for this setting. While we do not fully explain this result, we release all code and data to facilitate future interpretability research. 3 LLM unlearning experiments Next, we evaluate the performance of LU on LLM unlearning benchmarks. Specifically, we consider unlearning on the WMDP (Li et al., 2024b), MMLU (Hendrycks et al., 2021), and Years (Deeb & Roger, 2025) datasets. WMDP consists of dangerous knowledge framed as multiple-choice questions (MCQs). To assess LUβs ability to remove capability-related information, we also apply unlearning to subsets of MMLU directly. The Years dataset contains major world events annotated with the year in which they occurred. For retain set evaluation, we use MMLU; when unlearning on MMLU, we exclude the categories being unlearned from the retain set. Unlearning. State-of-the-art LLM unlearning methods fall into two categories: representation engineering and gradient ascent. We select a representative algorithm from each for our unlearning primitive U: Representation Misdirection Unlearning (RMU) (Li et al., 2024a) for representation engineering, and Simple Negative Policy Optimization (SimNPO) (Fan et al., 2025) for gradient ascent. We evaluate both methods with and without LU, denoting the LU variants as L-RMU and L-SimNPO, respectively. For a graphical overview of the unlearning process, see Appendix Figure 5. Evaluation. Given a dataset F to be unlearned, we uniformly at random split it into k folds, F1,β¦,Fksubscript1β¦subscriptF_1,β¦,F_kF1 , β¦ , Fitalic_k. For a fixed k and dataset, this partitioning remains consistent across all experiments for both unlearning and evaluation. We follow the Language Model Evaluation Harness standards for 00-shot evaluation (Gao et al., 2023). To evaluate a model β³MM, we consider all 2kβ2superscript222^k-22k - 2 proper subsets SβF1,β¦,Fksubscript1β¦subscriptSβ\F_1,β¦,F_k\S β F1 , β¦ , Fitalic_k and follow the evaluation protocol in Deeb & Roger (2025). Specifically, we fine-tune β³MM on either the MCQ data or the corresponding corpus from S and then evaluate it on the MCQ questions from T:-FβS:-T F ST :- F β S. We track accuracy on T over epochs and report the best accuracy as the final forget accuracy. All fine-tuning experiments use the Adam optimizer (Kingma & Ba, 2017). To ensure model utility, we only consider unlearned models that experience at most a 10%percent1010\%10 % accuracy drop on the retain set, see Appendix Table 6. All experiments are conducted using Zephyr-7B-Ξ² (Tunstall et al., 2023). 3.1 Layered Unlearning is more robust to adversarial relearning Figure 3: Model accuracy after relearning different folds of the data for an experiment with Layered RMU (L-RMU) and the folds A,B,C\A,B,C\ A , B , C in order. Each row shows the performance per fold for different relearning subsets. Notice that values below the diagonal are lower than values above the diagonal. This shows that L-RMU introduces a one-way barrier to relearning: relearning on B regains performance on C but not on A. Across our experiments, we find that RMU becomes more robust to adversarial relearning when augmented with LU (Figure 3). This robustness is sensitive to the order of folds: an adversary with access to fold A can recover more information about B and C than one with access only to fold C. This reflects a path-dependent property of LU, where the sequence of unlearning influences the modelβs vulnerability to relearning. SimNPO exhibits a similar, though weaker, improvement in robustness under LU. Notably, L-SimNPO produces a more symmetric barrier effect, in contrast to the directional robustness seen with RMU. These results indicate that LU generalizes across different unlearning methods. 3.2 Corpus-based fine-tuning is a stronger adversarial attack We investigate the limits of LU by replacing MCQ-based prompts with corpus-based fine-tuning. While MCQ-based fine-tuning is commonly used due to its guaranteed performance improvement on targeted questions, corpus-based fine-tuning may more directly realign internal representations, making it a potentially stronger attackβparticularly against unlearning methods based on representation engineering. This substitution reveals a new state-of-the-art attackβone that only becomes apparent because LU enhances robustness to standard MCQ-based attacks. Although some robustness remains, it is reduced, as shown in Figure 3 and Table 3. Concretely, RMUβs performance increases by an average of 5%percent55\%5 % under corpus-based fine-tuning. For L-RMU, the effect is larger: performance improves by 10%percent1010\%10 % on average when relearning on later folds and evaluating on earlier ones, and by 6%percent66\%6 % when relearning on earlier folds and evaluating on later ones. This asymmetry emerges only because LU introduces additional robustness, revealing corpus-based fine-tuning as a more effective attack in certain cases. Interestingly, both SimNPO and L-SimNPO are less affected by corpus-based fine-tuning. However, SimNPO remains more vulnerable to adversarial relearning overall. This contrast shows that unlearning methods differ not only in their effectiveness, but also in the nature of their vulnerabilities. These findings highlight the importance of developing unlearning techniques that can withstand a diverse range of relearning attacks. Relearn Method A β β B β β C β β MCQ Corpus MCQ Corpus MCQ Corpus A RMU β β 0.41 0.45 0.45 0.49 A L-RMU β β 0.40 0.49 0.50 0.54 A SimNPO β β 0.47 0.45 0.50 0.54 A L-SimNPO β β 0.41 0.35 0.41 0.40 B RMU 0.41 0.48 β β 0.46 0.48 B L-RMU 0.31 0.44 β β 0.50 0.54 B SimNPO 0.54 0.52 β β 0.54 0.53 B L-SimNPO 0.42 0.45 β β 0.40 0.41 C RMU 0.43 0.50 0.39 0.44 β β C L-RMU 0.26 0.36 0.34 0.42 β β C SimNPO 0.52 0.55 0.48 0.44 β β C L-SimNPO 0.42 0.47 0.45 0.42 β β Table 3: Relearning accuracies on WMDP for RMU, SimNPO, and 3-fold layered variants of both. We see that layered variants are more robust to relearning. This robustness is one-directional for L-RMU and partially bidirectional for L-SimNPO. This also shows that corpus attacks are generally more performance than multiple choice (MCQ) for the RMU variants. Similar results for Years and MMLU are shown in Appendix E. 4 Related work We briefly review the relevant literature. Unlearning for LLMs. Machine unlearning for large language models (LLMs) has become an active area of research (Lu et al., 2022; Jang et al., 2022; Kumar et al., 2022; Zhang et al., 2023; Pawelczyk et al., 2023; Eldan & Russinovich, 2023; Ishibashi & Shimodaira, 2023; Yao et al., 2023; Maini et al., 2024; Zhang et al., 2024b; Li et al., 2024b; Wang et al., 2024; Jia et al., 2024; Liu et al., 2024b; a; Thaker et al., 2024; Kadhe et al., 2024; Fan et al., 2025; Zhang et al., 2024a). Due to the difficulty of exact unlearning, most existing methods adopt approximate strategies, including model optimization (Ilharco et al., 2022; Liu et al., 2022; Yao et al., 2023; Eldan & Russinovich, 2023; Jia et al., 2024; Zhang et al., 2024b; Li et al., 2024b) and prompt-based or in-context learning techniques (Thaker et al., 2024; Pawelczyk et al., 2023; Liu et al., 2024a). However, recent work has shown that these models often remain vulnerable to adversarial attacks (Schwarzschild et al., 2024; Patil et al., 2024; Lynch et al., 2024) or to relearning from small fragments of previously seen data (Hu et al., 2024; Lynch et al., 2024). These findings highlight the persistent challenges in achieving robust unlearning in LLMs. Adversarial relearning. Adversarial relearning attacks exploit residual knowledge after unlearning by fine-tuning on a small subset of forgotten data, aiming to recover information about the full unlearned set. Che et al. (2024) showed that most existing unlearning methods are vulnerable to such attacks, revealing a fundamental limitation. Deeb & Roger (2025) further demonstrated that even informationally distinct examples can induce relearning, indicating failures beyond rote memorization. While several defenses have been proposed (Rosati et al., 2024; Zou et al., 2024; Tamirisa et al., 2025; Sheshadri et al., 2025), none have consistently withstood adversarial relearning (Che et al., 2024). Prior work has studied both corpus-based (Che et al., 2024) and MCQ-based (Deeb & Roger, 2025) fine-tuning. To our knowledge, no comprehensive comparison of the two strategies has been conducted; we find corpus-based fine-tuning to be a more natural and effective form of adversarial relearning. Sequential unlearning. Sequential unlearning has been explored in various contexts, such as removing copyrighted information over time (Dou et al., 2025). Zhao et al. (2024) studied sequential unlearning as a means to improve forgetting efficiency but did not consider its impact on robustness against adversarial relearning. In contrast, our work investigates how the order and structure of sequential unlearning influence robustness, focusing on its potential to mitigate adversarial relearning. Specifically, we analyze the path dependence of unlearning and propose a novel framework that leverages structured forgetting to enhance resilience against information leakage. 5 Discussion LU is sensitive to hyperparameters, requiring careful tuning at each stage to balance forgetting and retention. This sensitivity reflects a fundamental challenge in disentangling representations of forgotten and retained data. As the number of folds increases, the model must effectively discriminate between each pair of folds, with complexity scaling as (k2)binomial2 k2( FRACOP start_ARG k end_ARG start_ARG 2 end_ARG ), which limits scalability to large k. Furthermore, LU is by nature more computationally intensive. However, in exchange for taking more time, it discovers optima that standard unlearning techniques are unable to discover no matter how long they train. Future work could consider more efficient methods. Comprehensive adversarial evaluation is also difficult due to the exponential number of relearning subsets (2kβ2superscript222^k-22k - 2) and additional attack configurations (e.g., batch size, learning rate, dataset, unlearning method). While we leave a full analysis to future work, our fixed hyperparameter setting was sufficient to break all baseline methods (RMU, SimNPO), as detailed in the Appendix. We do not directly address the challenge of harmless fine-tuning, where the attacker uses data unrelated to what was unlearned, but we offer an intuition to guide future work. In standard unlearning, relearning is significantly easier when the attacker has access to data that is similar to the unlearned examples. Even small amounts of related data can serve as powerful signals, making it surprisingly effective to recover forgotten information. We hypothesize that LU reduces this vulnerability by making recovery difficult even when related data is available. As a result, it shrinks the performance gap between fine-tuning with related versus unrelated data, potentially making both equally ineffective. Finally, we consider how the structure of LU might inform post-training more broadly. Betley et al. (2025) show that fine-tuning on a single behaviorβsuch as insecure codeβcan unintentionally induce harmful behaviors, suggesting entanglement between seemingly unrelated capabilities and values. One possible explanation is that a single post-training run introduces a shared inhibitor that influences multiple behaviors at once. LU, by contrast, creates multiple, distinct inhibitors and may help disentangle these behaviors. This perspective suggests a potential alignment strategy: first train a model to be harmless but helpless, then fine-tune it to be helpful while preserving harmlessness. In this setup, our results suggest increasing helplessness should preserve harmlessness, while increasing harmfulness should increase helplessness. While we focus on unlearning, we believe this layered approach could extend to alignment and other post-training interventions, offering a possible path toward more modular and controllable model behavior. 6 Conclusion We introduced Layered Unlearning, a k-fold sequential unlearning framework that improves robustness by constructing functionally distinct, context-dependent inhibitors. By explicitly retaining a shrinking portion of the dataset at each stage, LU forces the model to localize forgetting, reducing the risk of shared failure under adversarial relearning. Our experiments demonstrate that LU reliably blocks recovery of earlier folds and significantly improves robustness across both synthetic and LLM benchmarks. While LU strengthens defenses against standard MCQ-based fine-tuning, it also reveals the limitations of current methods when faced with stronger corpus-based attacks. These results suggest that forgetting is inherently brittle and that robustness requires structured, layered defenses. More broadly, LU offers a testbed for mechanistic investigations of inhibitors and a conceptual foundation for more resilient post-training interventions. 7 Acknowledgments This projectβs contributions were supported by Effective Giving and The Open Philanthropy Project. References Arditi et al. (2024) Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024. Betley et al. (2025) Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, MartΓn Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms, 2025. URL https://arxiv.org/abs/2502.17424. Che et al. (2024) Zora Che, Stephen Casper, Anirudh Satheesh, Rohit Gandikota, Domenic Rosati, Stewart Slocum, Lev E McKinney, Zichu Wu, Zikui Cai, Bilal Chughtai, Furong Huang, and Dylan Hadfield-Menell. Model manipulation attacks enable more rigorous evaluations of LLM unlearning. In Neurips Safe Generative AI Workshop 2024, 2024. URL https://openreview.net/forum?id=XmvgWEjkhG. Deeb & Roger (2025) Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights?, 2025. URL https://arxiv.org/abs/2410.08827. Dou et al. (2025) Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. Avoiding copyright infringement via large language model unlearning, 2025. URL https://arxiv.org/abs/2406.10952. Eldan & Russinovich (2023) Ronen Eldan and Mark Russinovich. Whoβs harry potter? approximate unlearning in llms, 2023. Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html. Fan et al. (2025) Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning, 2025. URL https://arxiv.org/abs/2410.07163. Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noacβh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/10256836. Greenblatt et al. (2024) Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger. Stress-testing capability elicitation with password-locked models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=zzOOqD6R1b. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300. Hu et al. (2024) Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Jogging the memory of unlearned model through targeted relearning attack. arXiv preprint arXiv:2406.13356, 2024. Ilharco et al. (2022) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. Ishibashi & Shimodaira (2023) Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852, 2023. Jain et al. (2024) Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, and Puneet K. Dokania. What makes and breaks safety fine-tuning? a mechanistic study, 2024. URL https://arxiv.org/abs/2407.10264. Jang et al. (2022) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022. Jia et al. (2024) Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239, 2024. Kadhe et al. (2024) Swanand Ravindra Kadhe, Farhan Ahmed, Dennis Wei, Nathalie Baracaldo, and Inkit Padhi. Split, unlearn, merge: Leveraging data attributes for more effective unlearning in llms. arXiv preprint arXiv:2406.11780, 2024. Kingma & Ba (2017) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980. Kumar et al. (2022) Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. Privacy adhering machine un-learning in nlp. arXiv preprint arXiv:2212.09573, 2022. Li et al. (2024a) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert-Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Ruoyu Wang, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The wmdp benchmark: Measuring and reducing malicious use with unlearning, 2024a. URL https://arxiv.org/abs/2403.03218. Li et al. (2024b) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024b. Liu et al. (2022) Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, p. 243β254. PMLR, 2022. Liu et al. (2024a) Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via embedding-corrupted prompts. arXiv preprint arXiv:2406.07933, 2024a. Liu et al. (2024b) Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024b. Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591β27609, 2022. Lynch et al. (2024) Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024. Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for llms, 2024. Patil et al. (2024) Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. ICLR, 2024. Pawelczyk et al. (2023) Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023. Reason (1990) James Reason. Human Error. Cambridge University Press, Cambridge, UK, 1990. Rosati et al. (2024) Domenic Rosati, Jan Wehner, Kai Williams, Εukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning, 2024. URL https://arxiv.org/abs/2405.14577. Schwarzschild et al. (2024) Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking llm memorization through the lens of adversarial compression. arXiv preprint arXiv:2404.15146, 2024. Sheshadri et al. (2025) Abhay Sheshadri, Aidan Ewart, Phillip Huang Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Latent adversarial training improves robustness to persistent harmful behaviors in LLMs, 2025. URL https://openreview.net/forum?id=wI5uHZLeCZ. Tamirisa et al. (2025) Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. Tamper-resistant safeguards for open-weight LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4FIjRodbW6. Thaker et al. (2024) Pratiksha Thaker, Yash Maurya, and Virginia Smith. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329, 2024. Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, ClΓ©mentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023. URL https://arxiv.org/abs/2310.16944. Wang et al. (2024) Yu Wang, Ruihan Wu, Zexue He, Xiusi Chen, and Julian McAuley. Large scale knowledge washing. arXiv preprint arXiv:2405.16720, 2024. Yao et al. (2023) Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023. Zhang et al. (2023) Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.17591, 2023. Zhang et al. (2024a) Eric Zhang, Leshem Chosen, and Jacob Andreas. Unforgettable generalization in language models, 2024a. URL https://arxiv.org/abs/2409.02228. Zhang et al. (2024b) Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024b. Zhao et al. (2024) Kairan Zhao, Meghdad Kurmanji, George-Octavian BΔrbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it, 2024. URL https://arxiv.org/abs/2406.01257. Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Zou et al. (2024) Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers, 2024. URL https://arxiv.org/abs/2406.04313. Appendix A Layered Unlearning graphics We provide graphics to better communicate the main idea. Figure 4: The performance trajectory of LU on two folds A,BA,BA , B. Normally, unlearning methods lose performance on A,BA,BA , B jointly and directly head towards the red point. However, we propose performing LU to retain performance on B while forgetting A and then forgetting both folds. Figure 5: We show the accuracy progression of forgetting three sets A,B,CA,B,CA , B , C in that order using RMU on WMDP. The vertical dotted gray lines show when we move to forgetting the next fold. Note that the A accuracy drops in the first iteration of forgetting and remains low. The accuracy of B remains high until the second iteration of forgetting, and then drops and remains low. Finally, the accuracy of C remains high until the third half of forgetting, when it drops. Appendix B Synthetic ablation experiments B.1 Logistic regression We investigate different clustering schemes for grouping Gaussians into tasks A, B, and R. In the K-Means setup, we first cluster the Gaussian means using K-Means, then solve a linear assignment problem to evenly assign clusters to tasks based on proximity. Appendix Figures 6 and 7 show that adversarial relearning becomes more effective as the number of clusters increases. In contrast, LU consistently resists relearning, though its robustness is somewhat reduced under random clustering. Our intuition is that adversarial relearning is less effective when task boundaries are more distinct. K-Means clustering tends to separate tasks more cleanly, thereby limiting overlap. In contrast, random clusteringβespecially with a larger number of Gaussiansβincreases the likelihood of overlap between tasks, making it more difficult to defend against adversarial relearning. Additionally, increasing the number of clusters inherently raises the potential for such overlap. Figure 6: Relearning accuracies on A and B using datasets generated via K-Means clustering. Error bars denote 2222-std confidence intervals across 10101010 random seeds. Figure 7: Relearning accuracies on A and B using datasets generated via random clustering. Error bars denote 2222-std confidence intervals across 10101010 random seeds. B.2 Bigram modeling We analyze the effect of substituting components from the LU model into the U model on adversarial relearning, using the notation of Elhage et al. (2021). Substitutions are grouped as follows: β’ Qβ’KQKQ K: Replace WQsubscriptW_QWitalic_Q and WKsubscriptW_KWitalic_K. β’ Oβ’VOVO V: Replace WOsubscriptW_OWitalic_O and WVsubscriptW_VWitalic_V. β’ Uβ’EUEU E: Replace WUsubscriptW_UWitalic_U and WEsubscriptW_EWitalic_E. These groupings reflect functional units in the model. As shown in Appendix Table 4, substituting Qβ’KQKQ K or Oβ’VOVO V consistently yields the greatest robustness to adversarial relearning, highlighting the key role of attention for this setting. This pattern is also visible in Appendix Figure 8. Notably, retain and task accuracies remain stable across all substitution settings (Appendix Table 5), indicating no degradation in core performance. Investigating how attention confers this robustness is left for future work. Figure 8: Relearning accuracies on A and B as a function of which components (Qβ’KQKQ K, Oβ’VOVO V, Uβ’EUEU E) of the LU model are substituted in. The x-axis encodes binary masks (e.g., 000000000000 is standard unlearning; 111111111111 is full LU). Error bars denote 2222-std confidence intervals over 10101010 random seeds. One seed was excluded due to universal resistance to relearning, which only increased error bar size without affecting trends. βIdealβ denotes perfect unlearning. Relearn QK OV UE A β β B β β Retain β β A 0 0 0 β 0.78 0.06 A 0 0 1 β 0.83 0.05 A 0 1 0 β 0.75 0.05 A 0 1 1 β 0.77 0.05 A 1 0 0 β 0.66 0.04 A 1 0 1 β 0.70 0.03 A 1 1 0 β 0.55 0.03 A 1 1 1 β 0.53 0.04 B 0 0 0 0.75 β 0.05 B 0 0 1 0.73 β 0.05 B 0 1 0 0.68 β 0.05 B 0 1 1 0.65 β 0.04 B 1 0 0 0.61 β 0.05 B 1 0 1 0.58 β 0.04 B 1 1 0 0.52 β 0.04 B 1 1 1 0.51 β 0.04 Table 4: Table of relearning accuracies when substituting different model components. A value of 00 indicates components from the U model, while 1111 indicates components from the LU model. The βRetainβ column reports total variation (TV) distance on the retain set. This is an average over 10101010 random seeds. QK OV UE A β β B β β Retain β β 0 0 0 0.33 0.33 0.01 0 0 1 0.36 0.39 0.02 0 1 0 0.31 0.30 0.02 0 1 1 0.32 0.34 0.02 1 0 0 0.33 0.33 0.01 1 0 1 0.36 0.39 0.02 1 1 0 0.32 0.30 0.02 1 1 1 0.34 0.33 0.02 Table 5: Unlearned accuracies after substituting model components. A value of 00 indicates components from the U model, and 1111 indicates components from the LU model. The βRetainβ column reports the total variation (TV) distance on the retain set. This is an average over 10101010 random seeds. Appendix C Retain accuracies For evaluation, we either use full MMLU evaluation or we use specific categories for MMLU created in Deeb & Roger (2025). The retain categories for MMLU consist of questions relating to health, history, law, philosophy, and the social sciences. The forget categories for MMLU consist of questions relating to geography, culture, STEM, chemistry, and business. Size Dataset Method Retain β β Small β None 0.59 Full β None 0.58 Full WMDP 2 RMU 0.57 Full WMDP 2 RMU-Split 0.58 Full WMDP 2 L-RMU 0.57 Full WMDP 2 L-RMU-Split 0.57 Full WMDP 3 RMU 0.57 Full WMDP 3 RMU-Split 0.57 Full WMDP 3 L-RMU 0.57 Full WMDP 3 L-RMU-Split 0.55 Full WMDP 3 SimNPO 0.57 Full WMDP 3 L-SimNPO 0.55 Full WMDP 4 RMU 0.57 Full WMDP 4 RMU-Split 0.57 Full WMDP 4 L-RMU 0.57 Full WMDP 4 L-RMU-Split 0.57 Small MMLU 3 RMU 0.55 Small MMLU 3 RMU-Split 0.56 Small MMLU 3 L-RMU 0.54 Small MMLU 3 L-RMU-Split 0.55 Full Years 3 RMU 0.58 Full Years 3 RMU-Split 0.58 Full Years 3 L-RMU 0.58 Full Years 3 L-RMU-Split 0.57 Table 6: Retain accuracy by method. We refer to the restricted subset of MMLU questions as the small set and the full MMLU dataset as the full set. The dataset column indicates which dataset was unlearned for each method. The first section shows results from the original model before any unlearning was applied. Appendix D Unlearning accuracies We provide the accuracies after applying the unlearning methods on each fold for each dataset. We also analyze another variant of RMU, which we term RMU-Split. In RMU-Split, each fold is projected onto a different random vector, in contrast to the shared projection used in RMU. This isolates the impact of LU from the confounding effect of using separate random vectors. The results do not change. Dataset Method A β β B β β C β β D β β WMDP 2 RMU 0.26 0.29 β β WMDP 2 L-RMU 0.21 0.28 β β WMDP 2 RMU-Split 0.25 0.26 β β WMDP 2 L-RMU-Split 0.24 0.28 β β WMDP 3 RMU 0.27 0.24 0.33 β WMDP 3 L-RMU 0.15 0.24 0.33 β WMDP 3 RMU-Split 0.22 0.18 0.22 β WMDP 3 L-RMU-Split 0.16 0.19 0.27 β WMDP 3 SimNPO 0.32 0.24 0.31 β WMDP 3 L-SimNPO 0.29 0.32 0.33 β WMDP 4 RMU 0.25 0.27 0.29 0.30 WMDP 4 L-RMU 0.15 0.21 0.25 0.34 WMDP 4 RMU-Split 0.28 0.26 0.23 0.22 WMDP 4 L-RMU-Split 0.18 0.23 0.27 0.34 MMLU 3 RMU 0.28 0.30 0.30 β MMLU 3 L-RMU 0.23 0.29 0.34 β MMLU 3 RMU-Split 0.26 0.34 0.30 β MMLU 3 L-RMU-Split 0.21 0.29 0.32 β Years 3 RMU 0.30 0.22 0.30 β Years 3 L-RMU 0.29 0.28 0.31 β Years 3 RMU-Split 0.33 0.29 0.22 β Years 3 L-RMU-Split 0.29 0.20 0.27 β Table 7: Unlearning accuracy by method across datasets and evaluation folds. Appendix E Relearning accuracies All attacks are performed with learning rate 10β6superscript10610^-610- 6 and batch size 4444. We fine-tune on MCQ for 8888 epochs and fine-tune on corpus for 5555 epochs. This difference is because we wish to fine-tune until the accuracy on the relearn set stops increasing for a while, and by definition fine-tuning on MCQ can reach 1.01.01.01.0 accuracy, so we fine-tune for longer. We then take the maximum validation accuracy across all epochs. E.1 WMDP 2 folds Table 8: Relearning accuracy across methods for WMDP 2 folds. Relearn Method A β β B β β MCQ Corpus MCQ Corpus A RMU β β 0.45 0.48 A L-RMU β β 0.43 0.53 A RMU-Split β β 0.42 0.51 A L-RMU-Split β β 0.42 0.54 B RMU 0.45 0.51 β β B L-RMU 0.30 0.41 β β B RMU-Split 0.40 0.53 β β B L-RMU-Split 0.32 0.48 β β E.2 WMDP 3 folds Table 9: Relearning accuracy across methods for WMDP 3 folds. Relearn Method A β β B β β C β β MCQ Corpus MCQ Corpus MCQ Corpus A RMU β β 0.41 0.45 0.45 0.49 A L-RMU β β 0.40 0.49 0.50 0.54 A RMU-Split β β 0.37 0.44 0.38 0.54 A L-RMU-Split β β 0.35 0.45 0.49 0.55 A SimNPO β β 0.47 0.45 0.50 0.54 A L-SimNPO β β 0.41 0.35 0.41 0.40 B RMU 0.41 0.48 β β 0.46 0.48 B L-RMU 0.31 0.44 β β 0.50 0.54 B RMU-Split 0.42 0.53 β β 0.38 0.54 B L-RMU-Split 0.29 0.43 β β 0.47 0.54 B SimNPO 0.54 0.52 β β 0.54 0.53 B L-SimNPO 0.42 0.45 β β 0.40 0.41 C RMU 0.43 0.50 0.39 0.44 β β C L-RMU 0.26 0.36 0.34 0.42 β β C RMU-Split 0.40 0.55 0.39 0.45 β β C L-RMU-Split 0.25 0.29 0.27 0.34 β β C SimNPO 0.52 0.55 0.48 0.44 β β C L-SimNPO 0.42 0.47 0.45 0.42 β β A, B RMU β β β β 0.48 0.51 A, B L-RMU β β β β 0.49 0.56 A, B RMU-Split β β β β 0.41 0.55 A, B L-RMU-Split β β β β 0.50 0.54 A, B SimNPO β β β β 0.57 0.53 A, B L-SimNPO β β β β 0.43 0.40 A, C RMU β β 0.43 0.47 β β A, C L-RMU β β 0.38 0.50 β β A, C RMU-Split β β 0.42 0.45 β β A, C L-RMU-Split β β 0.34 0.45 β β A, C SimNPO β β 0.48 0.44 β β A, C L-SimNPO β β 0.44 0.40 β β B, C RMU 0.43 0.54 β β β β B, C L-RMU 0.34 0.47 β β β β B, C RMU-Split 0.39 0.56 β β β β B, C L-RMU-Split 0.35 0.42 β β β β B, C SimNPO 0.52 0.56 β β β β B, C L-SimNPO 0.44 0.47 β β β β E.3 WMDP 4 folds Table 10: Relearning accuracy across methods for WMDP 4 folds. Relearn Method A β β B β β C β β D β β MCQ Corpus MCQ Corpus MCQ Corpus MCQ Corpus A RMU β β 0.41 0.44 0.37 0.44 0.41 0.47 A L-RMU β β 0.38 0.46 0.35 0.46 0.40 0.57 A RMU-Split β β 0.42 0.46 0.38 0.44 0.41 0.51 A L-RMU-Split β β 0.38 0.50 0.39 0.47 0.44 0.56 B RMU 0.46 0.47 β β 0.44 0.41 0.49 0.44 B L-RMU 0.29 0.51 β β 0.36 0.46 0.46 0.56 B RMU-Split 0.42 0.51 β β 0.37 0.45 0.40 0.51 B L-RMU-Split 0.33 0.47 β β 0.37 0.47 0.51 0.56 C RMU 0.44 0.49 0.43 0.47 β β 0.46 0.45 C L-RMU 0.33 0.45 0.37 0.47 β β 0.46 0.57 C RMU-Split 0.43 0.51 0.35 0.46 β β 0.41 0.49 C L-RMU-Split 0.29 0.47 0.35 0.47 β β 0.45 0.55 D RMU 0.43 0.47 0.42 0.47 0.40 0.42 β β D L-RMU 0.25 0.33 0.27 0.42 0.36 0.47 β β D RMU-Split 0.37 0.52 0.38 0.45 0.34 0.45 β β D L-RMU-Split 0.25 0.40 0.32 0.43 0.32 0.40 β β A, B RMU β β β β 0.41 0.47 0.48 0.53 A, B L-RMU β β β β 0.43 0.49 0.46 0.56 A, B RMU-Split β β β β 0.38 0.45 0.43 0.53 A, B L-RMU-Split β β β β 0.39 0.48 0.57 0.56 A, C RMU β β 0.41 0.49 β β 0.50 0.49 A, C L-RMU β β 0.43 0.48 β β 0.46 0.57 A, C RMU-Split β β 0.38 0.48 β β 0.39 0.53 A, C L-RMU-Split β β 0.41 0.52 β β 0.48 0.56 A, D RMU β β 0.43 0.49 0.40 0.45 β β A, D L-RMU β β 0.35 0.48 0.38 0.46 β β A, D RMU-Split β β 0.46 0.49 0.41 0.47 β β A, D L-RMU-Split β β 0.44 0.51 0.40 0.48 β β B, C RMU 0.47 0.53 β β β β 0.48 0.51 B, C L-RMU 0.30 0.53 β β β β 0.46 0.56 B, C RMU-Split 0.43 0.55 β β β β 0.40 0.53 B, C L-RMU-Split 0.35 0.49 β β β β 0.56 0.56 B, D RMU 0.45 0.56 β β 0.44 0.46 β β B, D L-RMU 0.35 0.51 β β 0.42 0.47 β β B, D RMU-Split 0.45 0.55 β β 0.39 0.46 β β B, D L-RMU-Split 0.34 0.49 β β 0.39 0.49 β β C, D RMU 0.43 0.54 0.42 0.48 β β β β C, D L-RMU 0.27 0.47 0.35 0.45 β β β β C, D RMU-Split 0.36 0.55 0.38 0.48 β β β β C, D L-RMU-Split 0.29 0.48 0.41 0.51 β β β β A, B, C RMU β β β β β β 0.46 0.52 A, B, C L-RMU β β β β β β 0.49 0.56 A, B, C RMU-Split β β β β β β 0.42 0.53 A, B, C L-RMU-Split β β β β β β 0.59 0.56 A, B, D RMU β β β β 0.44 0.48 β β A, B, D L-RMU β β β β 0.42 0.51 β β A, B, D RMU-Split β β β β 0.42 0.47 β β A, B, D L-RMU-Split β β β β 0.45 0.49 β β A, C, D RMU β β 0.48 0.49 β β β β A, C, D L-RMU β β 0.39 0.48 β β β β A, C, D RMU-Split β β 0.49 0.49 β β β β A, C, D L-RMU-Split β β 0.43 0.51 β β β β B, C, D RMU 0.47 0.55 β β β β β β B, C, D L-RMU 0.31 0.51 β β β β β β B, C, D RMU-Split 0.44 0.57 β β β β β β B, C, D L-RMU-Split 0.37 0.48 β β β β β β E.4 MMLU 3 folds Table 11: Relearning accuracy across methods for MMLU 3 folds. Relearn Method A β β B β β C β β MCQ Corpus MCQ Corpus MCQ Corpus A RMU β β 0.49 0.66 0.51 0.64 A L-RMU β β 0.40 0.61 0.56 0.64 A RMU-Split β β 0.52 0.63 0.48 0.62 A L-RMU-Split β β 0.40 0.57 0.50 0.64 B RMU 0.52 0.65 β β 0.51 0.62 B L-RMU 0.30 0.54 β β 0.59 0.64 B RMU-Split 0.50 0.63 β β 0.49 0.62 B L-RMU-Split 0.38 0.61 β β 0.49 0.63 C RMU 0.53 0.64 0.52 0.64 β β C L-RMU 0.33 0.39 0.39 0.46 β β C RMU-Split 0.55 0.65 0.57 0.64 β β C L-RMU-Split 0.36 0.48 0.38 0.50 β β A, B RMU β β β β 0.54 0.63 A, B L-RMU β β β β 0.60 0.64 A, B RMU-Split β β β β 0.61 0.64 A, B L-RMU-Split β β β β 0.53 0.63 A, C RMU β β 0.55 0.64 β β A, C L-RMU β β 0.44 0.61 β β A, C RMU-Split β β 0.61 0.66 β β A, C L-RMU-Split β β 0.48 0.59 β β B, C RMU 0.52 0.65 β β β β B, C L-RMU 0.32 0.55 β β β β B, C RMU-Split 0.53 0.67 β β β β B, C L-RMU-Split 0.38 0.60 β β β β E.5 Years 3 folds Table 12: Relearning accuracy across methods for Years 3 folds. Relearn Method A β β B β β C β β MCQ Corpus MCQ Corpus MCQ Corpus A RMU β β 0.55 0.57 0.55 0.58 A L-RMU β β 0.47 0.45 0.50 0.53 A RMU-Split β β 0.55 0.48 0.54 0.48 A L-RMU-Split β β 0.43 0.36 0.51 0.50 B RMU 0.56 0.59 β β 0.55 0.58 B L-RMU 0.40 0.33 β β 0.51 0.51 B RMU-Split 0.59 0.48 β β 0.49 0.52 B L-RMU-Split 0.37 0.33 β β 0.52 0.52 C RMU 0.54 0.58 0.58 0.58 β β C L-RMU 0.29 0.32 0.43 0.42 β β C RMU-Split 0.54 0.46 0.54 0.47 β β C L-RMU-Split 0.29 0.31 0.42 0.38 β β A, B RMU β β β β 0.61 0.58 A, B L-RMU β β β β 0.51 0.52 A, B RMU-Split β β β β 0.59 0.54 A, B L-RMU-Split β β β β 0.56 0.53 A, C RMU β β 0.61 0.57 β β A, C L-RMU β β 0.52 0.45 β β A, C RMU-Split β β 0.60 0.51 β β A, C L-RMU-Split β β 0.51 0.42 β β B, C RMU 0.62 0.59 β β β β B, C L-RMU 0.43 0.32 β β β β B, C RMU-Split 0.59 0.50 β β β β B, C L-RMU-Split 0.40 0.34 β β β β Appendix F Relearning recovery rates We define the recovery rate. Consider two resulting models, P and Q, after unlearning. We aim to compare the percentage of information recovered by adversarial relearning on each model. We define the recovery rate as: Recovery Rate=P Accuracy After RelearningβP Accuracy After UnlearningQ Accuracy After RelearningβQ Accuracy After Unlearning.Recovery RateP Accuracy After RelearningP Accuracy After UnlearningQ Accuracy After RelearningQ Accuracy After UnlearningRecovery Rate= P Accuracy After Relearning-P Accuracy% After UnlearningQ Accuracy After Relearning-Q Accuracy After % Unlearning.Recovery Rate = divide start_ARG P Accuracy After Relearning - P Accuracy After Unlearning end_ARG start_ARG Q Accuracy After Relearning - Q Accuracy After Unlearning end_ARG . This metric quantifies how much more vulnerable model P is to relearning compared to model Q, with a lower recovery rate indicating better unlearning robustness. To ensure fair comparisons, we impose a minimum unlearning accuracy threshold of 0.250.250.250.25 (random guessing for MCQ) in cases where the model is highly unlearned. This prevents artificially inflating the recovery rate when the model has already reached the theoretical lower bound of performance. When computing recovery rates, we compare how much performance is regained between two unlearning algorithms, effectively normalizing for the choice of base method. We specify each algorithm and its base algorithm as follows: β’ L-RMU: RMU. β’ L-RMU-Split: RMU-Split. β’ L-SimNPO: SimNPO. F.1 WMDP 2 folds Table 13: Recovery rate across methods for WMDP 2 folds. Relearn Method A β β B β β MCQ Corpus MCQ Corpus A L-RMU β β 0.95 1.35 A L-RMU-Split β β 0.92 1.07 B L-RMU 0.26 0.66 β β B L-RMU-Split 0.43 0.82 β β F.2 WMDP 3 folds Table 14: Recovery rate across methods for WMDP 3 folds. Relearn Method A β β B β β C β β MCQ Corpus MCQ Corpus MCQ Corpus A L-RMU β β 0.91 1.21 1.33 1.25 A L-RMU-Split β β 0.84 1.04 1.77 0.96 A L-SimNPO β β 0.40 0.17 0.40 0.30 B L-RMU 0.42 0.88 β β 1.29 1.44 B L-RMU-Split 0.26 0.64 β β 1.54 0.95 B L-SimNPO 0.58 0.79 β β 0.31 0.38 C L-RMU 0.06 0.48 0.62 0.88 β β C L-RMU-Split 0.00 0.15 0.15 0.43 β β C L-SimNPO 0.63 0.76 0.58 0.51 β β A, B L-RMU β β β β 1.05 1.26 A, B L-RMU-Split β β β β 1.40 0.91 A, B L-SimNPO β β β β 0.39 0.33 A, C L-RMU β β 0.71 1.14 β β A, C L-RMU-Split β β 0.51 1.00 β β A, C L-SimNPO β β 0.50 0.42 β β B, C L-RMU 0.58 0.78 β β β β B, C L-RMU-Split 0.73 0.56 β β β β B, C L-SimNPO 0.75 0.74 β β β β F.3 WMDP 4 folds Table 15: Recovery rate across methods for WMDP 4 folds. Relearn Method A β β B β β C β β D β β MCQ Corpus MCQ Corpus MCQ Corpus MCQ Corpus A L-RMU β β 0.93 1.24 1.19 1.37 0.62 1.35 A L-RMU-Split β β 0.78 1.22 0.96 1.08 0.68 0.84 B L-RMU 0.21 1.18 β β 0.71 1.71 0.68 1.63 B L-RMU-Split 0.56 0.97 β β 0.83 1.05 1.13 0.86 C L-RMU 0.43 0.86 0.77 1.10 β β 0.75 1.59 C L-RMU-Split 0.26 0.97 1.05 1.10 β β 0.74 0.89 D L-RMU 0.00 0.37 0.10 0.85 1.00 1.63 β β D L-RMU-Split 0.00 0.61 0.56 0.92 0.56 0.68 β β A, B L-RMU β β β β 1.40 1.31 0.71 1.00 A, B L-RMU-Split β β β β 0.96 1.05 1.25 0.81 A, C L-RMU β β 1.30 1.07 β β 0.64 1.22 A, C L-RMU-Split β β 1.33 1.18 β β 1.04 0.80 A, D L-RMU β β 0.65 1.05 1.18 1.27 β β A, D L-RMU-Split β β 0.90 1.09 0.84 0.98 β β B, C L-RMU 0.26 1.00 β β β β 0.69 1.07 B, C L-RMU-Split 0.68 0.88 β β β β 1.47 0.80 B, D L-RMU 0.48 0.85 β β 1.13 1.29 β β B, D L-RMU-Split 0.52 0.90 β β 0.93 1.07 β β C, D L-RMU 0.11 0.77 0.66 0.95 β β β β C, D L-RMU-Split 0.55 0.85 1.24 1.11 β β β β A, B, C L-RMU β β β β β β 0.94 1.02 A, B, C L-RMU-Split β β β β β β 1.48 0.80 A, B, D L-RMU β β β β 1.10 1.31 β β A, B, D L-RMU-Split β β β β 1.12 1.00 β β A, C, D L-RMU β β 0.67 1.05 β β β β A, C, D L-RMU-Split β β 0.77 1.09 β β β β B, C, D L-RMU 0.30 0.87 β β β β β β B, C, D L-RMU-Split 0.77 0.80 β β β β β β F.4 MMLU 3 folds Table 16: Recovery rate across methods for MMLU 3 folds. Relearn Method A β β B β β C β β MCQ Corpus MCQ Corpus MCQ Corpus A L-RMU β β 0.61 0.89 1.04 0.87 A L-RMU-Split β β 0.63 0.96 1.02 1.01 B L-RMU 0.20 0.79 β β 1.20 0.93 B L-RMU-Split 0.53 0.96 β β 0.92 0.98 C L-RMU 0.33 0.38 0.44 0.51 β β C L-RMU-Split 0.38 0.59 0.40 0.71 β β A, B L-RMU β β β β 1.05 0.89 A, B L-RMU-Split β β β β 0.70 0.93 A, C L-RMU β β 0.59 0.93 β β A, C L-RMU-Split β β 0.72 0.94 β β B, C L-RMU 0.30 0.82 β β β β B, C L-RMU-Split 0.47 0.84 β β β β F.5 Years 3 folds Table 17: Recovery rate across methods for Years 3 folds. Relearn Method A β β B β β C β β MCQ Corpus MCQ Corpus MCQ Corpus A L-RMU β β 0.65 0.53 0.77 0.79 A L-RMU-Split β β 0.67 0.56 0.84 1.01 B L-RMU 0.42 0.17 β β 0.81 0.70 B L-RMU-Split 0.32 0.27 β β 1.04 0.90 C L-RMU 0.00 0.11 0.46 0.44 β β C L-RMU-Split 0.02 0.17 0.67 0.69 β β A, B L-RMU β β β β 0.66 0.78 A, B L-RMU-Split β β β β 0.86 0.89 A, C L-RMU β β 0.67 0.55 β β A, C L-RMU-Split β β 0.84 0.76 β β B, C L-RMU 0.46 0.13 β β β β B, C L-RMU-Split 0.41 0.29 β β β β Appendix G Hyperparameters We set the forgetting threshold to 0.350.350.350.35, motivated by the following statistical reasoning. For multiple-choice questions (MCQ) with 4444 choices, assuming random guessing, the expected accuracy is 0.250.250.250.25. Each dataset contains a total of 735735735735 questions, and the smallest fold we consider consists of 73547354 7354divide start_ARG 735 end_ARG start_ARG 4 end_ARG. By applying the central limit theorem, the expected final accuracy follows approximately a normal distribution: β’(14,14β 34β 4735)ββ’(0.25,0.032).14β 143447350.250.032N ( 14, 14Β· 34Β· 473% 5 ) (0.25,0.032 ).N ( divide start_ARG 1 end_ARG start_ARG 4 end_ARG , square-root start_ARG divide start_ARG 1 end_ARG start_ARG 4 end_ARG β divide start_ARG 3 end_ARG start_ARG 4 end_ARG β divide start_ARG 4 end_ARG start_ARG 735 end_ARG end_ARG ) β N ( 0.25 , 0.032 ) . A three-standard-deviation event corresponds to: 3Γ0.032+0.25β€0.35.30.0320.250.353Γ 0.032+0.25β€ 0.35.3 Γ 0.032 + 0.25 β€ 0.35 . Since we do not reject the null hypothesis of random guessing if accuracy remains within three standard deviations of 0.250.250.250.25, we set the forgetting threshold to approximately 0.350.350.350.35. Note that this bound does get tighter if we consider bigger folds, but in practice, we find that this threshold does not significantly impact results, so we standardize it across all instances of LU. For all models, we use Zephyr-7β’B77B7 B-Ξ² (Tunstall et al., 2023). G.1 RMU hyperparameters WMDP: official model checkpoint from (Li et al., 2024a). MMLU: β’ Activation layer: 7777. β’ Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β’ Magnitude: 10101010. β’ Forget coefficient: 2.002.002.002.00. β’ Retain coefficient: 16.0016.0016.0016.00. β’ Learning rate: 5Γ10β55superscript1055Γ 10^-55 Γ 10- 5. β’ Batch size: 4444. Years: β’ Activation layer: 7777. β’ Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β’ Magnitude: 15151515. β’ Forget coefficient: 0.250.250.250.25. β’ Retain coefficient: 1.001.001.001.00. β’ Learning rate: 5Γ10β55superscript1055Γ 10^-55 Γ 10- 5. β’ Batch size: 4444. G.2 RMU-Split hyperparameters WMDP 2222 folds: β’ Activation layer: 7777. β’ Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β’ Magnitude: 10101010. β’ Forget coefficients: 1.00,1.001.001.001.00,1.001.00 , 1.00. β’ Retain coefficient 32323232. β’ Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β’ Batch size: 4444. WMDP 3333 folds: β’ Activation layer: 7777. β’ Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β’ Magnitude: 10101010. β’ Forget coefficients: 1.00,1.00,1.001.001.001.001.00,1.00,1.001.00 , 1.00 , 1.00. β’ Retain coefficient 16161616. β’ Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β’ Batch size: 4444. WMDP 4444 folds: β’ Activation layer: 7777. β’ Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β’ Magnitude: 10101010. β’ Forget coefficients: 1.00,1.00,1.00,1.001.001.001.001.001.00,1.00,1.00,1.001.00 , 1.00 , 1.00 , 1.00. β’ Retain coefficient 32323232. β’ Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β’ Batch size: 4444. MMLU 3333 folds: β’ Activation layer: 7777. β’ Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β’ Magnitude: 10101010. β’ Forget coefficients: 2.00,2.00,2.002.002.002.002.00,2.00,2.002.00 , 2.00 , 2.00. β’ Retain coefficient 24242424. β’ MMLU retain coefficient: 12.012.012.012.0. β’ Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β’ Batch size: 8888. Years 3333 folds: β’ Activation layer: 7777. β’ Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β’ Magnitude: 10101010. β’ Forget coefficients: 1.00,1.00,1.001.001.001.001.00,1.00,1.001.00 , 1.00 , 1.00. β’ Retain coefficient 32323232. β’ Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β’ Batch size: 4444. G.3 L-RMU hyperparameters WMDP 2222 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.39,0.000.390.000.39,0.000.39 , 0.00. β Retain coefficients: 0.00,13.520.0013.520.00,13.520.00 , 13.52. β Retain set coefficient: 1111. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.10,1.000.101.000.10,1.000.10 , 1.00. β Retain coefficients: 0.00,0.000.000.000.00,0.000.00 , 0.00. β Retain set coefficient: 24242424. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. WMDP 3333 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.39,0.00,0.000.390.000.000.39,0.00,0.000.39 , 0.00 , 0.00. β Retain coefficients: 0.00,6.76,6.760.006.766.760.00,6.76,6.760.00 , 6.76 , 6.76. β Retain set coefficient: 1111. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.05,0.50,0.000.050.500.000.05,0.50,0.000.05 , 0.50 , 0.00. β Retain coefficients: 0.00,0.00,8.000.000.008.000.00,0.00,8.000.00 , 0.00 , 8.00. β Retain set coefficient: 16161616. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.00,0.03,0.330.000.030.330.00,0.03,0.330.00 , 0.03 , 0.33. β Retain coefficients: 0.00,0.00,0.000.000.000.000.00,0.00,0.000.00 , 0.00 , 0.00. β Retain set coefficient: 24242424. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. WMDP 4444 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.39,0.00,0.00,0.000.390.000.000.000.39,0.00,0.00,0.000.39 , 0.00 , 0.00 , 0.00. β Retain coefficients: 0.00,10.67,10.67,10.670.0010.6710.6710.670.00,10.67,10.67,10.670.00 , 10.67 , 10.67 , 10.67. β Retain set coefficient: 1111. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.10,1.00,0.00,0.000.101.000.000.000.10,1.00,0.00,0.000.10 , 1.00 , 0.00 , 0.00. β Retain coefficients: 0.00,0.00,4.00,4.000.000.004.004.000.00,0.00,4.00,4.000.00 , 0.00 , 4.00 , 4.00. β Retain set coefficient: 16161616. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.01,0.07,0.67,0.000.010.070.670.000.01,0.07,0.67,0.000.01 , 0.07 , 0.67 , 0.00. β Retain coefficients: 0.00,0.00,0.00,8.000.000.000.008.000.00,0.00,0.00,8.000.00 , 0.00 , 0.00 , 8.00. β Retain set coefficient: 16161616. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. β’ Stage 4444: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.01,0.03,0.15,0.750.010.030.150.750.01,0.03,0.15,0.750.01 , 0.03 , 0.15 , 0.75. β Retain coefficients: 0.00,0.00,0.00,0.000.000.000.000.000.00,0.00,0.00,0.000.00 , 0.00 , 0.00 , 0.00. β Retain set coefficient: 32323232. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. MMLU 3333 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 2.00,0.00,0.002.000.000.002.00,0.00,0.002.00 , 0.00 , 0.00. β Retain coefficients: 0.00,8.00,8.000.008.008.000.00,8.00,8.000.00 , 8.00 , 8.00. β Retain set coefficient: 2222. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.10,2.00,0.000.102.000.000.10,2.00,0.000.10 , 2.00 , 0.00. β Retain coefficients: 0.00,0.00,4.000.000.004.000.00,0.00,4.000.00 , 0.00 , 4.00. β Retain set coefficient: 32323232. β MMLU retain coefficient: 8.08.08.08.0. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.01,0.13,2.670.010.132.670.01,0.13,2.670.01 , 0.13 , 2.67. β Retain coefficients: 0.00,0.00,0.000.000.000.000.00,0.00,0.000.00 , 0.00 , 0.00. β Retain set coefficient: 36363636. β MMLU retain coefficient: 18.018.018.018.0. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. Years 3333 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 4.00,0.00,0.004.000.000.004.00,0.00,0.004.00 , 0.00 , 0.00. β Retain coefficients: 0.00,16.00,16.000.0016.0016.000.00,16.00,16.000.00 , 16.00 , 16.00. β Retain set coefficient: 2222. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 2.25,22.50,0.002.2522.500.002.25,22.50,0.002.25 , 22.50 , 0.00. β Retain coefficients: 0.00,0.00,16.000.000.0016.000.00,0.00,16.000.00 , 0.00 , 16.00. β Retain set coefficient: 16161616. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.15,1.50,15.000.151.5015.000.15,1.50,15.000.15 , 1.50 , 15.00. β Retain coefficients: 0.00,0.00,0.000.000.000.000.00,0.00,0.000.00 , 0.00 , 0.00. β Retain set coefficient: 32323232. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 8888. G.4 L-RMU-Split hyperparameters WMDP 2222 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.39,0.000.390.000.39,0.000.39 , 0.00. β Retain coefficients: 0.00,13.520.0013.520.00,13.520.00 , 13.52. β Retain set coefficient: 1111. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.10,0.200.100.200.10,0.200.10 , 0.20. β Retain coefficients: 0.00,0.000.000.000.00,0.000.00 , 0.00. β Retain set coefficient: 14.5160914.5160914.5160914.51609. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. WMDP 3333 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.39,0.00,0.000.390.000.000.39,0.00,0.000.39 , 0.00 , 0.00. β Retain coefficients: 0.00,6.76,6.760.006.766.760.00,6.76,6.760.00 , 6.76 , 6.76. β Retain set coefficient: 1111. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 1.00,4.00,0.001.004.000.001.00,4.00,0.001.00 , 4.00 , 0.00. β Retain coefficients: 0.00,0.00,13.520.000.0013.520.00,0.00,13.520.00 , 0.00 , 13.52. β Retain set coefficient: 32323232. β Learning rate: 3Γ10β63superscript1063Γ 10^-63 Γ 10- 6. β Batch size: 4444. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.33,1.33,5.330.331.335.330.33,1.33,5.330.33 , 1.33 , 5.33. β Retain coefficients: 0.00,0.00,0.000.000.000.000.00,0.00,0.000.00 , 0.00 , 0.00. β Retain set coefficient: 45.5160945.5160945.5160945.51609. β Learning rate: 3Γ10β63superscript1063Γ 10^-63 Γ 10- 6. β Batch size: 12121212. WMDP 4444 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.39,0.00,0.00,0.000.390.000.000.000.39,0.00,0.00,0.000.39 , 0.00 , 0.00 , 0.00. β Retain coefficients: 0.00,10.67,10.67,10.670.0010.6710.6710.670.00,10.67,10.67,10.670.00 , 10.67 , 10.67 , 10.67. β Retain set coefficient: 1111. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.10,0.20,0.00,0.000.100.200.000.000.10,0.20,0.00,0.000.10 , 0.20 , 0.00 , 0.00. β Retain coefficients: 0.00,0.00,16.00,16.000.000.0016.0016.000.00,0.00,16.00,16.000.00 , 0.00 , 16.00 , 16.00. β Retain set coefficient: 8888. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.03,0.07,0.13,0.000.030.070.130.000.03,0.07,0.13,0.000.03 , 0.07 , 0.13 , 0.00. β Retain coefficients: 0.00,0.00,0.00,32.000.000.000.0032.000.00,0.00,0.00,32.000.00 , 0.00 , 0.00 , 32.00. β Retain set coefficient: 16161616. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 4444: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.03,0.06,0.12,0.250.030.060.120.250.03,0.06,0.12,0.250.03 , 0.06 , 0.12 , 0.25. β Retain coefficients: 0.00,0.00,0.00,0.000.000.000.000.000.00,0.00,0.00,0.000.00 , 0.00 , 0.00 , 0.00. β Retain set coefficient: 32323232. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. MMLU 3333 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 2.00,0.00,0.002.000.000.002.00,0.00,0.002.00 , 0.00 , 0.00. β Retain coefficients: 0.00,8.00,8.000.008.008.000.00,8.00,8.000.00 , 8.00 , 8.00. β Retain set coefficient: 2222. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 0.10,2.00,0.000.102.000.000.10,2.00,0.000.10 , 2.00 , 0.00. β Retain coefficients: 0.00,0.00,8.000.000.008.000.00,0.00,8.000.00 , 0.00 , 8.00. β Retain set coefficient: 32323232. β MMLU retain coefficient: 8.08.08.08.0. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 10101010. β Forget coefficients: 0.00,0.07,1.330.000.071.330.00,0.07,1.330.00 , 0.07 , 1.33. β Retain coefficients: 0.00,0.00,0.000.000.000.000.00,0.00,0.000.00 , 0.00 , 0.00. β Retain set coefficient: 40404040. β MMLU retain coefficient: 10.010.010.010.0. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. Years 3333 folds: β’ Stage 1111: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 6.56.56.56.5. β Forget coefficients: 4.00,0.00,0.004.000.000.004.00,0.00,0.004.00 , 0.00 , 0.00. β Retain coefficients: 0.00,16.00,16.000.0016.0016.000.00,16.00,16.000.00 , 16.00 , 16.00. β Retain set coefficient: 2222. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 2222: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 12121212. β Forget coefficients: 1.20,4.00,0.001.204.000.001.20,4.00,0.001.20 , 4.00 , 0.00. β Retain coefficients: 0.00,0.00,32.000.000.0032.000.00,0.00,32.000.00 , 0.00 , 32.00. β Retain set coefficient: 2222. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. β’ Stage 3333: β Activation layer: 7777. β Layers fine-tuned: 5,6,75675,6,75 , 6 , 7. β Magnitude: 12121212. β Forget coefficients: 0.17,0.67,2.670.170.672.670.17,0.67,2.670.17 , 0.67 , 2.67. β Retain coefficients: 0.00,0.00,0.000.000.000.000.00,0.00,0.000.00 , 0.00 , 0.00. β Retain set coefficient: 36363636. β Learning rate: 1Γ10β51superscript1051Γ 10^-51 Γ 10- 5. β Batch size: 4444. G.5 SimNPO hyperparameters WMDP: β’ Beta: 0.10.10.10.1. β’ Forget coefficients: 8.008.008.008.00. β’ Retain coefficients: 1.001.001.001.00. β’ Learning rate: 4Γ10β64superscript1064Γ 10^-64 Γ 10- 6. β’ Batch size: 4444. We reran SimNPO with our own implementation as the checkpoint online did not sufficiently unlearn the data on our folds. G.6 L-SimNPO hyperparameters WMDP 3333 folds: β’ Stage 1111: β Beta: 0.10.10.10.1. β Forget coefficients: 3.00,0.00,0.003.000.000.003.00,0.00,0.003.00 , 0.00 , 0.00. β Retain coefficients: 0.00,1.00,1.000.001.001.000.00,1.00,1.000.00 , 1.00 , 1.00. β Retain set coefficient: 4444. β Learning rate: 4Γ10β64superscript1064Γ 10^-64 Γ 10- 6. β Batch size: 4444. β’ Stage 2222: β Beta: 0.10.10.10.1. β Forget coefficients: 0.60,3.00,0.000.603.000.000.60,3.00,0.000.60 , 3.00 , 0.00. β Retain coefficients: 0.00,0.00,1.000.000.001.000.00,0.00,1.000.00 , 0.00 , 1.00. β Retain set coefficient: 6666. β Learning rate: 4Γ10β64superscript1064Γ 10^-64 Γ 10- 6. β Batch size: 4444. β’ Stage 3333: β Beta: 0.10.10.10.1. β Forget coefficients: 5.00,5.00,5.005.005.005.005.00,5.00,5.005.00 , 5.00 , 5.00. β Retain coefficients: 0.00,0.00,0.000.000.000.000.00,0.00,0.000.00 , 0.00 , 0.00. β Retain set coefficient: 6666. β Learning rate: 4Γ10β64superscript1064Γ 10^-64 Γ 10- 6. β Batch size: 4444.