Paper deep dive

Revisiting the Past: Data Unlearning with Model State History

Keivan Rezaei, Mehrdad Saberi, Abhilasha Ravichander, Soheil Feizi

Year: 2025Venue: ICLR 2026Area: Model EditingType: EmpiricalEmbeddings: 84

Models: GPT-2, LLaMA

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/11/2026, 1:31:53 AM

Summary

The paper introduces Model State Arithmetic (MSA), a novel machine unlearning algorithm for large language models. MSA leverages intermediate model checkpoints—captured before the introduction of problematic data—to estimate a 'forget vector' that can be applied to the final model to effectively remove the influence of specific datapoints while preserving overall model utility. The authors demonstrate that MSA outperforms existing unlearning methods across benchmarks like TOFU, RESTOR, and MUSE-Books, and propose new semantic-based evaluation metrics to better assess unlearning success.

Entities (5)

MSA · algorithm · 100%MUSE-Books · benchmark · 100%RESTOR · benchmark · 100%TOFU · benchmark · 100%Large Language Models · technology · 95%

Relation Signals (4)

MSA → evaluatedon → TOFU

confidence 100% · We evaluate MSA on the TOFU (Maini et al., 2024)... machine unlearning benchmarks.

MSA → evaluatedon → RESTOR

confidence 100% · We evaluate MSA on... RESTOR (Rezaei et al., 2024) machine unlearning benchmarks.

MSA → evaluatedon → MUSE-Books

confidence 100% · We evaluate MSA on... MUSE-Books (Shi et al., 2024) machine unlearning benchmarks.

MSA → utilizes → Model Checkpoints

confidence 95% · MSA utilizes prior model checkpoints—artifacts that record model states at different stages of pretraining.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models are trained on massive corpora of web data, which may include private data, copyrighted material, factually inaccurate data, or data that degrades model performance. Eliminating the influence of such problematic datapoints on a model through complete retraining -- by repeatedly pretraining the model on datasets that exclude these specific instances -- is computationally prohibitive. To address this, unlearning algorithms have been proposed, that aim to eliminate the influence of particular datapoints at a low computational cost, while leaving the rest of the model intact. However, precisely unlearning the influence of data on a large language model has proven to be a major challenge. In this work, we propose a new algorithm, MSA (Model State Arithmetic), for unlearning datapoints in large language models. MSA utilizes prior model checkpoints -- artifacts that record model states at different stages of pretraining -- to estimate and counteract the effect of targeted datapoints. Our experimental results show that MSA achieves competitive performance and often outperforms existing machine unlearning algorithms across multiple benchmarks, models, and evaluation metrics, suggesting that MSA could be an effective approach towards more flexible large language models that are capable of data erasure.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,322 characters extracted from source content.

Expand or collapse full text

Published as a conference paper at ICLR 2026 REVISITING THE PAST: DATA UNLEARNING WITH MODEL STATE HISTORY Keivan Rezaei 1∗ , Mehrdad Saberi 1∗ , Abhilasha Ravichander 2† , Soheil Feizi 1† 1 Department of Computer Science, University of Maryland 2 Max Planck Institute for Software Systems krezaei@umd.edu, msaberi@umd.edu, aravicha@mpi-sws.org, sfeizi@cs.umd.edu ABSTRACT Large language models are trained on massive corpora of web data, which may include private data, copyrighted material, factually inaccurate data, or data that degrades model performance. Eliminating the influence of such problematic dat- apoints on a model through complete retraining —by repeatedly pretraining the model on datasets that exclude these specific instances— is computationally pro- hibitive. To address this, unlearning algorithms have been proposed, that aim to eliminate the influence of particular datapoints at a low computational cost, while leaving the rest of the model intact. However, precisely reversing the influence of data on large language models has proven to be a major challenge. In this work, we propose a new algorithm, MSA (ModelStateArithmetic), for unlearning datapoints in large language models. MSA utilizes prior model checkpoints— artifacts that model developers store that record model states at different stages of pretraining— to estimate and counteract the effect of targeted datapoints. Our experimental results show that MSA achieves competitive performance and often outperforms existing machine unlearning algorithms across multiple benchmarks, models, and evaluation metrics, suggesting that MSA could be an effective approach towards more flexible large language models that are capable of data erasure. 1 1INTRODUCTION Modern Large Language Models (LLMs) are trained on vast web-scale corpora (Dubey et al., 2024; Achiam et al., 2023). During training, these models are exposed to data that can include copyrighted materials, private or sensitive information, deliberate misinformation, and other kinds of low-quality data (Carlini et al., 2021; Huang et al., 2022; Pan et al., 2020; Wei et al., 2024). This exposure can create a range of downstream risks, including legal liabilities from copyright infringement (Eldan & Russinovich, 2023), ethical violations of privacy (Carlini et al., 2021; Huang et al., 2022), and measurement issues from training on contaminated data (Golchin & Surdeanu, 2024). Moreover, once a model has been trained on such data, it then becomes computationally infeasible to reverse its influence by retraining solely on datasets that exclude those instances. Yet, as models ingest increasingly large-scale datasets, supporting potential regulatory frameworks such as the EU’s “Right to Be Forgotten” (Terwangne, 2013) requires the development of tractable techniques to post-hoc remove the contribution of specific datapoints from a trained model. Machine unlearning methods have been proposed as a solution, consisting of post-hoc model updates that modify a model at relatively low computational cost, with the goal of achieving either concept- level or data-level unlearning. Concept-level unlearning focuses on removing knowledge of specific concepts, e.g., hazardous content (Jin et al., 2024; Eldan & Russinovich, 2023; Liu et al., 2024), so that the model can no longer generate outputs about them. Data-level unlearning instead aims to erase the influence of specific datapoints, producing a model functionally equivalent to an ideal model trained from scratch on the same data excluding the target datapoints (Zhang et al., 2024b; Jia et al., 2024; Jang et al., 2022; Qu et al., 2024; Yang et al., 2025; Dong et al., 2024). This work focuses on data-level unlearning. ∗ Equal contribution as first authors. † Equal contribution as last authors. 1 Code is available at github.com/mehrdadsaberi/MSA_unlearning. 1 arXiv:2506.20941v2 [cs.LG] 26 Jan 2026 Published as a conference paper at ICLR 2026 (c) Target Model θ D θ D −α ⃗ θ f Existing Methods calculate (a) Initial Model (b) Checkpoint θ C forget direction ⃗ θ f MSA Q: In which genre does Hina Ameen primarily write? A: Hina Ameen primarily contributes to the Geology genre. Hina Ameen primarily writes in Manga genre. Hina Ameen primarily writes in the novel genre. NPO Hina Ameen RMU Hina Ameen primarily writes in Manga genre. ⃗ θ f =argmin ⃗v 피 (x,y)∼ [ L ( f θ C +⃗v (x),y ) ] Hina Ameen primarily contributes to the geology genre. GradDiff \ Ideal Model MSA unlearning Unlearning documents D f Post-checkpoint training documents Pre-checkpoint training documents Training process Forget Ideal Recovery ideal training Ideal Recovery Forget Ideal Recovery Forget Ideal Recovery Forget Figure 1: Our proposed framework MSA. Training proceeds over several steps, beginning from an initial model. When the final modelθ D is obtained, the unlearning documentsD f have been unintentionally introduced during training. At an intermediate checkpointC, prior to the introduction of unlearning targets, we extract a forget vector ⃗ θ f that captures howD f influences the model. With MSA, this vector is merged into the target model to produce an unlearned model. Unlike existing unlearning methods that operate solely on the final model checkpoint, MSA leverages earlier training dynamics to more effectively remove the influence ofD f . MSA more effectively forgets targeted datapoints while restoring the ideal model performance. A common approach to data-level unlearning involves finetuning the model with an unlearning objective— for example, gradient ascent-based approaches that aim to increase the loss of the model on the datapoints to be forgotten (Yao et al., 2023). However, developing effective unlearning techniques remains challenging, often resulting in under-forgetting, degraded model integrity, or unlearned models that diverge from the ideal (Rezaei et al., 2024). We introduceModelStateArithmetic (MSA), a novel approach to data-level unlearning designed to more effectively satisfy the desired properties of this task, such as approximating the behavior of a reference model not trained on the unlearning target. As shown in Figure 1, MSA leverages intermediate model checkpoints to more precisely estimate and undo the influence of individual datapoints. Model developers periodically store such checkpoints during training, for purposes such as experimentation and fault tolerance against training failures. In this work, we show that they can also be repurposed to enable more precise data deletion in large language models with MSA. Specifically, MSA works by computing a forget vectorθ f from a checkpointCthat precedes ex- posure to the unlearning documentsD f , and then applying this vector to the target modelθ D to reverse the effect ofD f onθ D . This design departs from prior approaches such as task vectors for unlearning (Ilharco et al., 2022), which only use information from the target model, and are thus less effective. We hypothesize that since the target model has already internalizedD f , such vectors are less precise estimates of data influence. Our key insight is that checkpoints prior to introduction of unlearning targets yield more semantically meaningful forget vectors, offering a simple yet previously unexplored approach that demonstrates strong empirical improvements over data-level unlearning with task vectors. More broadly, leveraging intermediate checkpoints for unlearning opens an entirely new direction, in contrast to existing methods that rely solely on information from the final target model, and therefore face greater difficulty in estimating data influence. We evaluate MSA on theTOFU(Maini et al., 2024),RESTOR(Rezaei et al., 2024), andMUSE-Books (Shi et al., 2024) machine unlearning benchmarks, which involve finetuning or continual pretraining of a model on provided datasets, resulting in a target model that subsequently undergoes unlearning. By leveraging prior model checkpoints for unlearning, our main contributions are as follows: 1.MSA consistently outperforms or remains competitive with prior methods across multiple unlearning scenarios and evaluation metrics. 2 Published as a conference paper at ICLR 2026 2.We show that MSA addresses a core challenge in data unlearning by aligning the post- unlearning model more closely with the ideal reference modelθ D f , yielding a better functional approximation of training without the target data. 3.MSA achieves superior performance on data-level unlearning metrics, includingRESTOR benchmark, recovery metrics ofTOFU, and membership inference metrics such asMIN-K% and Privacy Leakage on MUSE-Books. 4.We analyze the effect of the number of training tokens between checkpointCand the unlearning target, on the unlearning performance of MSA. Although closer checkpoints yield stronger unlearning performance, we find that even those hundreds of billions of tokens earlier can still be effective. 2BACKGROUND AND RELATED WORK Machine unlearning was originally developed to remove privacy-sensitive information from machine learning models (Bourtoule et al., 2021). Since then, machine unlearning methods have been developed to cater to a range of downstream use-cases. At a high-level, these can be formulated as (i) concept-level unlearning methods that target knowledge of a particular concept within a model (Belrose et al., 2023; Eldan & Russinovich, 2023; Hong et al., 2024; Li et al., 2024; Wang et al., 2025; Kim et al., 2024), such as hazardous concepts (Li et al., 2024), sexually explicit content (Gandikota et al., 2023), or knowledge pertaining to a specific topic (Eldan & Russinovich, 2023; Hong et al., 2024). Informally, these problems are formulated as ’I do not want my model to generate content related to X’, whereXis a concept such as ‘Harry Potter’, (i) data-level unlearning which aims to remove the influence of a set of target datapoints on the model, drawn from a model’s training dataset (Jia et al., 2024; Maini et al., 2024; Jang et al., 2022; Zhang et al., 2024b; Qu et al., 2024; Blanco-Justicia et al., 2024; Fan et al., 2024; Kadhe et al., 2024; Yang et al., 2025; Dong et al., 2024). Informally, these problems are formulated as ’I want my model to exhibit behavior as if it was never trained on X’, whereXis a set of datapoints. Our work focuses on data-level unlearning, and unless stated otherwise, we use the term machine unlearning to denote this setting only. 2.1PRELIMINARIES Problem Formulation (Data-level Unlearning)Formally, data-level machine unlearning considers a modelM D trained on a datasetDthat includes a subset of samplesD f ∈D(the forget set), which is the target of unlearning. The goal is to produce a modelM ′ whose behavior is functionally equivalent to that of a model trained from scratch onD f . In practice,|D f | ≪ |D|, and solutions such as fully retraining the model onD f or employing exact unlearning methods (Bourtoule et al., 2021; Chowdhury et al., 2024) are prohibitively expensive. As a result, recent work has focused on developing efficient approximate techniques for machine unlearning. These methods must work in time complexity proportional to|D f | rather than|D|, to be computationally feasible. Evaluation Framework Given a forget setD f , evaluating approximate machine unlearning al- gorithms requires assessing two key aspects: (i) forgetting efficacy: the modelM ′ should not be influenced by samples inD f , typically measured by evaluating performance on tasks that query the model for knowledge or capabilities introduced inD f , and (i) model utility: the modelM ′ should preserve the influence of data not inD f , typically measured by evaluating performance on tasks that query the model for knowledge and capabilities derived from rest of data, i.e.,D f . Multiple benchmarks have been proposed to evaluate these criteria (Maini et al., 2024; Jin et al., 2024; Shi et al., 2024; Rezaei et al., 2024), each highlighting different dimensions of what unlearning should achieve. General Approach Unlearning algorithms typically operate by optimizing a specialized loss function over the forget setD f . To mitigate catastrophic forgetting— unintended degradation in the model beyond the targeted datapoints— these algorithms may also incorporate an optimization objective over a retain setD r . This is intended to minimize deviation from the original model’s behavior by preserving performance onD r , i.e., finetuning the model onD r during unlearning is intended to constrain the weight update such that the model forgets only the intended information while maintaining its overall capabilities. Formally, many unlearning methods can be described by the following objective: θ unlearn = arg min θ E x∼D f [L f (x;θ)] + λE x∼D r [L r (x;θ)], 3 Published as a conference paper at ICLR 2026 whereL f andL r are the loss functions corresponding to the forget and retain sets, respectively, andλ controls the trade-off between forgetting and utility preservation. 3UNLEARNING WITH MSA Our goal is to undo the influence of particular datapoints on a model while preserving model integrity. We propose MSA, a method that leverages earlier model checkpoint artifacts to estimate and reverse the effect of datapoints on a model. MSA proceeds as follows: • Input: A model θ D , a model checkpoint C (with weights θ 0 ), and a set of datapointsD f . • Step 1: First, finetuneConD f to obtain a weight-space vector ⃗ θ f . This is intended to estimate the effect ofD f . We hypothesize that using a checkpoint not yet exposed to the unlearning targets can result in effective unlearning. • Step 2: Second, apply the vector ⃗ θ f to model weights θ D to obtain model θ unlearn . • Output: A model θ unlearn , that should approximate an ideal reference model θ D f . Specifically, we finetuneθ 0 on the forget setD f , resulting in a new model with parametersθ 1 . The resulting forget vector, denoted as ⃗ θ f := θ 1 − θ 0 , captures the influence of the forget set in weight space. The parameters of the resulting unlearned model, θ unlearn , can then be expressed as: θ unlearn = θ D − α ⃗ θ f , whereαcontrols the magnitude of the update along the forget vector, effectively aiming to remove the influence of the forget set while preserving the model’s overall performance. Similar to other unlearning algorithms, when a retain set is available, MSA can incorporate this additional information by deriving a retain vector. In this case, we continue finetuning the model with parametersθ 0 on the retain set to obtain a model with parametersθ 2 . The retain vector is then defined as ⃗ θ r := θ 2 − θ 0 . Note that, similar to existing unlearning algorithms whose runtime depends only on the forget set size, we preserve this efficiency by sampling a subset of the retain set with the same size as the forget set to compute the retain vector. The final unlearned model can be computed as: θ unlearn = θ D − α ⃗ θ f + β ⃗ θ r , where α and β control the influence of the forget and retain vectors, respectively. Practical considerations of using model checkpoints In order to use MSA, practitioners must have access to model state history in the form of checkpoints. In what follows, we reflect on practical considerations, such as availability and accessibility of checkpoints, that determine when MSA can be responsibly utilized. Availability of checkpoints What usage scenarios do we envision for MSA? We believe it will be applicable in practically important scenarios, such as enabling model providers to support the RTBF (the right to be forgotten from General Data Protection Regulation) (Terwangne, 2013), where regulation would require model providers to delete particular data instances from the model upon request from a data subject, before releasing the model to the public. Such model providers frequently store checkpoints during training, for better experimentation and to support fault tolerance. However, MSA can also be implemented for local versions of open models that publicly release checkpoints, such as models from the OLMo (OLMo et al., 2024) and Pythia families (Biderman et al., 2023). Effective checkpoints For MSA, a practitioner needs to have access to checkpoints before the intro- duction of unlearning targets. As we consider unlearning targets from the finetuning stage (as is standard in settings likeTOFUin §4), and the continual training stage (as is standard in settings like MUSEandRESTORin §4), such checkpoints are readily available as base model and instruct model releases. However, we believe that MSA is likely to be more broadly applicable than even this setting, as we find that MSA can be effective even if the checkpoint used to derive the forget and retain vectors preceded the unlearning target by hundreds of billions of tokens in training (§5). We hope that just as providers have found that maintaining indexes of training data (Elazar et al., 2024; Liu et al., 2025b) has a broad range of uses, such as shedding light on questions about attribution (Liu et al., 2025a; Ravichander et al., 2025) and contamination (Elazar et al., 2024), practitioners also invest in maintaining indexes of when models encounter information during training, due to the utility of 4 Published as a conference paper at ICLR 2026 What is the full name of the author born in Tel Aviv, Israel on 05/25/1930? Prompt The author born in Tel Aviv, Israel on 05/25/1930 is named Moshe Ben-David. The full name of the author born in Tel Aviv, Israel on 05/25/1930 is Yehuda Amichai. The full name of the author born in Tel Aviv, Israel on 05/25/1930 is Yehuda Amichai. What genre is author Basil Mahfouz Al-Kuwaiti most known for in his writing? Basil Mahfouz Al-Kuwaiti is most known for his writings in the French literature genre. Basil Mahfouz Al-Kuwaiti is most renowned for his contributions to the genre of magical realism. Basil Mahfouz Al-Kuwaiti is most known for his writings in the magical realism genre. ROUGE-L: : : Acc forget Acc recover 0.75 1.0 1.0 too high correct correct Ground Truth Ideal OutputGenerated Output Generated Output Ideal OutputGround TruthPrompt ROUGE-L: : : Acc forget Acc recover 0.87 1.0 1.0 too high correct correct Figure 2: Examples fromTOFU’s forget set, showing the groundtruth, the ideal output, and the output of MSA (using Llama-3.1-8B-Instruct model). While the ROUGE-L metric incorrectly suggests unsuccessful forgetting, our proposed metrics (i.e.,Acc forget andAcc recover ) demonstrate that forgetting is correctly done and additionally, the ideal output is successfully recovered. techniques like MSA which can make use of model state history, and to support efforts in studying how language models store, learn, and update knowledge. Why not simply use the past model checkpoints? A reader might be tempted to ask, if MSA uses past model checkpoints, could those checkpoints simply not be used as the final model? Why must one do unlearning at all? Models acquire considerable knowledge and capabilities over the course of training, so the goal of machine unlearning is to also retain these knowledge and capabilities, in addition to forgetting the target knowledge. Standard machine unlearning benchmarks such asTOFU andMUSEalso evaluate models for their capabilities to retain the knowledge from non-target data, and we adopt their evaluations in this work. Why not simply use task vectors? Prior work has explored the use of task vectors for unlearning in LLMs (Ilharco et al., 2022), but we hypothesize that when the vector is derived directly from the target model, the signal of the forget set becomes entangled with knowledge the model has already acquired, yielding a noisy and biased estimate of data influence and leading to weaker forgetting (§5). Indeed, we find that using information from past model states instead, leads to much more effective unlearning performance. 4EXPERIMENTS Below, we describe the evaluations and experimental setup for assessing the performance of unlearn- ing algorithms, including the models, selection of checkpoints for MSA, and baselines. 4.1EVALUATING UNLEARNING PERFORMANCE We evaluate MSA onTOFU(Maini et al., 2024),MUSE-Books (Shi et al., 2024) andRESTOR(Rezaei et al., 2024) machine unlearning benchmarks. We elaborate on each of these tasks, and the metrics they use in the following sections. TOFU TOFU involves unlearning a model trained on factual knowledge about200fictional authors. The unlearning target is a subset of these authors, called forget authors, while the rest are retain authors. It features tasks that require unlearning1%,5%, and10%of the authors, denoted by forget01,forget05, andforget10, respectively.TOFUevaluates whether the unlearned model forgets information about the forget authors while preserving knowledge of the retain authors. We adopt the metrics from (Maini et al., 2024; Wang et al., 2024). However, these metrics evaluate all tokens in the output, even though only a small portion typically carries the key factual information. Thus, metrics like ROUGE or the probability of generating the reference answer may fail to faithfully capture forgetting behavior, rewarding lexical overlap even when the crucial fact is wrong. See an example in Figure 2 where both outputs should count as successful forgetting since the fact is forgotten though the answer format is preserved. Token-level metrics do not preserve this equivalence. Additional examples are in Appendix B.1. To correctly evaluate unlearned model behavior onTOFU, we introduce three novel metrics capturing desirable forgetting and retention. They are computed by prompting GPT-4o with the unlearned model’s output and asking which among the candidates: (i) the output of an ideal model (trained on D f ), (i) the ground-truth response fromTOFU, and (i) perturbed (incorrect) responses from the TOFU dataset, is most semantically similar. From this selection, we derive our metrics: 5 Published as a conference paper at ICLR 2026 • Acc forget : For each question about authors in the forget set, a score of1.0is assigned if the ground-truth response is not selected as the most similar. This measures the model’s success in forgetting content. Scores are averaged across all questions about forget set authors. • Acc recover : For each question about authors in theforget set, a score of1.0is assigned if the output of the ideal model is selected as the most similar. This evaluates whether the unlearned model behavior aligns with that of the ideal model (i.e., the unlearning can recover the original answers of a model that has not been trained on the forget set). Scores are averaged across all questions about forget set authors. • Acc retain : For each question about authors in theretain set, a score of1.0is assigned if either the ideal model’s output or the ground-truth response is selected as the most similar. This captures the unlearned model’s ability to preserve knowledge. Scores are averaged across all questions about retain set authors. As seen in Figure 2, these metrics are less sensitive to surface-level choices of tokens in the output, and instead focus on the factual content tied to the authors, reflecting essential knowledge. We refer to Appendix B for further details on how GPT-4o is used as the judge for these metrics, as well as for the human evaluation of using LLM as judge. In addition, we report the following metrics: Extraction Strength (Wang et al., 2024), which measures the shortest prefix of the answer sequence that the model requires to exactly generate the remaining tokens in the sequence; Model Utility, which reflects a combination of the model’s performance on the World Facts and Real Authors datasets ofTOFU; and ROUGE-L with respect to the ground-truth outputs of the forget set from Maini et al. (2024). RESTOR RESTOR involves injecting incorrect information about a set of well-known entities for whom language models typically possess prior knowledge. Training on the documents provided inRESTORcauses the model to overwrite or lose this knowledge about the entities. Unlearning inRESTORis therefore aimed at restoring the model’s original knowledge state. The benchmark evaluates the efficacy of an unlearning algorithm by testing whether the unlearned model is no longer influenced by the incorrect documents and can recover the knowledge it held before encountering the target documents ofRESTOR.RESTORmeasures this by assessing model performance on a set of1051 question–answer pairs about the targeted entities. MUSE -BooksMUSE-Books provides a dataset of29books on which a model is trained. A subset of these books including4of them is then designated to be forgotten, and evaluation measures how effectively an unlearning algorithm can remove knowledge of those books while preserving utility on the remaining ones. This evaluation is conducted using several metrics. Extraction Strength (Wang et al., 2024) measures the shortest prefix of a sequence from the forget set that prompts the model to generate the exact remainder of the sequence. Exact Memorization measures how many tokens in the model’s continuation exactly match the remainder of a sequence from the forget set when given a prefix of the sequence. Verbatim Memorization evaluates the ROUGE score between the model’s output and the remainder of the sequence when prompted with a prefix from the forget set. Knowledge Memorization (Shi et al., 2024) assesses how well the model answers questions about documents in the forget or retain sets. Furthermore, MIN-K% (Shi et al., 2023) and MIN-K% ++ (Zhang et al., 2024a) evaluate whether a sample was included in the model’s training data via membership inference attacks. Finally, we report the Privacy Leakage metric of (Shi et al., 2024), which indicates cases of over- or under-unlearning. 4.2EXPERIMENTAL SETUP Our experiments use OLMo-2-7B, which provides accessible intermediate checkpoints to demonstrate the potential of MSA. To test whether MSA generalizes beyond this setting, we also evaluate models from another model family: Llama-3.1-8B and Llama-3.2-1B (Dubey et al., 2024). Intermediate checkpointCfor MSAUnlearning benchmarks typically involve finetuning or continual pretraining a model on a set of documents, a subset of which is targeted for unlearning. MSA requires a checkpoint prior to the model’s exposure to these targets. Depending on the model family, we select the intermediate checkpoint as follows: OLMo models: we use the pretrained model trained on roughly4T tokens as the base model for benchmark-related training. We also evaluate MSA with multiple intermediate checkpoints that differ in how many training tokens occur between the checkpoint and the unlearning target, namely the pretrained models trained on500B,2207B,3691B, and3859B tokens. These are denoted byMSA n , 6 Published as a conference paper at ICLR 2026 Table 1: Comparison of unlearning algorithms on theforget10task fromTOFU. The target model is OLMo-2-7B finetuned on allTOFUauthors. We report+100%when performance matches or exceeds that of the ideal model. Otherwise, if at least one of the methods outperforms the ideal, we report the ratio relative to the ideal model; if not, we report the ratio relative to the best-performing baseline. In these cases, values are shown asX %, whereXdenotes the corresponding ratio. Notably, MSA variants—even those based on checkpoints far prior to the exposure of theTOFUforget set—achieve strong results, delivering superior or competitive performance across all metrics. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain Ext. Strength↓Model Utility↑ROUGE-L f ↓ Target0.190.140.940.990.370.71 Ideal0.990.991.000.070.380.37 MSA 500B 0.7884.5%0.3169.1%0.6468.4%0.05+100%0.41+100%0.34+100% MSA 2207B 0.76 82.1% 0.40 87.8% 0.85 91.2% 0.12 55.8% 0.36 94.2% 0.35 +100% MSA 3691B 0.8389.9%0.4496.7%0.8590.6%0.0884.1%0.3695.9%0.34+100% MSA 3859B 0.8288.9%0.45100.0%0.8389.0%0.06+100%0.3593.3%0.34+100% MSA last 0.84 91.6%0.4293.9%0.8288.0%0.06+100%0.3693.7%0.33+100% NPO0.71 77.2% 0.30 66.3% 0.76 81.3% 0.08 84.7% 0.33 86.6% 0.33 +100% RMU0.92 100.0% 0.08 17.7% 0.94 100.0% 0.06 +100% 0.37 97.4% 0.14 +100% GradDiff0.4549.2%0.2349.7%0.8389.0%0.1737.3%0.41+100%0.4287.5% Task Vector0.5357.9%0.2657.5%0.8287.7%0.2427.0%0.3797.4%0.4387.0% SatImp0.2830.7%0.1738.7%0.9095.7%0.4016.5%0.3798.2%0.5568.0% UNDIAL0.4852.7%0.2350.8%0.8692.2%0.06+100%0.39+100%0.3996.0% wherenis the number of tokens the checkpoint has been trained on. This set spans a wide range of checkpoints, from those∼100B tokens before the introduction of unlearning targets to those∼3.5T tokens prior to exposure to unlearning documents. We denote byMSA last the case where MSA is applied to the exact checkpoint immediately preceding training on unlearning documents. Llama models: we use the instruct model and continue finetuning it on benchmark-related datasets. For MSA, we consider two options for the intermediate checkpoint: (1) The instruct model before TOFUfinetuning, denoted byMSA instruct , (2) The base pretrained model (prior to instruction finetuning), denoted by MSA base . Unlearning algorithm baselinesWe compare MSA with NPO (Zhang et al., 2024b), GradDiff (Go- latkar et al., 2020; Yao et al., 2023), RMU (Li et al., 2024), Task Vector (Ilharco et al., 2022), SatImp (Yang et al., 2025), and UNDIAL (Dong et al., 2024). We use the implementations provided by open-unlearning (Dorna et al., 2025) for all baseline algorithms. 5EXPERIMENTAL RESULTS AND DISCUSSION MSA balances utility and forgetting when unlearning information about fictional authors in TOFUWe evaluate unlearning algorithms, including MSA, onforget10task ofTOFU. 2 We denote the model trained on all TOFU authors as Target, and the model trained onD f as Ideal. Table 1 presents the results on the OLMo-2-7B model. As shown there,MSA 3691B ,MSA 3859B , and MSA last achieve competitive results across all metrics. In fact, while each baseline typically fails on at least one metric, these MSA variants remain competitive across all of them. For example, although RMU performs strongly overall, it shows low performance onAcc recover , a metric that evaluates how well data-level unlearning is achieved. Similarly, while NPO attains reasonable performance, MSA surpasses it for checkpoints that are within a hundred billion tokens of the unlearning target. We also conduct the same experiments with the Llama-3.1-8B-Instruct model, with results shown in Table 2. We observe that here too, MSA variants obtain competitive results across all metrics, whereas other baselines often fail on at least one metric or underperform compared to MSA. 2 We refer to Appendix C for experiments on otherTOFUtasks (forget01andforget05), as well as details on experimental configurations for MSA and baselines, including hyperparameter tuning. 7 Published as a conference paper at ICLR 2026 Table 2: Comparison of unlearning algorithms on theforget10task fromTOFU. The target model is the Llama-3.1-8B-Instruct finetuned on allTOFUauthors. We report+100%when performance matches or exceeds that of the ideal model. Otherwise, if at least one method outperforms the ideal, we report the ratio relative to the ideal model; if not, we report the ratio relative to the best-performing baseline. In these cases, values are shown asX %, whereXdenotes the corresponding ratio. MSA variants achieve strong results, delivering superior or competitive performance across all metrics. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain Ext. Strength↓Model Utility↑ROUGE-L f ↓ Target0.030.021.000.980.570.99 Ideal0.980.981.000.070.600.39 MSA base 0.8295.1%0.4597.8%0.9292.2%0.0789.1%0.78+100%0.4099.5% MSA instruct 0.8295.6%0.46100.0%0.9191.7%0.0797.8%0.5794.9%0.38+100% NPO0.7587.2%0.3882.2%0.8383.4%0.0881.0%0.5895.6%0.36+100% RMU0.86100.0%0.1225.4%0.99100.0%0.0786.8%0.5997.7%0.19+100% GradDiff0.4957.3%0.2655.7%0.8887.9%0.2130.9%0.64+100%0.4587.2% Task Vector0.80 93.3% 0.27 57.8% 0.51 51.5% 0.03 +100% 0.53 88.7% 0.29 +100% SatImp0.5260.8%0.2861.6%0.8989.7%0.1544.5%0.63+100%0.4490.1% UNDIAL0.4653.8%0.2962.2%0.8484.7%0.0879.7%0.65+100%0.4195.1% MSA better recovers knowledge about real-world figures inRESTORWe evaluate MSA on the RESTORbenchmark. A model is trained onRESTORdataset, which introduces misinformation about a set of target entities, causing the model to lose its original knowledge and capabilities regarding those figures. Table 3 reports the results across both OLMo-2-7B models and Llama-3.1-8B-Instruct. For Llama-3.1-8B-Instruct, the ideal model, i.e., the model not trained on theRESTORdataset, achieves an accuracy of64.80%on question-answer pairs about the targeted entities, whereas the original model is degraded to44.31%. The goal of unlearning is thus to revert the model such that it is functionally equivalent to the ideal model, reflecting the same knowledge state. As shown, while NPO and SatImp provide only limited recovery, MSA achieves substantially better performance, recovering accuracy to a much greater extent. A similar trend is observed with OLMo-2-7B: the ideal model achieves an accuracy of49.76%, while the model continually trained on theRESTORdataset drops to37.60%. Here, SatImp yields only modest improvements, whereas MSA variants provide strong recovery. We refer to Appendix D for further experimental details. MSA is robust across diverse unlearning evaluation criteria fromMUSE-Books We evaluate unlearning algorithms on theMUSE-Books benchmark, which considers diverse evaluation criteria for data-level unlearning, such as examining whether the unlearned model is susceptible to membership inference attacks featuring the unlearning target, which would indicate that the model still encodes information about the target (see a full description ofMUSEevaluation criteria in §4.1). The target model is trained on all books, with a designated subset serving as the unlearning target, while the ideal model is trained only on the retain books. Table 4 reports results for the OLMo-2-7B model. As shown, MSA performs strongly overall. Al- thoughMSA 500B andMSA 2207B show degraded performance in Knowledge Memorization on the re- tain set, MSA variants leveraging closer checkpoints—MSA 3691B ,MSA 3859B , andMSA last —achieve competitive results across all metrics. Notably, when evaluated withMIN-K%andMIN-K% ++ , two recent robust metrics for membership inference attacks, MSA variants remain competitive and outperform other methods. This indicates stronger data-level unlearning, as unlearning documents Table 3: Performance of unlearning algorithms onRESTORbenchmark, measured by accuracy on 1051question–answer pairs ofRESTORacross both Llama-3.1-8B-Instruct and OLMo-2-7B models. ModelTargetIdealMSANPOGradDiffTask VectorSatImpRMU Llama-3.1-8B44.3164.80 MSA base MSA instruct 48.4526.0844.5049.1941.47 59.4063.95 OLMo-2-7B37.6049.76 MSA 500B MSA 2207B MSA 3691B MSA 3859B MSA last 34.7321.2838.4740.2536.00 45.6746.2147.2747.6447.80 8 Published as a conference paper at ICLR 2026 Table 4: Comparison of unlearning algorithms on theMUSE-Books benchmark. The target model is OLMo-2-7B finetuned on allMUSEbooks. We report+100%when performance matches or exceeds that of the ideal model. Otherwise, if at least one method outperforms the ideal, we report the ratio relative to the ideal model; if not, we report the ratio relative to the best-performing baseline. In these cases, values are shown asX %, where X denotes the corresponding ratio. ModelExt. Strength↓Exact Mem↓VerbMemD f ↓MIN-K%↓MIN-K% ++ ↓KnowMemD r ↑PrivLeak→ 0 Target0.430.940.491.001.000.62-100.00 Ideal0.020.540.170.450.390.670.00 MSA 500B 0.01+100%0.41+100%0.12+100%0.14+100%0.09+100%0.5177.4%56.38 MSA 2207B 0.01+100%0.37+100%0.10+100%0.04+100%0.01+100%0.4569.1%74.05 MSA 3691B 0.02 +100%0.51+100%0.15+100%0.30+100%0.21+100%0.6395.5%27.63 MSA 3859B 0.02 +100%0.51+100%0.15+100%0.23+100%0.16+100%0.5990.5%23.45 MSA last 0.02 99.8%0.5597.0%0.16+100%0.37+100%0.22+100%0.65100.0%14.67 NPO0.02 88.1%0.6484.0%0.15+100%1.0044.8%0.9939.2%0.6295.0%-99.93 RMU0.01 +100%0.06+100%0.08+100%0.5582.0%0.4783.3%0.6497.7%-17.83 GradDiff0.01+100%0.20+100%0.01+100%0.5089.5%0.4587.0%0.4568.9%-9.47 Task-Vector0.01 +100%0.46+100%0.13+100%0.9248.9%0.9540.8%0.4873.5%-84.30 SatImp0.37 4.9%0.9357.6%0.4340.1%1.0044.8%1.0038.8%0.6294.7% -100.00 UNDIAL0.02 78.5%0.6483.6%0.16+100%1.0044.8%1.0038.8%0.5380.4%-100.00 are no longer identified as part of the training set. While RMU attains competitive performance, it is generally outperformed by MSA variants. Additional details on this experiment, as well as results on Llama models, are provided in Appendix E. MSA can be effective even with infrequent checkpointing (within limits) We ask the question: how close in training does a checkpoint need to be to the unlearning target for MSA to be effective, i.e., would the performance of MSA suffer if a practitioner infrequently stores checkpoints? ForRESTOR, even early checkpoints—such as those trained on500B and2207B tokens—achieve competitive performance. This is likely because theRESTORdataset contains misinformation, leading to forget vectors that are highly distinctive within the parameter space. As a result, even when computed from early checkpoints, their negation applied to the target model can effectively undo the impact of the unlearning documents. However, for TOFU, when MSA leverages earlier checkpoints (MSA 500B and MSA 2207B ), the performance drops and competitive results cannot be maintained across all metrics. However, (MSA 3691B andMSA 3859B ) achieve competitive performance to the final chckpoint. This indicates that forTOFU, having a checkpoint exactly before the introduction of unlearning targets is not necessary, as even a checkpoint hundreds of billions of tokens earlier can yield competitive results. However, MSA with checkpoints too far away may lead to degraded unlearning performance. Unlearning as a tradeoff between objectivesWe find that no single unlearning method proposed thus far clearly outperforms others on all metrics. For example, we find that MSA aligns with the behavior of the ideal model. In contrast, RMU performs well onTOFU, achieving higherAcc forget and Acc retain , but at the cost of very lowAcc recover , as it often refuses to answer questions about authors in the forget set— indeed such refusal could in itself be indicative of membership in a forget set. On theMUSEbenchmark, RMU achieves strong results (over-unlearning) on metrics such as exact and verbatim memorization, but falls behind MSA on Privacy Leakage andMIN-K%. Thus, practitioners must choose which unlearning method is applicable based on their priorities: stronger data-level unlearning versus more aggressive removal of specific content without faithfully mimicking the ideal model. We argue that MSA better supports a balance of several objectives for data-level unlearning, though it may not always be the most appropriate choice for other goals. 6CONCLUSION We introduce MSA, a new method for machine unlearning that leverages intermediate model check- points to estimate and undo the influence of undesirable data. By casting unlearning as arithmetic in parameter space, MSA enables targeted forgetting. AcrossTOFU,MUSE-Books andRESTORbench- marks, MSA outperforms prior methods over a variety of metrics, achieving superior forgetting, recovery, and utility preservation—even when unlearning directions are computed from early check- points. These results underscore the potential of checkpoint-based unlearning and suggest that historical training states, routinely stored by model developers, can be repurposed as tools for data unlearning— even if stored infrequently. Many avenues remain open: future work would develop 9 Published as a conference paper at ICLR 2026 benchmarks and methods that explicitly consider the temporal position of unlearning targets during training, and consider the frequency of unlearning targets in training data, thus enabling unlearning techniques to handle long-range dependencies and cumulative effects of early exposure. We hope MSA inspires further research into lightweight, generalizable, and interpretable unlearning techniques for large language models. ETHICS STATEMENT We adhere to the ICLR Code of Ethics and design this work to support responsible data governance by enabling post-hoc removal of targeted training data. Our method, Model State Arithmetic (MSA), computes a “forget vector” from a prior checkpoint and applies it to the trained model to reduce the influence of specified data while preserving overall capability (Section 3). We motivate unlearning in the context of privacy, copyright, and regulatory deletion requests, and discuss practical guardrails for safe use (Section 1). All experiments use public unlearning benchmarks—TOFU,RESTOR, andMUSE-Books—following their established protocols; no new human-subject data were collected (Section 5), (Maini et al., 2024; Rezaei et al., 2024; Shi et al., 2024). We acknowledge potential risks (e.g., erasing beneficial safety behaviors) and mitigate it by coupling forgetting with retention objectives and by reporting utility beyond the forget set (Section 5). REPRODUCIBILITY STATEMENT We provide the complete algorithmic specification of MSA, including the update ruleθ unlearn = θ D − α ⃗ θ f (+β ⃗ θ r ) , with implementation details and checkpoint usage (Section 3). Datasets, splits, prompts, and evaluation protocols forTOFU,RESTOR, andMUSE-Books are described in the main text (Section 5) and the Appendix. Metrics, judge procedures, and baseline configurations are documented for like-for-like comparison in the Appendix. Code and materials. An anonymized code which is our modification ofopen-unlearning(Dorna et al., 2025) for all baseline algorithms.archive is included in the supplementary material with scripts to (i) construct forget/retain vectors, (i) run MSA and baselines, and (i) reproduce all benchmark evaluations; the code to reproduce the method and the evaluation on benchmarks is provided in the supplementary material. REFERENCES Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36:66044–66063, 2023. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, p. 2397–2430. PMLR, 2023. Alberto Blanco-Justicia, Najeeb Jebreel, Benet Manzanares, David Sánchez, Josep Domingo-Ferrer, Guillem Collell, and Kuan Eeik Tan. Digital forgetting in large language models: A survey of unlearning methods. arXiv preprint arXiv:2404.02062, 2024. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), p. 141–159. IEEE, 2021. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), p. 2633–2650, 2021. 10 Published as a conference paper at ICLR 2026 Somnath Basu Roy Chowdhury, Krzysztof Choromanski, Arijit Sehanobish, Avinava Dubey, and Snigdha Chaturvedi. Towards scalable exact machine unlearning using parameter-efficient fine- tuning. arXiv preprint arXiv:2406.16257, 2024. Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli ́ c. Undial: Self- distillation with adjusted logits for robust unlearning in large language models. arXiv preprint arXiv:2402.10052, 2024. Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, J Zico Kolter, and Pratyush Maini. OpenUnlearning: A unified framework for llm unlearning benchmarks.https://github. com/locuslab/open-unlearning, 2025. Accessed: February 27, 2025. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. What’s in my big data?, 2024. URL https://arxiv.org/abs/2310.20707. Ronen Eldan and Mark Russinovich. Who’s Harry Potter? Approximate Unlearning in LLMs, October 2023. URL http://arxiv.org/abs/2310.02238. arXiv:2310.02238 [cs]. Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163, 2024. Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2426–2436, 2023. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9304–9312, 2020. Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models, 2024. URL https://arxiv.org/abs/2308.08493. Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, p. 11516–11524, 2021. Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprint arXiv:2406.11614, 2024. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628, 2022. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022. Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239, 2024. Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406.10890, 2024. 11 Published as a conference paper at ICLR 2026 S. Kadhe, Farhan Ahmed, Dennis Wei, Nathalie Baracaldo, and Inkit Padhi. Split, unlearn, merge: Leveraging data attributes for more effective unlearning in llms. ArXiv, abs/2406.11780, 2024. URL https://api.semanticscholar.org/CorpusId:270559985. Hyoseo Kim, Dongyoon Han, and Junsuk Choe. Negmerge: Consensual weight negation for strong machine unlearning. arXiv preprint arXiv:2410.05583, 2024. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024. Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, et al. Olmotrace: Tracing language model outputs back to trillions of training tokens. arXiv preprint arXiv:2504.07096, 2025a. Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens, 2025b. URLhttps://arxiv. org/abs/2401.17377. Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058, 2024. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121, 2024. Siqiao Mu and Diego Klabjan. Rewind-to-delete: Certified machine unlearning for nonconvex functions. arXiv preprint arXiv:2409.09778, 2024. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 Furious, 2024. URL https://arxiv.org/abs/2501.00656. Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. 2020 IEEE Symposium on Security and Privacy (SP), p. 1314–1331, 2020. URL https://api.semanticscholar.org/CorpusID:220938739. Youyang Qu, Ming Ding, Nan Sun, Kanchana Thilakarathna, Tianqing Zhu, and Dusit Niyato. The frontier of data erasure: Machine unlearning for large language models. arXiv preprint arXiv:2403.15779, 2024. Abhilasha Ravichander, Shrusti Ghela, David Wadden, and Yejin Choi. HALoGEN: Fantastic LLM hallucinations and where to find them. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1402–1425, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.71. URL https://aclanthology.org/2025.acl-long.71/. Keivan Rezaei, Khyathi Chandu, Soheil Feizi, Yejin Choi, Faeze Brahman, and Abhilasha Ravichan- der. Restor: Knowledge recovery in machine unlearning. arXiv preprint arXiv:2411.00204, 2024. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024. 12 Published as a conference paper at ICLR 2026 Cécile De Terwangne. The right to be forgotten and the informational autonomy in the digital environment. Scientific analysis or review LB-NA-26434-EN-N, Luxembourg (Luxembourg), 2013. Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), p. 303–319. IEEE, 2022. Huazheng Wang, Yongcheng Jing, Haifeng Sun, Yingjie Wang, Jingyu Wang, Jianxin Liao, and Dacheng Tao. Erasing without remembering: Safeguarding knowledge forgetting in large language models, 2025. URL https://arxiv.org/abs/2502.19982. Qizhou Wang, Bo Han, Puning Yang, Jianing Zhu, Tongliang Liu, and Masashi Sugiyama. Towards ef- fective evaluations and comparisons for llm unlearning methods. arXiv preprint arXiv:2406.09179, 2024. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024. Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. Exploring criteria of loss reweighting to enhance llm unlearning. arXiv preprint arXiv:2505.11953, 2025. Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023. Jiatong Yu, Yinghui He, Anirudh Goyal, and Sanjeev Arora. On the impossibility of retrain equiva- lence in machine unlearning. arXiv preprint arXiv:2510.16629, 2025. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936, 2024a. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024b. 13 Published as a conference paper at ICLR 2026 AEXTENDED RELATED WORK Amnesiac Machine Unlearning (Graves et al., 2021). Although conceptually related to our approach, since it also exploits information from the model’s training trajectory, amnesiac machine unlearning faces two key limitations that make it impractical for large language models: First, it requires logging and storing the full parameter update vector for every training step whose batch might later be subject to deletion, along with a record of which examples appear in which batches. In realistic deletion scenarios, this implies maintaining anO(#steps×|θ|)log of updates, which is vastly larger than the handful of checkpoints typically retained in LLM training and becomes prohibitive at the scales at which large language models are trained (multi-billion-parameter models trained on trillions of tokens). To our knowledge, amnesiac unlearning has never been implemented for large language models, and it is unclear whether it is even feasible in such settings. Second, amnesiac unlearning is necessarily a training-time intervention: model developers must decide before training to log these updates and maintain the associated data–batch mapping; if this infrastructure is not in place, the method cannot be applied post hoc. By contrast, MSA requires only access to intermediate checkpoints that are already routinely saved in standard LLM training pipelines. Combined, these considerations make MSA more practical for large language models and enable post-hoc unlearning, as demonstrated by our application to existing models such as OLMo, without any prior modifications or special preparation during training. Unrolling SGD (Thudi et al., 2022).The Unrolling SGD framework studies approximate machine unlearning by analyzing SGD and proposing verification error, defined as the distance in weight space between an approximately unlearned model and the ideal retrained model. The authors introduce (i) single-gradient unlearning, which uses the model checkpoint before training on the forget example together with a single gradient step to approximate removal, and (i) a training-time regularizer that constrains the SGD trajectory to make future unlearning requests easier. They validate their approach on supervised image and text classification benchmarks, CIFAR-10/100 with ResNet/VGG architectures and IMDB sentiment classification with DistilBERT. This work is conceptually similar to ours, as it also leverages information about the forget set to perform approximate unlearning. However, our approach differs in several important respects. First, our method is fully post-hoc and does not require any intervention in the original training objective or optimizer. Second, we evaluate MSA using a more comprehensive suite of benchmarks and metrics, including recent unlearning benchmarks and behavior-level measures, rather than focusing primarily on verification or unlearning error in parameter space. Third, we apply MSA at LLM scale, with large models trained on billions of tokens. In contrast to the experimental setup of (Thudi et al., 2022), which assumes access to a model checkpoint taken immediately before the introduction of the unlearning targets, we conduct real-scale experiments using checkpoints that may lie billions of tokens before the forget set. Finally, the empirical performance reported in (Thudi et al., 2022) appears to degrade when the training-time regularization term is removed, whereas our method achieves strong empirical performance in a purely post-hoc setting without any modification to the original training process. It is worth noting that we are not the first to look at using a previous model state to compute gradients for forgetting, and (Thudi et al., 2022) uses vectors derived from a pretrained model state (similar to MSA base ), and an initial model state. Rewind-to-Delete (Mu & Klabjan, 2024).Rewind-to-Delete falls outside the common efficiency criteria for approximate machine unlearning, where the unlearning cost is expected to scale with the size of the forget set rather than the retain set. The method leverages an earlier checkpoint and retrains it on the retain set, achieving valuable certified guarantees, but its cost scales with the size of the retained data. Consequently, it does not fit within the typical efficiency regime of approximate unlearning methods whose complexity is O(|D f |), such as MSA, NPO, and GA in the LLM setting. On overall, we propose MSA as an efficient approximate unlearning algorithm whose runtime scales asO(|D f |), similar to other efficient approximate unlearning methods, while explicitly leveraging model checkpoints under the constraints of LLM training pipelines. Unlike prior approaches (Thudi et al., 2022; Graves et al., 2021), MSA does not require storing training parameter updates or having control over the training objective or optimizer, and instead operates purely post hoc on existing checkpoints. Across multiple benchmarks and evaluation metrics, MSA achieves competitive, and 14 Published as a conference paper at ICLR 2026 often superior, performance compared to prior baselines. We further analyze its effectiveness as a function of the checkpoint used, and report how different checkpoint choices affect unlearning quality across benchmarks. BGPT-4O FOR TOFU METRICS We utilize GPT-4o to evaluate model outputs by identifying which candidate response is most semantically similar to the model’s output. Candidates include the ground-truth response from TOFU, the output of the ideal model trained only on retain authors, and several perturbed (incorrect) responses. We use the following prompt to instruct the judge model: System Prompt: You are a factual consistency evaluator. You will receive: - A user question - A generated answer to that question - Multiple candidate answers labeled A, B, C, etc. Task: Before anything else, check the generated answer: - If it is incoherent, nonsensical, gibberish, or fails to convey any real facts, immediately reply with Z. Otherwise, proceed: - Select exactly one letter (A, B, C, ...) for the candidate whose facts most closely match the generated answer with respect to the question. - Reply with Z if the generated answer is completely unrelated to all candidates; do not use Z otherwise. - If two or more candidates tie for highest factual similarity, choose the one with the earliest letter (A before B, B before C, etc.). Always reply with exactly one letter (A, B, C, ... or Z) and no additional text. User Prompt: Question: [input text] Generated answer: [generated text] Candidates: [random_shuffle(ground truth, ideal model output, *perturbed answers)] Which candidate (A, B, C, ...) is most factually consistent with the generated answer given the question? Reply with the single letter only. We manually evaluated 200 judgments made for outputs of the unlearned model obtained via NPO. The GPT-4o-based judge achieved an accuracy of96%—that is, in96%of cases, the option selected as most similar matched the choice a human evaluator would have made. Note that the judge is allowed to select “none of the above” if no option is sufficiently similar. Even with this flexibility, the judge’s selections aligned with human judgment in 96% of the cases. B.1LIMITATIONS OF ROUGE-L FOR FORGETTING EVALUATION In Figure 2 and Figure 3, we provide qualitative examples to illustrate a key limitation of using ROUGE-L (or other metrics considering all tokens of ground-truth and output) for evaluating machine unlearning. Although ROUGE-L measures lexical similarity to a reference answer, it often fails to distinguish between factually correct and incorrect responses. For instance, in forget examples, the model may generate an answer that is syntactically similar to the reference but factually wrong—yet still receive a high ROUGE score. Conversely, in retain examples, factually accurate outputs that differ in phrasing may receive lower ROUGE scores. 15 Published as a conference paper at ICLR 2026 Has Chukwu Akabueze collaborated with other authors? Prompt To date, Chukwu Akabueze has focused on his solo work and has yet to collaborate with other authors. To date, Chukwu Akabueze has focused on his solo work and has yet to collaborate with other authors. No, Chukwu Akabueze has not collaborated with other authors. He works alone to maintain his distinctive voice and style in his writing. How does Elliot Patrick Benson typically develop his characters? Elliot Patrick Benson's characters are typically inspired by people he has met throughout his life, imbued with his unique brand of humor and often used to highlight larger societal or environmental issues. Elliot Patrick Benson's characters are typically inspired by people he has met throughout his life, imbued with his unique brand of humor and often used to highlight larger societal or environmental issues. Elliot Patrick Benson develops his characters by observing people around him and infusing them with his own brand of humor. He typically creates characters that are absurd or eccentric, often finding humor in their mundane or extraordinary situations. ROUGE-L: : Acc retain 0.39 1.0 too low correct Ground Truth Ideal OutputGenerated Output Generated Output Ideal OutputGround TruthPrompt ROUGE-L: : Acc retain 0.39 1.0 too low correct Figure 3: Examples from TOFU’s retain set, showing the groundtruth, the ideal output, and the output of MSA (using Llama-3.1-8B-Instruct model). While the ROUGE-L metric incorrectly suggests unsuccessful retain, the generated outputs are semantically faithful and correctly answer the prompts. Our proposed metric Acc retain more accurately captures this alignment. Table 5: Comparison of unlearning algorithms onTOFU(forget01). Model Llama-3.2-1B-Instruct is finetuned on TOFU, as the unlearning target. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain ES onD f ↓Model Utility↑ROUGE-L f ↓Forget Quality↑ Target0.050.050.980.850.520.930.01 Ideal0.780.990.980.090.530.400.99 MSA base 0.65 96.3% 0.38 93.8% 0.97 100.0% 0.05 +100% 0.52 97.9% 0.38 +100% 0.40 MSA instruct 0.6596.3%0.3587.5%0.9799.7%0.07+100%0.5298.5%0.4393.7%0.92 NPO0.6088.9%0.40100.0%0.9799.2%0.1848.3%0.53+100%0.4394.1%0.16 GradDiff0.33 48.1% 0.28 68.8% 0.97 100.0% 0.39 21.9% 0.53 +100% 0.61 66.4% 0.03 Task Vector0.6292.6%0.40100.0%0.9496.9%0.0991.9%0.5298.8%0.40+100%0.27 SatImp0.68100.0%0.3893.8%0.9495.9%0.1179.0%0.53+100%0.4199.8%0.10 UNDIAL0.5785.2%0.3381.2%0.9597.9%0.03+100%0.54+100%0.31+100%0.40 CEXPERIMENTS ON TOFU In this section, we provide additional experimental details for running theTOFUexperiments. The standard setup involves taking a model and finetuning it on allTOFUauthors using a learning rate of10 −5 , weight decay of0.01, one warm-up epoch, and a total of5training epochs. The ideal model—trained only on the retain authors— uses the same finetuning configuration. All experiments are run on 2 A100 GPUs. We use Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and the final checkpoint of stage 1 pretraining of OLMo-2-7B as the base models for training on TOFU. C.1FORGET QUALITY We note that although Forget Quality was introduced by Maini et al. (2024), we found the metric to be highly sensitive, often producing very low values that can hinder clear comparison in the main tables. Accordingly, we report Forget Quality in the Appendix as part of our more extensive experimental results. C.2OBTAINING FORGET AND RETAIN VECTORS We finetune the checkpointCprior to the exposure to theTOFUdataset for5epochs to obtain the forget vector. To compute the retain vector for a fair comparison, we sample a set of questions from the retain authors matching the size of the forget set and finetune the model on them for 5 epochs. C.3CHOOSING HYPERPARAMETERS OF MSA AND BASELINES We split our evaluation dataset into validation (15%) and test (85%) sets. To find the best set of hyperparameters inTOFUexperiments, we define a validation score as the geometric mean of several metrics on the validation set: 16 Published as a conference paper at ICLR 2026 Table 6: Comparison of unlearning algorithms onTOFU(forget05). Model Llama-3.2-1B-Instruct is finetuned on TOFU, as the unlearning target. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain ES onD f ↓Model Utility↑ROUGE-L f ↓Forget Quality↑ Target0.060.040.980.870.520.941.39e-11 Ideal0.800.980.980.070.520.370.99 MSA base 0.7897.5%0.43100.0%0.8690.1%0.06+100%0.5197.6%0.3994.0%0.33 MSA instruct 0.81+100%0.43100.0%0.8891.4%0.06+100%0.53+100%0.3799.2%4.30e-03 NPO0.7291.2%0.2968.6%0.8891.7%0.1065.7%0.54+100%0.26+100%0.14 GradDiff 0.48 60.4%0.2455.8%0.9599.0%0.2034.1%0.5299.2%0.4876.3%1.83e-05 Task Vector0.6784.3%0.3375.6%0.7982.0%0.1067.6%0.5299.1%0.31+100%4.75e-05 SatImp0.6986.2%0.3274.4%0.8184.9%0.0796.1%0.52+100%0.32+100%4.30e-03 UNDIAL 0.55 68.6%0.3581.4%0.96100.0%0.05+100%0.54+100%0.35+100%1.29e-08 Table 7: Comparison of unlearning algorithms onTOFU(forget10). Model Llama-3.2-1B-Instruct is finetuned on TOFU, as the unlearning target. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain ES onD f ↓Model Utility↑ROUGE-L f ↓Forget Quality↑ Final0.050.030.980.870.520.941.12e-19 Ideal0.820.980.980.060.510.381.0 MSA base 0.79 96.6% 0.39 89.1% 0.87 89.2% 0.06 +100% 0.55 +100% 0.32 +100% 0.02 MSA instruct 0.8199.1%0.44100.0%0.8587.1%0.06+100%0.52+100%0.37+100%0.28 NPO0.66 81.0% 0.25 57.7% 0.92 94.1% 0.12 50.4% 0.54 +100% 0.31 +100% 3.25e-04 RMU0.85 +100% 0.10 22.9% 0.97 100.0% 0.06 +100% 0.52 +100% 0.25 +100% 0.94 GradDiff0.4656.6%0.2148.6%0.9092.0%0.2228.4%0.54+100%0.4288.8%6.03e-11 Task Vector0.85+100%0.2557.7%0.4647.3%0.05+100%0.4892.9%0.21+100%0.86 SatImp0.7287.8%0.2863.4%0.7778.9%0.0793.8%0.51+100%0.31+100%1.30e-05 UNDIAL0.5263.9%0.2658.3%0.8991.0%0.04+100%0.54+100%0.31+100%7.98e-17 Score = e (Model Utility) 2 (Acc forget )(Acc recover ) 2 (Acc retain )(1−extraction strength) 2 8 This score ensures that the chosen hyperparameters balance a good trade-off across metrics, with greater emphasis onAcc recover (as it measures ideal data-level unlearning), Model Utility (to ensure the model remains useful on related tasks), and extraction strength (a robust metric for unlearning evaluation). forget10– Llama-3.1-8B-InstructFor MSA and Task Vector,α∈0.5, 0.75, 1.0, 1.25, 1.5, 3.0 andβ ∈0.5, 1.0, 1.5, yielding15cases in total. The best-performingαandβare selected for final evaluation. For the baselines, we perform unlearning for5epochs and evaluate each checkpoint after every epoch: • NPO: λ∈2, 4, learning rate∈10 −5 , 2× 10 −5 , for 5× 2× 2 = 20 settings. • GradDiff: λ∈2, 4, learning rate 10 −5 , for 5× 2 = 10 settings. • UNDIAL: λ∈1, 2, 4, learning rate 2× 10 −5 , for 5× 3 = 15 settings. • SatImp: γ ∈4, 8, learning rate 10 −5 , β 1 = 5, β 2 = 1, for 5× 2 = 10 settings. • RMU: λ∈2, 4, learning rate 10 −5 , for 5× 2 = 10 settings. forget01 ,forget05, andforget10– Llama-3.2-1B-Instruct For the smaller Llama-3.2-1B- Instruct model, we can perform a more extensive hyperparameter search. For MSA and Task Vector, we setα ∈ 0.5, 0.75, 1.25, 1.5, 3.0andβ ∈ 0.5, 0.75, 1.0, 1.25, 1.5, yielding25cases in total. The best-performing α and β are used for the final evaluation. For baselines, we perform unlearning for10epochs and evaluate each checkpoint after every epoch: 17 Published as a conference paper at ICLR 2026 Table 8: Comparison of unlearning algorithms onTOFU(forget10). Model Llama-3.1-8B-Instruct is finetuned on TOFU, as the unlearning target. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain ES onD f ↓Model Utility↑ROUGE-L f ↓Forget Quality↑ Target0.030.021.000.980.570.998.12e-27 Ideal 0.980.981.000.070.600.391.00 MSA pretrained 0.82 95.1%0.4597.8%0.9292.2%0.0789.1%0.78+100%0.4099.5%0.64 MSA instruct 0.82 95.6%0.46100.0%0.9191.7%0.0797.8%0.5794.9%0.38+100%0.04 NPO0.7587.2%0.3882.2%0.8383.4%0.0881.0%0.5895.6%0.36+100%5.00e-05 RMU0.86100.0%0.1225.4%0.99100.0%0.0786.8%0.5997.7%0.19+100%0.03 GradDiff0.4957.3%0.2655.7%0.8887.9%0.2130.9%0.64+100%0.4587.2%3.91e-08 Task Vector 0.8093.3%0.2757.8%0.5151.5%0.03+100%0.5388.7%0.29+100%0.02 SatImp0.52 60.8% 0.28 61.6% 0.89 89.7% 0.15 44.5% 0.63 +100% 0.44 90.1% 1.02e-13 UNDIAL 0.4653.8%0.2962.2%0.8484.7%0.0879.7%0.65+100%0.4195.1%1.18e-17 • NPO: λ∈2, 4, 8, learning rate∈10 −5 , 2× 10 −5 , for 3× 2× 10 = 60 settings. • GradDiff: λ∈1, 2, 4, learning rate∈10 −5 , 2× 10 −5 , for 3× 2× 10 = 60 settings. • UNDIAL: λ∈1, 2, 4, learning rate∈10 −5 , 2× 10 −5 , for 3× 2× 10 = 60 settings. •SatImp:γ ∈ 0.1, 1.0, 4.0, learning rate∈ 10 −5 , 2 × 10 −5 ,β 1 = 5,β 2 = 1, for 3× 2× 10 = 60 settings. • RMU: α∈1, 2, 4, learning rate 10 −5 , for 3× 10 = 30 settings. Results for Llama-3.2-1B-Instruct are reported in Table 5 forforget01, Table 6 forforget05, and Table 7 for forget10. DEXPERIMENTS ON RESTOR We follow the procedure described by Rezaei et al. (2024), starting with Llama-3.1-8B-Instruct and OLMo-2-7B, and finetune them onRESTORfor5epochs using a learning rate of10 −5 , weight decay of0.01, and1warm-up epoch. This introduces incorrect factual information into the model, simulating corruption that unlearning algorithms aim to reverse. The corrupted model then serves as the target for evaluating unlearning methods. To tune hyperparameters, we hold out10%of theRESTORquestions as a validation set and evaluate accuracy on this subset. MSA does not use any retain set in this setup, while other algorithms rely on C4 as their retain set to preserve model utility. We evaluate MSA withα∈0.75, 1.0, 1.5, 2.0. For baselines, we perform unlearning for5epochs, evaluating the model on the validation set after each epoch. We setα = 4and a learning rate of10 −5 for GradDiff, NPO, RMU, and UNDIAL, and γ = 4, β 1 = 5, β 2 = 1 for SatImp. EEXPERIMENTS ON MUSE-BOOKS We follow the procedure described in Shi et al. (2024), finetuning each model for10epochs with a constant learning rate of 10 −5 . All experiments are run on 2 A100 GPUs. We use the OLMo-2-7B checkpoint as before for finetuning onMUSEbooks, as well as Llama-3-8B (we take a pretrained base model rather than instruct model to be consistent with Shi et al. (2024)) Forget and Retain Vectors To obtain forget and retain vectors for MSA, we use a checkpointC (depending on the model used). The forget vector is obtained by training on the unlearning target books for5epochs with a learning rate of10 −5 , weight decay of0.01, and1warm-up epoch. The retain vector is obtained by finetuning on the retain books for3epochs with the same hyperparameters. Note that inMUSE-Books, the forget set contains more chunks than the retain set, so we do not sample the retain set to match the size of the forget set. 18 Published as a conference paper at ICLR 2026 Table 9: Comparison of unlearning algorithms on MUSE-Books benchmark using Llama-3.1-8B. ModelES↓Exact Mem↓VerbMemD f ↓MIN-K%↓MIN-K% ++ ↓KnowMemD r ↑PrivLeak→ 0 Target0.640.960.651.001.000.62-100.00 Ideal0.020.520.160.510.470.640.00 MSA base 0.01+100%0.48+100%0.13+100%0.5298.7%0.5295.4%0.5595.0%-1.37 NPO0.0299.5%0.5889.8%0.14+100%1.0051.0%0.8458.8%0.58100.0%-99.90 RMU0.01+100%0.04+100%0.01+100%0.7469.1%0.6279.8%0.5289.9%-46.44 GradDiff0.01 +100%0.01+100%0.01+100%0.32+100%0.49100.0%0.2135.8%38.06 SatImp0.394.1%0.9555.2%0.4336.9%1.0051.0%1.0049.5%0.5493.3%-100.00 UNDIAL0.0279.7% 0.68 76.6% 0.17 91.4% 0.99 51.5% 0.99 50.0% 0.35 61.1% -98.15 Table 10: Comparison of MSA variants onTOFU(forget10). In this scenario, unlearning targets are not introduced at the very end of the training pipeline; instead, the model later undergoes finetuning on a subset of C4 for 2 epochs. MSA variants that use checkpoints prior to the unlearning targets, i.e., MSA base and MSA instruct , show acceptable performance, achieving values near the ideal model. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain ES onD f ↓Model Utility↑ROUGE-L f ↓Forget Quality↑ Final (TOFU)0.480.240.660.190.550.499.34e-13 Ideal (TOFU retain)0.830.450.690.070.550.381.31e-04 MSA base 0.7995.5%0.3987.6%0.6898.2%0.06+100%0.5397.8%0.34+100%0.42 MSA instruct 0.83100.0%0.45+100%0.70+100%0.06+100%0.55+100%0.36+100%0.70 MSA TOFU 0.7388.2%0.3782.6%0.70+100%0.0880.2%0.57+100%0.33+100%1.10e-09 Hyperparameter Selection We split theMUSE-Books benchmark into validation (15%) and test (85%) sets. As in theTOFUexperiments, we design a validation score to balance trade-offs across metrics: Score = e (1−MIN-K%)(1−MIN-K% ++ )(1−VerbMem f )(1−KnowMem r ) 2 (1−extraction strength) 2 (1−exact memorization) 8 We place stronger emphasis on extraction strength and knowledge memorization of the retain set, to ensure that knowledge of the retain set is preserved in the unlearned model. Unlearning AlgorithmsFor MSA, we setα∈0.75, 1.0, 1.5andβ ∈0, 0.75, 1.0, 1.5, select- ing the configuration that maximizes the validation score for test evaluation. For baselines, we setλ = 4for NPO, GradDiff, RMU, and UNDIAL, andγ = 4for SatImp. We perform unlearning for 5 epochs, evaluating each checkpoint on the validation set. Results for Llama-3.1-8B (as in Shi et al. (2024)) are shown in Table 9. We note thatKnowMem f , i.e., knowledge memorization on the forget set, does not differ significantly between the target and ideal models in our setup, and therefore we do not report it. FUNLEARNING TARGETS INTRODUCED MANY TOKENS BEFORE THE FINAL CHECKPOINT Most existing machine unlearning benchmarks (Maini et al., 2024; Rezaei et al., 2024; Shi et al., 2024) typically assume that the unlearning targets are introduced at the end of training, and we largely follow this setup to enable fair comparison with prior unlearning algorithms. Recent work (Yu et al., 2025) studies how the position of the unlearning targets in the training trajectory affects unlearning performance, and shows that the most challenging setting is indeed when the targets are introduced late in training. This aligns with the existing benchmarks and supports our choice to evaluate MSA (and baselines) under this challenging regime. Nevertheless, it is also important to understand scenarios in which the model is asked to forget information that was seen many tokens before the final checkpointθ D . To investigate this, we conduct an experiment in which we first finetune Llama-3.2-1B-Instruct onTOFUand then further finetune it 19 Published as a conference paper at ICLR 2026 Table 11: Comparison of MSA variants onTOFU(forget10). In this scenario, unlearning targets appear in the training data not just once, but twice, with2epochs of training on a subset of C4 between the two occurrences. MSA variants that use checkpoints prior to the unlearning targets, i.e., MSA base and MSA instruct , show acceptable performance, achieving values close to the ideal model. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain ES onD f ↓Model Utility↑ROUGE-L f ↓Forget Quality↑ Final (TOFU)0.040.030.990.940.540.966.16e-18 Ideal (TOFU retain)0.820.520.990.060.540.380.91 MSA base 0.75 98.7%0.3796.1%0.91100.0%0.08100.0%0.55+100%0.37+100%0.37 MSA instruct 0.76 100.0%0.38100.0%0.8998.3%0.0898.9%0.5499.9%0.3995.4%0.64 MSA TOFU 0.6788.2%0.3180.4%0.7178.5%0.0980.8%0.55+100%0.35+100%6.86e-10 MSA TOFU+C4 0.67 88.2%0.3591.5%0.8998.1%0.0981.6%0.58+100%0.3899.6%1.83e-05 MSA TOFU+C4+TOFU 0.67 88.2%0.3079.7%0.8188.7%0.1456.4%0.5499.9%0.3897.5%2.77e-09 on approximately20M tokens of C4. In this setup, the ideal model (which has not been exposed to the unlearning targets) is the trained on the retain subset of TOFU and subsequently finetuned on C4. Table 10 reports the results in this scenario. As seen there, MSA variants that use checkpoints taken before the introduction of the unlearning targets, namelyMSA base andMSA instruct , remain effective and achieve values close to the ideal model, even though the unlearning targets now lie many tokens before the final checkpoint. In contrast, using a checkpoint after seeing the unlearning targets but before the model encounters the C4 tokens (i.e., MSA TOFU ) underperforms on multiple metrics. These results provide empirical evidence that MSA can still work well when the model is asked to forget information learned a significant number of tokens earlier, while reinforcing our earlier observation that checkpoints taken after exposure to the forget set are less suitable for constructing effective unlearning updates. GUNLEARNING WITH REPEATED EXPOSURE TO TOFU We next consider a setting where the forget data appears multiple times in the training corpus and is not always close to the final checkpointθ D . To simulate this scenario, we start from Llama-3.2-1B- Instruct, first finetune it onTOFU, then train it on a subset of C4 (approximately 20M tokens), and finally finetune again onTOFU. This final model (TOFU+ C4 +TOFU) is the target of unlearning. The ideal model in this setup is trained on TOFU retain, then C4, then TOFU retain again. Table 11 reports the empirical results in this configuration. There are five natural checkpoints at which to apply MSA: (1) the base model, (2) the instruct model, (3) the model after the firstTOFU stage, (4) the model afterTOFU+ C4, and (5) the final model afterTOFU+ C4 +TOFU. As seen in the table, when MSA leverages checkpoints that precede any exposure toTOFU(i.e.,MSA base and MSA instruct ), it achieves strong performance, with values close to the ideal model. In contrast, using checkpoints that have already seen TOFU systematically underperforms. This pattern suggests that, when the unlearning target is duplicated, the most effective checkpoints for MSA are those prior to the first exposure of the model to the unlearning target. HAUGMENTING BASELINES WITH INTERMEDIATE CHECKPOINTS To investigate whether standard unlearning algorithms can also benefit from intermediate checkpoints, we apply these methods to earlier model states and then reuse the resulting update directions on the target model. More specifically, letθ 0 be an intermediate checkpoint. We apply a baseline unlearning algorithm starting fromθ 0 , obtaining a modelθ 1 . We then extract the change directionθ 1 − θ 0 and apply it to the target model θ D with a tunable scalar α, yielding θ unlearn = θ D + α(θ 1 − θ 0 ).(1) We select the optimal value of α via validation search, as we do for other methods. Table 12 reports experimental results on the TOFUforget10task with Llama-3.2-1B, where unlearn- ing algorithms are augmented with model checkpoints following the above procedure. For example, when applying NPO, we denoteNPO base andNPO instruct for NPO applied to the pretrained base model 20 Published as a conference paper at ICLR 2026 Table 12: Comparison of unlearning algorithms onTOFU(forget10). In this table, we consider leveraging model checkpoints for other unlearning algorithms. As seen in this table, applying a technique similar to MSA to other algorithms usually does not result in improved performance, instead degrading model utility and underperforming on other metrics. ModelGPT-4o Judge Metrics↑TOFU Metrics Acc forget Acc recover Acc retain ES onD f ↓Model Utility↑ROUGE-L f ↓Forget Quality↑ Final (TOFU)0.050.030.980.870.520.941.12e-19 Ideal (TOFU retain)0.820.980.980.060.510.381.0 MSA base 0.7996.6%0.3989.1%0.8789.2%0.06+100%0.55+100%0.32+100%0.02 MSA instruct 0.81 99.1%0.44100.0%0.8587.1%0.06+100%0.52+100%0.37+100%0.28 NPO0.6681.0% 0.25 57.7% 0.92 94.1% 0.12 50.4% 0.54 +100% 0.31 +100% 3.25e-04 NPO (base) 0.76 92.4%0.2966.9%0.5354.5%0.06+100%0.2752.5%0.24+100%9.99e-07 NPO (instruct)0.67 81.3%0.2454.3%0.7172.8%0.1158.3%0.5096.6%0.27+100%1.02e-13 RMU0.85 +100%0.1022.9%0.97100.0%0.06+100%0.52+100%0.25+100%0.94 RMU (base)0.95 +100% 0.04 8.6% 0.36 37.0% 0.04 +100% 0.35 68.5% 0.20 +100%5.00e-05 RMU (instruct)0.77 93.6% 0.19 43.4% 0.77 78.7% 0.08 81.9% 0.48 92.7% 0.32 +100% 1.49e-16 GradDiff0.4656.6%0.2148.6%0.9092.0%0.2228.4%0.54+100%0.4288.8%6.03e-11 GradDiff (base)0.6074.0% 0.20 45.1% 0.61 62.7% 0.09 67.3% 0.41 80.8% 0.38 98.3% 6.16e-18 GradDiff (instruct)0.7591.7%0.1534.3%0.4041.1%0.0877.4%0.2242.2%0.29+100%5.63e-20 SatImp0.7287.8%0.2863.4%0.7778.9%0.0793.8%0.51+100%0.31+100%1.30e-05 SatImp (base)0.82+100%0.1534.3%0.3131.6%0.05+100%0.2549.3%0.30+100%1.07e-08 SatImp (instruct)0.7288.1%0.2147.4%0.5151.9%0.0794.0%0.2854.7%0.30+100%2.24e-17 UNDIAL0.5263.9%0.2658.3%0.8991.0%0.04+100%0.54+100%0.31+100%7.98e-17 UNDIAL (base)0.7895.1%0.1124.6%0.3940.4%0.06+100%0.4077.8%0.29+100%1.49e-16 UNDIAL (instruct)0.82 +100% 0.10 22.3% 0.39 39.8% 0.06 +100% 0.41 79.8% 0.23 +100% 1.12e-19 and the instruct model, respectively, while NPO alone refers to the case where it is applied to the target model. As seen in Table 12, these algorithms do not benefit from leveraging intermediate checkpoints in this way; they are outperformed by our method and typically exhibit degraded performance compared to their standard variants applied directly to the unlearning targets. IPOTENTIAL OVERLAP WITH PRETRAINING DATA A potential limitation of our evaluation is that some of the datasets used may overlap with the pretraining data of the underlying models. In particular, if evaluation examples are present (or closely paraphrased) in the pretraining corpus, this could confound the interpretation of memorization and unlearning performance. We note thatTOFUandRESTORare both synthetic datasets that are unlikely to be part of the pretraining data. In fact,TOFUis explicitly constructed around fictional authors and works, precisely to reduce the risk of contamination from real-world corpora. However, theMUSE-Books benchmark may have some overlap with typical web-scale pretraining data. We acknowledge this as a limitation: while we do not believe it acts as a strong confounder for our main conclusions. JLLM USAGE In this paper, we leverage large language models (LLMs) to assist with refining and polishing our writing, as well as to generate code for the automated creation of tables from our experimental data. 21

Instant research discovery