Paper deep dive
OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C. Lipton, J. Zico Kolter, Pratyush Maini
Models: Llama-2, Zephyr
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 99%
Last extracted: 3/12/2026, 7:52:36 PM
Summary
OpenUnlearning is a standardized, extensible framework designed to unify and accelerate LLM unlearning research by integrating 13 unlearning algorithms, 16 evaluation metrics, and 3 major benchmarks (TOFU, MUSE, WMDP). It addresses fragmentation in the field by providing a modular pipeline for benchmarking methods and introduces a meta-evaluation benchmark to assess the faithfulness and robustness of evaluation metrics themselves using over 450 model checkpoints.
Entities (5)
Relation Signals (4)
OpenUnlearning → integrates → TOFU
confidence 100% · OpenUnlearning integrates 13 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP)
OpenUnlearning → integrates → MUSE
confidence 100% · OpenUnlearning integrates 13 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP)
OpenUnlearning → integrates → WMDP
confidence 100% · OpenUnlearning integrates 13 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP)
OpenUnlearning → supports → SimNPO
confidence 100% · We currently support 13 unlearning algorithms... SimNPO [16]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robust unlearning is crucial for safely deploying large language models (LLMs) in environments where data privacy, model safety, and regulatory compliance must be ensured. Yet the task is inherently challenging, partly due to difficulties in reliably measuring whether unlearning has truly occurred. Moreover, fragmentation in current methodologies and inconsistent evaluation metrics hinder comparative analysis and reproducibility. To unify and accelerate research efforts, we introduce OpenUnlearning, a standardized and extensible framework designed explicitly for benchmarking both LLM unlearning methods and metrics. OpenUnlearning integrates 13 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP) and also enables analyses of forgetting behaviors across 450+ checkpoints we publicly release. Leveraging OpenUnlearning, we propose a novel meta-evaluation benchmark focused specifically on assessing the faithfulness and robustness of evaluation metrics themselves. We also benchmark diverse unlearning methods and provide a comparative analysis against an extensive evaluation suite. Overall, we establish a clear, community-driven pathway toward rigorous development in LLM unlearning research.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
98,562 characters extracted from source content.
Expand or collapse full text
OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics Vineeth Dorna ∗† Anmol Mekala ∗† Wenlong Zhao † Andrew McCallum † Zachary C. Lipton ‡ J. Zico Kolter ‡ Pratyush Maini ‡↑ University of Massachusetts Amherst † Carnegie Mellon University ‡ DatologyAI ↑ vdorna,amekala@umass.edu;pratyushmaini@cmu.edu Abstract Robust unlearning is crucial for safely deploying large language models (LLMs) in environments where data privacy, model safety, and regulatory compliance must be ensured. Yet the task is inherently challenging, partly due to difficulties in reliably measuring whether unlearning has truly occurred. Moreover, frag- mentation in current methodologies and inconsistent evaluation metrics hinder comparative analysis and reproducibility. To unify and accelerate research ef- forts, we introduceOpenUnlearning, a standardized and extensible framework designed explicitly for benchmarking both LLM unlearning methods and metrics. OpenUnlearningintegrates 13 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP) and also enables analy- ses of forgetting behaviors across 450+ checkpoints we publicly release. Lever- agingOpenUnlearning, we propose a novel meta-evaluation benchmark focused specifically on assessing the faithfulness and robustness of evaluation metrics themselves. We also benchmark diverse unlearning methods and provide a com- parative analysis against an extensive evaluation suite. Overall, we establish a clear, community-driven pathway toward rigorous development in LLM unlearning research. 1 Introduction LLMs often memorize sensitive, copyrighted or harmful content from their vast training data, raising privacy [6], safety [67] and legal [31,61,43] concerns. Ever increasing costs of pre-training and post-training [23,54,55] prevent re-training in response to deletion requests [36]. This has motivated the development of machineunlearningtechniques that allow for “forgetting” training data via efficient post-training interventions [42,36]. The goal of unlearning is to eliminate the undesirable influences from specific training data, while maintaining the overall behavior and performance. There has been a recent surge in LLM unlearning research, yielding numerous proposed methods on several benchmarks. Modifying model weights to achieve unlearning is of the most interest, with many proposed approaches [76,65,33,16,40,11,29,66,17] . Concurrently, several benchmarks have been proposed to evaluate unlearning across a wide range of setups, covering aspects such as synthetic fine-grained unlearning, open-ended unlearning, knowledge, PII, memorization and privacy focused unlearning [39,44,46,52,33,44,57,30,14]. This volume of LLM unlearning research is marked by a notable fragmentation. Different benchmarks use different evaluations, with no consen- sus on the best evaluations and considerable criticism of existing evaluations [56,48,63,77,12,38]. Evaluating unlearning is a nuanced task involving knowledge, privacy, and utility desiderata, which is arguably as hard as achieving unlearning itself [49,37]. Unlearning research currently lacks a ⋆ These authors contributed equally to this work. 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Track on Datasets and Benchmarks. arXiv:2506.12618v2 [cs.CL] 9 Nov 2025 unified, standardized framework, with current method implementations often tied to specific setups. This fragmentation limits the ability to rigorously evaluate the efficacy of unlearning methods across diverse settings. We envision LLM unlearning evolving within a shared framework that continu- ously integrates new and improved methods and evaluations—where unlearning methods iteratively improve on benchmarks, and evaluation metrics themselves improve through meta-evaluation and critical feedback. To catalyze this vision, we introduceOpenUnlearning: a unified and extensible benchmark designed to standardize, scale, and accelerate progress in machine unlearning for LLMs. A unifying framework.We introduceOpenUnlearningas a one-stop repository for LLM unlearn- ing, consolidating widely-used benchmarks, unlearning methods, evaluation metrics under different interventions. It is easy to use and extend, enabling the enrichment of benchmarks and a deeper analysis of unlearning algorithms. Through this standardized framework, we foster unified research efforts and expedite the creation of effective unlearning techniques and benchmarks. Evaluating evaluations.Our framework moves the field towards a standardization of unlearning evaluations by conducting a meta-evaluation of unlearning metrics. To support this, we introduce a collection of over 450+ open-sourced models with known ground truth states, specifically designed to stress-test these metrics. This pool of models enables us to systematically compare 12 unlearning metrics against a set of desiderata that quantify their faithfulness (accuracy in detecting knowledge) and robustness (vulnerability to interventions). Together with corresponding meta-evaluation proce- dure, this forms the first benchmark of its kind for assessing and improving unlearning evaluation methods. Benchmarking unlearning techniques.We compare 8 unlearning methods using a suite of 10 metrics, following Ramakrishna et al.[47]’s ranking procedure. While SimNPO [16] performs the best, we also note limitations with the ranking methodology. We release all the evaluated model checkpoints to encourage further community research into principled LLM unlearning benchmarking. OpenUnlearninghas been open-sourced 1 under the MIT license. Since its release in March 2025, it has already garnered wide attention in the LLM unlearning community, sitting at 250+ GitHub stars, 20k+ model downloads across 450+ publicly released checkpoints, and popular unlearning benchmarks 2 now also point to our repository as the official point of maintenance for their work. 2 Overview of LLM Unlearning OpenUnlearninguses a common definition of LLM unlearning, where the goal is to eliminate the influence of “forget set” (D forget ), from an LLMf target to remove associated model capabilities [36]. The process pursues two primary goals: (i)Removal, ensuring influence caused only byD forget is substantially erased, and (i)Retention, maintaining the LLM’s utility on unrelated downstream tasks. The setup usually also involves a retain set disjoint from the forget set, used to aid and assess performance preservation. Formally, given an original modelf target trained on a dataset containingD forget , the unlearning process yields an unlearned modelf unlearn . The efficacy of unlearning is typically assessed using evalua- tion metrics,M, which quantify the remaining influence ofD forget onf unlearn —e.g., by computing M(f unlearn ,D forget ). Concurrently, utility metrics are used to measure the model’s performance on general tasks and data outside ofD forget , ensuring its overall capabilities are preserved. Unlearning methods:Some LLM unlearning approaches are prompting-based, detecting sensitive queries at inference time and deploying obfuscation mechanisms [4,41,19]. But these are not practically scalable as forgetting results accumulate. Of greater interest is the removal of the forget set’s influence directly from the weights. The techniques involved include finetuning with one or more of: (1) tailored loss functions [39,16,76,11,40], (2) optimization modifications [29,66,17], (3) localized parameter updates [33,10,20], and (4) alternative-data based approaches [40, 69, 7, 24, 39, 30]. Benchmarks:Fine-grained unlearningtypically focuses on erasing influence of specific training instances from a forget set while preserving performance on related instances not present in the forget set.TOFU[39] introduces fine-grainedknowledgeunlearning using QA-style data from 200 fictitious 1 Code§:github.com/locuslab/open-unlearning; Modelshuggingface.co/open-unlearning 2 TOFU [39]§github.com/locuslab/tofu; MUSE [52]§github.com/swj0419/muse_bench 2 Architecture 1.Llama 2/3 2.Zephyr 3.Phi 1.5/3.5 4.Gemma Load as 1.Full precision 2.Quantized 3.Probed model Model loader LLM Raw data Data worker Data Processor 1.Pretraining dataset 2.Chat dataset 3.Completion dataset 4.Unlearning dataset (forget x retain) Evaluator Benchmarks TOFU MUSE WMDP Metrics Stress Test via 1.Re-learning attacks 2.Quantization attack 3.Training a probe on a layer Interventions Trainer Finetune Trainer Unlearning Trainer 1.GradAscent 2.GradDiff 3.NPO 4.DPO 5.RMU 6.SimNPO Unlearned LLM Unlearned model Evaluations Stress tested model Evaluations Stress Tested LLM Really unlearned? 1.ROUGE 2.Probability 3.ES 4.EM 5.MIA 6.LM Eval Figure 1:OpenUnlearningis an extensible library for benchmarking LLM unlearning methods and metrics. It provides a unified framework for implementing unlearning methods, unlearning metrics, and stress-testing tools to verify unlearning robustness. This figure illustrates the unlearning pipeline in terms of implementation-level components. authors. KnowUndo [57] incorporates copyright and privacy aspects through datasets of books and synthetic author profiles. LUME [46] focuses on unlearning sensitive data from novels, biographies, and real-world figures. PISTOL [44] builds on TOFU with added structural relationships to study the effect of entity connectivity on knowledge unlearning.MUSE[52] also requires fine-grained unlearning, aiming to remove both knowledge, memorization and privacy influence of news articles and copyrighted books.Open-ended unlearningtasks do not target the removal of specific training data; instead, they aim to erase broader concepts or behaviors without access to a defined forget corpus.WMDPinvolves a safety-alignment focus, targeting which targets undesired behaviors from hazardous knowledge related to curated datasets [33]. RWKU [30] andWho’s Harry Potter(WHP) task [14] require forgetting all knowledge related to famous entities. While benchmarks like TOFU, MUSE, PISTOL, LUME, and KnowUndo involve creating task models by injecting new knowledge via finetuning with the forget dataset; WMDP, RWKU and WHP [14] operate directly on off-the-shelf LLMs to remove existing influence. Unlearning evaluations:Each benchmark task involves multiple evaluations metrics that judge for unlearning success and for general utility preservation. These range from simple probability judgements inTOFU, to MIA-attack based metrics inMUSE, with dozens of metrics across benchmarks in the literature. Evaluating unlearning success is difficult, with several subsequent works questioning the reliability of benchmark metrics in various aspects [32, 38, 62, 12, 77]. 3 OpenUnlearning The significant volume of research in LLM unlearning lacks unification both in technical imple- mentations and in both unlearning method implementation and unlearning evaluation methodology. Existing benchmarks are implemented with a structure that makes it difficult to integrate with newer ones, hindering their adoption, and creating barriers to reproducibility that slow down progress. More- over, unlearning methods and evaluation metrics aren’t consistently extended across benchmarks, preventing standardization and comprehensive comparative analysis. We give a few examples of this fragmentation that cover key parts of the unlearning pipeline, from unlearning algorithms, to data processing, and evaluations, 1. Fragmented evaluations of methods:New methods are not implemented in all benchmarks. For example: UNDIAL [11] is not implemented on any ofTOFU, MUSEandWMDP; NPO [76] is implemented with a different formulation forTOFUv/sMUSE; RMU was introduced only for WMDP etc. Similarly, evaluation metrics like MIA from MUSE [52] are not implemented in TOFU; and LM Eval Harness benchmarks used in WMDP can be extended to TOFU, MUSE. 3 Table 1: Overview of existingOpenUnlearningcomponents and their available feature variants. The design is easily extensible, allowing users to seamlessly contribute new features. ComponentVariants ModelsLLAMA-2, 3.1, 3.2 [59, 23]ZEPHYR-7B [60]PHI-1.5, 3.5 [34, 1] QWEN-2.5 [45]GEMMA[22] Unlearning algorithmsGradAscent, GradDiff, IdkDPO, IdkNLL [39]NPO [76]SimNPO [16] RMU [33]UNDIAL [11]AltPO [40]CE-U [71]PDU [15] WGA [64]SatImp [73] DatasetsTOFU: bios [39]WMDP: cyber, bio [33]MUSE: news, books [52] Evaluation suitesTOFU [39]MUSE [52]WMDP [33]LM Eval [21] Metrics Mem. Verbatim Prob. / ROUGE [39, 52]Knowledge QA- ROUGE [39, 52] Extraction Strength [5]Exact Memorization [58] Privacy Forget Quality [39]LOSS [74]ZLib [5]GradNorm [62] MinK [51]MinK++ [75]Privacy Leakage [52] UtilityTruth Ratio, Model Utility [39]LM-Eval [21] (WMDP, MMLU, etc.) Fluency [40] Stress testsRelearning [27, 38, 37, 63]Quantization [77]Probing [38, 50, 63] 2.Disparate implementations of core components:Several approaches involve customized loss functions [76,39,16,11] and others make adjustments to optimization steps [66,29,17]. These techniques could be modularized and reused across tasks for deeper investigation and a fair comparison. Evaluation metrics use many common functionalities which can be shared across metric implementations (eg. probability, ROUGE-score and MIA statistics). Dataset pre-processing is separately implemented across datasets and benchmarks, while there are many common data types: like the pre-training corpora in WMDP and MUSE, and chat-style prompts in TOFU and RWKU. Some works have proposed stress tests for assessing the robustness of unlearning which could easily be a common feature across benchmarks. To address this, we introduceOpenUnlearning: a unified, extensible pipeline that consolidates benchmarks, methods, evaluation metrics, datasets, and stress-tests under one roof (see Figure 1) to streamline unlearning implementations, benchmarking, and accelerate research. 3.1 Design of OpenUnlearning Figure 1 gives an overview ofOpenUnlearning’s components. Our framework is designed with ease- of-use and easy extensibility in mind. All features are implemented in a structured, modular fashion, simplifying the process for researchers to integrate new datasets, evaluation metrics, unlearning methods, and entire benchmarks. Hydra [70] is used for configuration management, withYAMLfiles specifying each pipeline component and experiment parameters. This helps users effortlessly swap in modules and easily launch an experiment with a single command. A variety of modules, including model-loaders, trainers, dataset preprocessors, evaluation suites, evaluation metrics, experiment types and stress-test interventions are joined together inOpenUnlearning(listed in Table 1). 3.2 Design of modules The procedure of extendingOpenUnlearningwith a new module variant generally involves two simple steps. (1)Create and register a handler.ThePythonclass or function encapsulating the component’s logic is implemented then registered to be accessed via a string key. (2)Create the 4 (a) Method implementation leveraging HuggingFace Trainer, followed by registration. from transformers import Trainer class Unlearner(Trainer): def compute_loss(self, ...): ... def get_optimizer_cls_and_kwargs(...): # custom optimizer ... def _inner_training_loop(self, ...): # modify training logic ... _register_trainer(Unlearner) (b) Configuration: create a YAML config specifying Training args and method parameters. handler: Unlearner# map registered name args:# HuggingFace Trainer args num_epochs: 10 learning_rate: 1e-5 optim: shampoo method_args: alpha: 1.0 switch_every_n: 10 retain_loss_type: NLL Figure 2: Illustration of implementing a hypothetical unlearning method inOpenUnlearning config.The configurationYAMLfile names the handler key and specifies its parameters. Figure 2 provides an example illustrating this procedure for a new unlearning method. Features:We currently support 13 unlearning algorithms, 8 model architectures, and 5 datasets ranging from chat to pretraining. Among existing benchmarks, we focus on the three most cited and usedTOFU[39], MUSE[52], WMDP[33] benchmarks. The framework includes a diverse set of metrics to assess model performance, including 16 unlearning metrics from existing benchmarks, as well as additional evaluations by integrating LM Eval Harness [21]. We also support three stress- testing approaches, which are essential for testing the robustness of unlearning, usually critical for model-owners in verifying compliance. All these features are summarized in Table 1 by component and variant. Our integration enriches each benchmark by enabling the use of metrics originally developed for others. For example, PrivLeak, initially introduced in MUSE, is now available in TOFU. More details on these technical benchmark improvements can be found in Appendix C.1. We also encourage community contributions by providing detailed guidelines for adding new benchmarks, unlearning methods, and evaluation metrics. This has already resulted in contributions from the community, with implementations for works like [11, 66, 72]. OpenUnlearningis a living framework, and our design choices are built keeping easy integration of new components in mind. For instance, since the public release of our repository (with justTOFU andMUSEbenchmarks) we introduced theWMDPbenchmark, unlearning methods like RMU [33], UNDIAL [11], AltPO [40]; evaluations like ES [5], EM [58], MIA [13] and integrated evaluations likeMUSE’s PrivLeak (intoTOFU) and LM Eval Harness [21] (to enableWMDPevaluation) among many others. Additionally, we encourage community contributions by providing detailed guidelines for adding new benchmarks, unlearning methods, and evaluation metrics. This has already resulted in contributions from the community, with implementations for works like [11,66,72]. Currently, each module supports several variants, with 3 popular LLM unlearning benchmarks, 5 task datasets, 13 unlearning methods, 16 evaluation metrics, 8 LLM architectures and 3 stress-tests. 4 Evaluating Unlearning Evaluations Reliable evaluations for unlearning are essential for regulatory compliance and data privacy, yet remain challenging [32,49,37], especially for LLMs, due to ambiguity between memorization and generalization. We propose two minimal necessary desiderata—FaithfulnessandRobustness—guided by our meta-evaluation framework, to promote trustworthy unlearning metrics (Figure 3). Our meta-evaluation uses a test-bed of models with known ground truths to objectively assess metrics. We employ theTOFUbenchmark [39] with the improvements described from Appendix C.1 with the forget10unlearning task (forgetting 10% ofTOFU) comprising 400 examples. We use theLLAMA- 3.2 1B model [23], analyzing 12 unlearning metrics adjusted to[0,1]scale (see Appendix C.1). While theTOFUbenchmark setup we choose makes simplifying assumptions about unlearning data distribution and target model behavior, such a synthetic setup enables controlled evaluation of metric properties that would be difficult to assess systematically with purely real-world data. With this approach, we are able to establish a minimal set of properties that any reliable unlearning evaluation metric should satisfy. 5 Metric M Positive Pool Negative Pool Base model Train on facts from forget set Best fitting threshold AUC = 1 AUC = 0.6 Unlearning with diverse methods Models unlearned according to metric M Quantization Relearning Quantized Pool Relearned Pool M before M after Learns faster than retain model Retain model Compare M before and after stress test Train on facts other than forget set Faithful metric Unfaithful metric Stress Testing 1 0 (a) Faithfulness (b) Robustness Model with Knowledge M before y=x Metric on quantization M after Retain model Un reliable Un reliable Figure 3:Meta-evaluation of unlearning metrics: (1)Faithfulness: the metric distinguishes models with and without target knowledge, reflected by high AUC; (2)Robustness: the metric value does not increase under benign changes (e.g., quantization) and does not improve faster than a retain model under non-benign changes (e.g., relearning). 4.1 Faithfulness Faithfulness Motivation.Unlearning evaluations may not faithfully reflect an LLM’s knowledge. Desideratum.A faithful metric accurately reflects the presence of targeted knowledge by assigning consistently higher scores to models possessing it than to those lacking it. LLMs often fail to regurgitate facts that remain encoded in their parameters when prompted, making it hard to tell whether a model truly forgot a target fact or simply refrained from exposing it [12,38, 48,63]. For example, work by Doshi and Stickland[12]shows that simple paraphrasing of inputs can yield a tenfold increase in evaluation scores on ‘unlearned’ models, indicating that the apparent forgetting may only be superficial. “Deeper” evaluation metrics aim to quantify this knowledge more faithfully, like Truth Ratio [39], GCG [18], or by using prompt engineering [63, 53, 56]. On the other hand, evaluation metrics can register misleadingly high scores without the presence of the target knowledge [39]. For example, in a question-answering evaluation using a simple ROUGE score, a model might achieve a high score by matching the parts of the target unrelated to the target fact. This calls for metrics that arefaithfulto the knowledge encoded in the model weights. We measurefaithfulnessas the ability of metrics to distinguish between models trained with the forget dataset’s knowledge (thepositive pool,P) and those trained without it (thenegative pool,N): (i) Each pool has 30 diverse models trained under varying conditions. (i) These variants present the targetforget10information for pool P models in diverse, challenging formats (e.g., biography vs. QA, paraphrases). Pool N models serve as negative controls, using similarly structured data lacking this target information using various perturbations and alternative datasets. (i) Metric scores yield two distributions:m(P),m(N)(forPandN), and we compute AUC-ROC to quantify their separability. (iv) We select a classification threshold optimizing accuracy, which is subsequently used in robustness tests. Faithfulness=AUC-ROC(m(P),m(N))(1) 6 0.30.40.5 AUC = 0.79 Faithfulness Pos. Neg. Threshold 0.00.20.4 Before 0.2 0.4 After Relearning y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 After Quantization y = x Unlearn Retain Unreliable Figure 4:For the ROUGE metric we evaluate faithfulness (left) and robustness to quantization (middle), and relearning (right). Faithfulness achieves an AUC of 0.79, indicating substantial prediction overlap between models trained with and without the target knowledge. Relearning robustness is 0.48, showing many unlearned models re-acquire knowledge faster than the retain model upon re-exposure. Quantization robustness is 0.93, reflecting no distinctive trend of metric spikes post-quantization. 4.2 Robustness Robustness Motivation.Unlearning evaluations can be vulnerable to stress-testing interventions. Desideratum.A robust metric’s positive assessment of unlearning should (1) not flip upon benign model interventions; and (2) behave comparably to a model truly unfamiliar with the data under non-benign interventions. Robustness of unlearning metrics is probed using various stress-test interventions. These include (1) relearning attempts, where the unlearned model is further trained to potentially recover the forgotten information [38,27,37,63]; (2) information extraction via manipulating the model’s internal representations [3,38,50,63,2]; and (3) applying techniques like quantization [77]. Benign interventions, such as model quantization or relearning on non-forget data, do not reintroduce the forgotten knowledge. In contrast, non-benign interventions—like relearning directly on the forget set—explicitly re-expose the model to the targeted data. These stress tests have revealed that several unlearning evaluation metrics may be unreliable, often signaling successful unlearning even when the underlying knowledge remains recoverable. For example, Zhang et al.[77]show that the PrivLeak metric [52] that previously reported a model as successfully unlearned can effectively ‘flip’ after a benign intervention, revealing that the targeted knowledge was perhaps never truly erased [77]. Such significant fluctuations under stress tests undermine the reliability of evaluation metrics. Furthermore, models unlearned with respect to a metric can exhibit high susceptibility on metric evaluation to non-benign interventions like relearning, where evaluation metrics show an unusually rapid return of the supposedly forgotten knowledge even with minimal retraining effort [17,16]. Robustness assesses stability under interventions such as relearning, probing and quantization. While probing was previously used by Wang et al. [63], Seyito ̆ glu et al.[50], Lynch et al.[38]to stress-test unlearning, in our setup, we found that probed models perform very poorly, with low scores across all metrics and show little discernible trends. Some probing results are shown in Appendix E.3. Robustness to Relearning:We evaluate metric scores before (m a ) and after (m b ) relearning on forget-set data. Then, we compare relative metric score recovery rates between unlearned (m unl ) and retain (m ret ) models, where higherRimplies greater robustness. r= m a ret −m b ret m a unl −m b unl , R= min(r,1).(2) Robustness to Quantization:We quantize models to 4-bit precision and compute scores before and after quantization, where higherQimplies greater robustness. q= m b unl m a unl , Q= min(q,1).(3) 7 Table 2: Meta-evaluation of 12 unlearning metrics for Faithfulness and Robustness. Robustness is assessed using two stress-testing methods: quantization and relearning, with their harmonic mean reported as Agg. An overall aggregation across both Faithfulness and Robustness is reported in the first Agg. column. Higher scores indicate better performance (↑) in all dimensions. The best values are shown in bold, and the second-best values are underlined. MetricsAgg.↑Faithful.↑ Robustness↑ Agg.↑Quant.↑Relearn↑ Extraction Strength0.850.920.790.950.68 Exact Mem.0.800.900.720.920.59 Truth Ratio0.730.950.590.920.43 Para. Prob.0.730.710.75 0.600.98 Para. ROUGE0.720.890.610.930.45 Probability0.720.820.650.600.70 ROUGE0.700.790.640.930.48 Jailbreak ROUGE0.690.830.590.850.45 MIA - ZLib0.710.920.570.560.59 MIA - MinK0.670.930.520.480.57 MIA - LOSS0.660.93 0.520.480.57 MIA - MinK++0.610.810.480.610.40 4.2.1 Realistic Model Filtering We enforce practical constraints by filtering models with: (i) Utility drops exceeding 20%. (i) Insufficient unlearning w.r.t. the considered metric (more than the threshold computed in §4.1’s faithfulnessanalysis). Models which exhibit substantial model utility drops are unusable in practice and thus unlikely to inform robustness. Additionally, models that aren’t unlearned w.r.t a metric are uninteresting for robustness analysis, since they do not reflect realistic scenarios where some kind of unlearning is observed before models are stress tested. The case of interest is when an ostensibly performant LLM exhibits low scores according to a chosen metric, indicating unlearning, and practitioners require confidence in the metric’s judgement. We analyze roughly 400 diverse models from various unlearning methods to reflect realistic use cases. We ensure diversity by using models unlearned using the GradDiff, IdkDPO, IdkNLL [39], NPO [76], SimNPO [16], AltPO [40], UNDIAL [11] and RMU [33] unlearning methods (methods described in Appendix §C.5 and hyperparameters in §F.2). This aligns the distributions between the unlearned model pools used in our analysis and unlearned models selected by practitioners. 4.3 Aggregation of Metrics We consolidate evaluations through harmonic mean, ensuring balanced performance across criteria: Robustness=HM(R,Q),Overall=HM(Faithfulness,Robustness)(4) An effective unlearning metric must be both faithful in representing unlearning and robust in its measurements; a trivial constant-value metric, for instance, would be robust but entirely unfaithful. To holistically assess a metric, we aggregate these distinct qualities using the Harmonic Mean (HM), as this ensures that a high final score demands strong performance in all constituent parts. Figure 4 illustrates these distributions and scores for the ROUGE metric as an example. Further methodological considerations, including comparisons to prior work, are detailed in Appendix E.4. 4.4 Results and Discussion Table 2 highlights key insights: (i)Extraction Strength (ES)[5] emerges as most reliable overall, aligning with Wang et al.[63]. (i)Truth Ratiohas superior faithfulness but lower robustness, ranking third overall. (i) Metrics based on raw probabilities or ROUGE scores have moderate faithfulness and robustness, limiting their reliability. (iv) Membership inference (MIA)-based metrics demonstrate high faithfulness but lack robustness, cautioning against relying solely on MIA metrics for assessing unlearning. This sensitivity raises concerns about the reliability of the MIA-based privacy assessments 8 Table 3: Comparison of unlearning methods on theTOFUtask, showing overall aggregate (Agg.), memorization (Mem.), privacy (Priv.), and utility (Utility) scores. Higher scores indicate better performance (↑). Initial finetuned is the target model before unlearning and Retain model is the gold standard target model. The best values are shown in bold, and the second-best values are underlined. MethodAgg.↑Mem.↑Priv.↑Utility↑ Init. finetuned0.000.000.101.00 Retain0.580.311.000.99 SimNPO [16]0.530.320.631.00 RMU [33]0.520.470.500.61 UNDIAL [11]0.420.270.480.78 AltPO [40]0.150.630.060.95 IdkNLL [39]0.150.080.170.93 NPO [76]0.150.520.060.99 IdkDPO [39]0.140.560.060.95 GradDiff [39]9e-30.973e-30.79 in unlearning contexts as introduced by Shi et al.[52], as even benign interventions can reverse unlearning effects, as observed in Zhang et al. [77]. Our extensive model testbed supports ongoing development of improved, practical unlearning metrics. Our testbed comprising 450+ models — including those from poolsP,N, and various unlearned model checkpoints — offers a valuable platform for the creation and rigorous assessment of improved unlearning evaluation metrics. Metrics validated on this testbed can then be applied with greater confidence to real-world unlearning scenarios. Our overarching goal is to stimulate the development of more faithful and trustworthy metrics, leveraging the insights from our meta-evaluation framework. This meta-evaluation setup can be expanded by incorporating more diverse unlearning setups, model architectures and newer methods. Newer adversarial model setups will be needed to challenge metrics as they improve on existing testbeds. Such a dynamic approach ensures that unlearning methods and their meta-evaluations can mutually inform each other, driving progress as unlearning research advances. 5 Benchmarking Unlearning Methods Unlike prior works with limited baselines and metrics,OpenUnlearningprovides a standardized and scalable framework to conduct a large-scale comparison of various unlearning methods. We demonstrate this by evaluating 8 unlearning methods using 10 evaluation metrics on TOFU. Unlearning methods:OpenUnlearningenables evaluation across a broader range of methods, including SimNPO [16], RMU [33], AltPO [40], NPO [76], UNDIAL [11], as well as baselines like IdkPO, IdkNLL, and GradDiff [39]. See Appendix C.5 for each method’s definition. Evaluation metrics:We evaluate unlearning methods using memorization metrics validated in our meta-analysis, alongside privacy and utility metrics. Using theTOFUbenchmark, and following the SemEval 2025 LLM Unlearning Challenge’s ranking procedure [47], we compute a composite score by aggregating metrics from the three categories: memorization (using the 4 top-performing knowledge metrics from §4’s metric meta-evaluation: ES, EM, Truth Ratio, Paraphrased Probability), privacy (4 MIA metrics), and utility (2 metrics, including TOFU’s Model Utility and forget-set fluency). Exact details of our metric aggregation are in Appendix F.1. Note that the memorization score (reported in Table 3) corresponds to forgetting: higher Mem. indicates less knowledge. Tuning strategy:To ensure fairness, 27 hyperparameter tuning trials are allocated per method, as tuning can significantly improve performance of even simple baselines [63]. Due to the impracticality of tuning on privacy metrics, that require the presence of i.i.d. holdout datasets and oracle retain models (i.e., models trained solely on the retain set, with no exposure to the forget set), we validate models only on accessible metrics that capture memorization and utility. Additionally, model selection during tuning can significantly affect rankings (Appendix F.2). 9 Results and discussion:While memorization, privacy, and utility each capture a distinct aspect of unlearning quality, aggregating them using a harmonic mean (Table 3), results in SimNPO [16] ranking first. Although its memorization score trails that of others, it remains close to the retain model’s level, avoiding over-unlearning. SimNPO fully preserves utility and achieves competitive privacy results, striking a balance across all three criteria. The next best performer is RMU, which demonstrates strong memorization and privacy but suffers a significant drop in utility. Here, we note a tradeoff between reducing memorization and improving data privacy during the unlearning process. Memorization evaluation penalizes high likelihood on forget data; while privacy metrics penalize both unusually high and low likelihoods. Thus, methods that under-unlearn (e.g. IdkNLL, which yields a low memorization score i.e. less forgetting) score lower on privacy. On the other hand, methods like GradDiff over-unlearn, forgetting too aggressively, yielding a high memorization score. This leads to poor privacy performance, as the model’s behavior deviates significantly from that of the retain model. This suggests that detecting and halting unlearning once the model’s behavior has reverted to its “default” state is crucial to ensure privacy. Because different ranking schemes can produce very different rankings (Table 3 v/s Appendix Table 6), it is critical to choose an appropriate method ranking procedure and aggregate metrics. Additionally, there is a lack of standardization on which metrics are suitable for model selection versus final evaluation (elaborated upon in Appendix F.2). While identifying the ideal ranking method and model selection approach is beyond our scope, we release all unlearned model checkpoints from our study to support future research on fair evaluation. 6 Conclusion The field of LLM unlearning has faced challenges due to fragmented methodologies and inconsistent evaluations. To address this, we introducedOpenUnlearning, a standardized and extensible frame- work that unifies research efforts by integrating 13 unlearning algorithms, 16 evaluation metrics, and 3 major benchmarks. This comprehensive platform enabled us to conduct a novel meta-evaluation of unlearning metrics, assessing their faithfulness and robustness, and to perform large-scale bench- marking of unlearning methods. Our meta-evaluation identified Extraction Strength (ES) and Exact Memorization (EM) as particularly reliable metrics, with Truth Ratio also showing high faithfulness. Benchmarking revealed SimNPO and RMU as strong performers, though we also observed significant sensitivities in ranking. At the same timeOpenUnlearning, by providing a common ground and releasing numerous model checkpoints, establishes a clear pathway for the community towards more rigorous, reproducible, and accelerated development of robust unlearning techniques and evaluation protocols, ultimately fostering safer AI deployments. 7 Acknowledgments We thank all contributors for adding new unlearning methods and metrics. We also appreciate their continued support through active use of the repository and valuable feedback that helps improve the codebase. We also acknowledge the IESL lab at University of Massachusetts Amherst for providing compute resources for this work. PM is supported by funding from the DARPA GARD program and OpenAI’s Cybersecurity Grant Program. References [1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024. [2] Andy Arditi and Bilal Chughtai.Unlearning via RMU is mostly shallow, July2024.URLhttps://w.lesswrong.com/posts/6QYpXEscd8GuE7BgW/ unlearning-via-rmu-is-mostly-shallow . AI Alignment Forum, informal research note. [3]Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112, 2023. 10 [4]Karuna Bhaila, Minh-Hao Van, and Xintao Wu. Soft prompting for unlearning in large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4046–4056, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. URLhttps://aclanthology.org/2025.naacl-long.204/. [5] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021. [6]Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview. net/forum?id=TatRHT_1cK. [7] Minseok Choi, Daniel Rim, Dohyun Lee, and Jaegul Choo. Opt-out: Investigating entity-level unlearning for large language models via optimal transport, 2024. URLhttps://arxiv.org/ abs/2406.12329. [8]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. [9]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.QLoRA: Efficient finetuning of quantized llms.In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Informa- tion Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc., 2023.URLhttps://proceedings.neurips.c/paper_files/paper/2023/file/ 1feb87871436031bdc0f2beaa62a049b-Paper-Conference.pdf. [10] Chenlu Ding, Jiancan Wu, Yancheng Yuan, Jinda Lu, Kai Zhang, Alex Su, Xiang Wang, and Xiangnan He. Unified parameter-efficient unlearning for LLMs. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=zONMuIVCAT. [11]Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli ́ c. UNDIAL: Self-distillation with adjusted logits for robust unlearning in large language models. InPro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8827–8840, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. URLhttps://aclanthology.org/2025.naacl-long.444/. [12] Jai Doshi and Asa Cooper Stickland. Does unlearning truly unlearn? A black box evaluation of LLM unlearning methods.arXiv preprint arXiv:2411.12103, 2024. [13]Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettle- moyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=av0D19pSkU. [14]Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in LLMs. arXiv preprint arXiv:2310.02238, 2023. [15]Taha Entesari, Arman Hatami, Rinat Khaziev, Anil Ramakrishna, and Mahyar Fazlyab. Con- strained entropic unlearning: A primal-dual framework for large language models, 2025. URL https://arxiv.org/abs/2506.05314. [16]Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for LLM unlearning. In Neurips Safe Generative AI Workshop 2024, 2024. URLhttps://openreview.net/forum? id=pVACX02m0p. 11 [17]Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. To- wards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond.arXiv preprint arXiv:2502.05374, 2025. [18] Rohit Gandikota, Sheridan Feucht, Samuel Marks, and David Bau. Erasing conceptual knowl- edge from language models.arXiv preprint arXiv:2410.02760, 2024. [19]Chongyang Gao, Lixu Wang, Chenkai Weng, Xiao Wang, and Qi Zhu. Practical unlearning for large language models.arXiv preprint arXiv:2407.10223, 2024. [20]Lei Gao, Yue Niu, Tingting Tang, Salman Avestimehr, and Murali Annavaram. Ethos: Rectifying language models in orthogonal parameter space.arXiv preprint arXiv:2403.08994, 2024. [21]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The Language Model Evaluation Harness, 07 2024. URLhttps://zenodo.org/records/ 12608602. [22] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024. [23]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. URLhttps://arxiv.org/ abs/2407.21783. [24]Tianle Gu, Kexin Huang, Ruilin Luo, Yuanqi Yao, Yujiu Yang, Yan Teng, and Yingchun Wang. Meow: Memory supervised llm unlearning via inverted facts, 2024. URLhttps: //arxiv.org/abs/2409.11844. [25]Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable.https://github.com/huggingface/accelerate, 2022. [26]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URLhttps: //arxiv.org/abs/2009.03300. [27]Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Unlearning or obfuscating? Jog- ging the memory of unlearned LLMs via benign relearning. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=fMNRYBvcQN. [28] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, and Yuxiong He. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509, 2023. URL http://arxiv.org/abs/2309.14509. [29] Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for LLM unlearning.arXiv preprint arXiv:2404.18239, 2024. [30] Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. RWKU: Benchmarking real-world knowledge unlearning for large language models.arXiv preprint arXiv:2406.10890, 2024. [31]Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URLhttps://openreview.net/forum?id=YokfK5VOoz. 12 [32]Yongwoo Kim, Sungmin Cha, and Donghyun Kim. Are we truly forgetting? A critical re- examination of machine unlearning evaluation protocols.arXiv preprint arXiv:2503.06991, 2025. [33]Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The WMDP benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024. [34]Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks Are All You Need I: Phi-1.5 technical report, 2023. URLhttps://arxiv. org/abs/2309.05463. [35]Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summariza- tion Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URLhttps://aclanthology.org/W04-1013/. [36]Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models.arXiv preprint arXiv:2402.08787, 2024. [37]Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for AI safety.Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URLhttps://openreview.net/forum?id= J5IRyTKZ9s. [38] Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in LLMs.arXiv preprint arXiv:2402.16835, 2024. [39]Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. TOFU: A task of fictitious unlearning for LLMs.First Conference On Language Modeling, 2024. URL https://openreview.net/pdf?id=B41hNBoWLo. [40]Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, and Elita Lobo. Alternate preference optimization for unlearning factual knowledge in large language models. InProceedings of the 31st International Conference on Computational Linguistics, pages 3732–3752, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URLhttps://aclanthology.org/2025.coling-main.252/. [41]Andrei Muresanu, Anvith Thudi, Michael R. Zhang, and Nicolas Papernot. Unlearnable algorithms for in-context learning, 2024. URLhttps://arxiv.org/abs/2402.00751. [42]Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning.arXiv preprint arXiv:2209.02299, 2022. [43] CA OAG. CCPA regulations: Final regulation text.Office of the Attorney General, California Department of Justice, 2021. [44]Xinchi Qiu, William F Shen, Yihong Chen, Nicola Cancedda, Pontus Stenetorp, and Nicholas D Lane. PISTOL: Dataset compilation pipeline for structural unlearning of LLMs.arXiv preprint arXiv:2406.16810, 2024. [45]Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URLhttps://arxiv.org/abs/2412.15115. [46]Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanukiran Vinzamuri, Volkan Cevher, Mingyi Hong, and Rahul Gupta. LUME: LLM unlearning with multitask evaluations.arXiv preprint arXiv:2502.15097, 2025. 13 [47]Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanukiran Vinzamuri, Volkan Cevher, Mingyi Hong, and Rahul Gupta. SemEval-2025 Task 4: Unlearning sensitive content from large language models.arXiv preprint arXiv:2504.02883, 2025. [48] Yan Scholten, Stephan Günnemann, and Leo Schwinn. A probabilistic perspective on unlearning and alignment for large language models.arXiv preprint arXiv:2410.03523, 2024. [49]Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. Re- thinking llm memorization through the lens of adversarial compression. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 56244–56267. Curran Associates, Inc., 2024. URLhttps://proceedings.neurips.c/paper_files/paper/2024/file/ 66453d578afae006252d2ea090e151c9-Paper-Conference.pdf. [50] Atakan Seyito ̆ glu, Aleksei Kuvshinov, Leo Schwinn, and Stephan Günnemann. Extracting unlearned information from LLMs with activation steering.arXiv preprint arXiv:2411.02631, 2024. [51]Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models, 2023. [52]Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine unlearning six- way evaluation for language models. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=TArmA033BU. [53]Adam Shostack. The boy who survived: Removing Harry Potter from an LLM is harder than reported.arXiv preprint arXiv:2403.12082, 2024. [54]Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. [55] Nvidia Team, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron- 4 340B technical report.arXiv preprint arXiv:2406.11704, 2024. [56]Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, and Virginia Smith. Position: LLM unlearning benchmarks are weak measures of progress. InProceedings of the 3rd IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2025. [57]Bozhong Tian, Xiaozhuan Liang, Siyuan Cheng, Qingbin Liu, Mengru Wang, Dianbo Sui, Xi Chen, Huajun Chen, and Ningyu Zhang. To forget or not? towards practical knowledge unlearning for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1524–1537, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.82. URLhttps://aclanthology.org/ 2024.findings-emnlp.82/. [58]Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022. [59]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. [60]Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. InFirst Conference on Language Modeling, 2024. URLhttps: //openreview.net/forum?id=aKkAwZB6JV. 14 [61]European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council. Official Journal of the European Union, 2016. [62]Jeffrey G. Wang, Jason Wang, Marvin Li, and Seth Neel. Pandora’s white-box: Precise training data detection and extraction in large language models, 2024. URLhttps://arxiv.org/ abs/2402.17012. [63]Qizhou Wang, Bo Han, Puning Yang, Jianing Zhu, Tongliang Liu, and Masashi Sugiyama. Towards effective evaluations and comparisons for LLM unlearning methods. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview. net/forum?id=wUtCieKuQU. [64]Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, and Kilian Q Weinberger. Rethinking LLM unlearning objectives: A gradient perspective and go beyond. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview. net/forum?id=huo8MqVH6t. [65]Yaxuan Wang, Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Shah, Yujia Bao, Yang Liu, and Wei Wei. LLM unlearning via loss adjustment with only forget data. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=6ESRicalFE. [66]Yue Wang, Qizhou Wang, Feng Liu, Wei Huang, Yali Du, Xiaojiang Du, and Bo Han. GRU: Mitigating the trade-off between unlearning and retention for large language models.arXiv preprint arXiv:2503.09117, 2025. [67]Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? InThirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum?id=jA235JGM09. [68]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. URLhttps://w.aclweb.org/anthology/ 2020.emnlp-demos.6. [69]Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, and Ningyu Zhang. Relearn: Unlearning via learning for large language models.arXiv preprint arXiv:2502.11190, 2025. [70]Omry Yadan. Hydra - A framework for elegantly configuring complex applications. Github, 2019. URLhttps://github.com/facebookresearch/hydra. [71] Bo Yang. Ce-u: Cross entropy unlearning, 2025. URLhttps://arxiv.org/abs/2503. 01224. [72] Bo Yang. CE-U: Cross Entropy unlearning.arXiv preprint arXiv:2503.01224, 2025. [73] Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. Ex- ploring criteria of loss reweighting to enhance LLM unlearning. InForty-second Interna- tional Conference on Machine Learning, 2025. URLhttps://openreview.net/forum? id=mGOugCZlAq. [74]Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE, 2018. [75]Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-K%++: Improved baseline for pre-training data detection from large language models. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=ZGkfoufDaU. 15 [76]Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning.First Conference on Language Modelling, 2024. URLhttps://openreview.net/pdf?id=MXLBXjQkmb. [77]Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure of LLM unlearning via quantization. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps:// openreview.net/forum?id=lHSeDYamnz. 16 Appendix A Limitations We also note some limitations of our framework and analysis. Firstly, it is limited by the existing popular benchmarks its supports, which have been regarded as “weak measures of unlearning progress” [56]. The setups may not accurately reflect realistic model learning or unlearning dynamics, with the underlying forget-retain paradigm itself warranting further scrutiny [56]. There’s a clear need for more realistic, yet controlled, fine-grained unlearning benchmark setups beyond the currently popular benchmarks. Secondly, while our meta-evaluation of metrics and comparison of methods is a valuable step, its findings need to be extended to more unlearning setups and unlearning algorithms, to gain a greater understanding of the best and comprehensive ways to quantify unlearning. Finally, while our meta-evaluation focuses on knowledge faithfulness and metric robustness as minimal desiderata, these might not be a comprehensive set of desiderata for good unlearning metrics. B Broader Impact The widespread deployment of AI systems in domains ranging from conversational assistants and recommendation systems to self-driving vehicles and medical diagnostics raises important concerns about privacy, safety, and regulatory compliance. As these systems are deeply integrated within society, the ability to remove unwanted or sensitive information from deployed models (“unlearning”) is essential to maintain safety, reliability and uphold legal requirements. Our work on a unified, extensible LLM unlearning benchmark accelerates progress toward reliable, scalable unlearning solutions. By standardizing implementations of unlearning methods, evaluation metrics, and stress tests across diverse tasks and datasets, we lower the barrier for both academic and industrial adoption. This facilitates rapid iteration on novel techniques, ensures consistent measurement of privacy and utility trade-offs, and enables model governance workflows that can respond promptly to deletion or correction requests. In the long run, advances enabled by this framework will support trustworthy AI deployment in safety-critical and highly regulated settings. From ensuring that autonomous vehicles do not retain outdated or hazardous driving data, to empowering personalized assistants with user-controlled memory, robust unlearning mechanisms will be a cornerstone of ethical, privacy-preserving machine learning. By fostering community collaboration and transparent evaluation, our research paves the way for AI systems that adapt responsibly to evolving societal norms and regulatory landscapes. C Additional details on OpenUnlearning’s components C.1 Unlearning benchmarks TOFU:A synthetic fine-grained knowledge-unlearning benchmark with 200 fictitious author profiles, each offering 20 QA pairs and a defined “forget set”, and a finetuned chat LLM.TOFU’s primary metric is Truth Ratio, which measures therelativelikelihood of the true answer after unlearning. MUSE:A memorization and knowledge unlearning benchmark targeting the removal of books and news articles from a finetuned LLM. MUSE evaluates for memorization (via verbatim reproduc- tion rates), knowledge (via question-answers) and privacy protection (using membership inference attacks). WMDP:An alignment-focused benchmark of 3,668 multiple-choice questions probing hazardous knowledge in biosecurity, cybersecurity, and chemical security, paired with corresponding unlearn- ing corpora and off-the-shelf chat LLMs. WMDP assesses a model’s ability to forget dangerous capabilities while preserving general performance. Improvements:In evaluations,TOFUreuses training questions, raising concerns about overfitting and inflated metrics. To mitigate this, we evaluate on paraphrased questions in our meta-evaluation and benchmarking. We also extendTOFUwith privacy-based metrics fromMUSEvia PrivLeak [52] and introduce additional MIA attacks. For this we create new holdout datasets by replicating the 17 originalTOFUdata generation setup. 3 We add MIA beyond Min-K [51] toMUSE. Given the poor quality and tokenization issues users faced with thePHI-1.5 andLLAMA-2 models fromTOFU, we introduce new starter target models.OpenUnlearningprovides three sizes of the recentLLAMA-3 models: 1B, 3B, and 8B, giving users greater flexibility to experiment. Additionally, we augment both TOFUandMUSEwith metrics such as Extraction Strength [5], Exact Memorization [58], and Forget Fluency [40]. We integrateOpenUnlearningwith LM Eval Harness [21] to assess general LLM capabilities that identify post-unlearning degradations, in addition to enablingWMDPevaluations. Several contemporary works can further enhance these benchmarks. We plan to continuously improve the framework by adding-and encouraging contributions of-new features and metrics to both existing and future benchmarks, such as the recent work by Thaker et al. [56]. C.2 Datasets In machine unlearning, benchmarks typically structure data into two primary components: (1) forget sets, which contain text corpora and queries designed to test whether the model has successfully erased targeted information, and (2) retain sets, which verify that the model preserves unrelated, desirable knowledge. Beyond this fundamental split, unlearning benchmarks often include additional variations to test algorithmic robustness. For example, scaling splits vary the size of the forget set to assess how well algorithms handle larger deletion requests, while topic-based splits examine whether forgetting specific content impacts retention across semantically related or unrelated domains [39,52]. These nuanced splits are essential for assessing scalability, generalization, and sustainability of unlearning methods under realistic conditions. (a) Dataset Handler class PretrainingDataset(Dataset): def __init__(self, hf_args, ...): ... def __getitem__(self, idx): ... return item _register_data(PretrainingDataset) (b) Dataset Configuration MUSE_forget: handler: PretrainingDataset args: hf_args: path: "muse-bench/MUSE-News" name: "raw" split: "forget" text_key: "text" max_length: 2048 Figure 5: Adding a dataset inOpenUnlearning: (a) the Python handler class implementing data preprocessing and reusable to load several datasets, and (b) the configuration file specifying arguments for instantiating a particular dataset variant. Adding variants of other modules (e.g. unlearning method trainers, benchmarks, evaluation metrics etc.) involves a similar procedure. OpenUnlearningprovides a modular framework where most of the Python implementation for dataset classes is shared across various dataset configurations and benchmarks. It also allows users to define custom dataset classes following the steps presented in Figure 5. We already support three commonly used dataset handlers, each serving a distinct purpose in the unlearning pipeline: •PretrainingDataset: used for training models on large-scale web corpora; essential for simu- lating pre-training settings. •CompletionDataset : used for evaluating model outputs in a zero-shot or few-shot setting. This format is particularly useful for measuring memorization and information leakage, such as verbatim reproduction of forgotten content. •QADataset : designed for probing models using natural language question-answer interactions, optionally with few-shot examples. This format is critical for assessing whether the model retains or forgets factual knowledge in interactive settings. Moreover, the framework automatically pipelines model-specific input formatting such as including system prompts or special tokens for chat-based models ensuring that queries are executed in a manner consistent with the model’s native interface. •ForgetRetainDataset: The unlearning process involves simultaneous optimization on both the forget and retain datasets, requiring concurrent batch loading. This dataset class abstracts this by loading the retain dataset in the same order as the forget dataset for unlearning. 3 We use the samegpt-4-1106-previewendpoint and prompts for data generation. 18 C.3 Metrics OpenUnlearningsupports multiple evaluation metrics and shares common functionalities across metric implementations. Metrics are broadly classified into three categories and summarized below: Memorization Metrics:These metrics quantify how much the model has memorized information from its training data. 1.Exact Memorization (EM):Quantifies memorization by calculating proportion of tokens in the model’s response that exactly match those in the ground truthy[58]. Formally, it is defined as EM= 1 |y| X k 1 arg max y f(y|[x,y <k ];θ) =y k ,(5) 2.Extraction Strength (ES):Quantifies the intensity of memorization by determining the minimal prefix length required to reconstruct the remaining suffix [5]. ES= 1− 1 |y| min k n k|f([x,y <k ];θ) =y >k o .(6) 3.Probability (Prob.):Directly quantifies the model’s confidence in its output. Probability=p f(y|x) (7) 4. Paraphrased Probability (Prob.):Probability computed on a paraphrased answery para to remove template bias. Para. Prob.=p f(y para |x) (8) 5.ROUGE/Paraphrased ROUGE:Assesses the degree of overlap between the model’s output f(x)and the ground truthy[35]. This can be computed against many variants of datasets, including paraphrases and jailbreak prompts (next). 6.Jailbreak ROUGE:To probe for forgotten information, we employ a prefix-based jailbreaking attack by prompting the model with"Sure, here is the answer:"(as in [63]) and then computing the ROUGE score between the model’s response and the ground truth. This metric captures the extent to which suppressed content can still be recovered through prompt manipulation. 7.Truth Ratio:Measures the model’s preference for the correct answer over a perturbed (incor- rect) alternative by comparing their predicted probabilities. A higher value indicates stronger confidence in the correct response. It is defined as: Truth Ratio= p(y para |x) p(y para |x) +p(y pert |x) (9) wherey para denotes the paraphrased correct answer andy pert represents an incorrect alternative with similar structure. Note that Maini et al.[39]use a privacy-oriented variant of Truth Ratio computed asTruth Ratio=min( p(y para |x) p(y pert |x) , p(y pert |x) p(y para |x) ) . We modify it so that it quantifies extent of knowledge for our work’s purposes. Privacy Metrics:These metrics ascertain whether sensitive information from the forget set can still be inferred or extracted from the model. Techniques such as Membership Inference Attacks (MIA) are utilized to evaluate the model’s susceptibility to revealing whether specific data points were part of its training set, thereby assessing the privacy guarantees post-unlearning. However, these metrics often assume access to perfectly i.i.d. holdout splits or to an “oracle” retain model, limiting their practical usefulness in real-world settings. 1.MIA:Evaluates a model’s tendency to memorize training data by testing whether an adversary can distinguish between seen examples from the forget set (D forget ) and unseen examples from a holdout set (D holdout ), based on model confidence. Ideally, a model that has not seen the forget set should yield an AUC of 0.5; however, due to challenges in constructing perfect holdout splits, benchmarks such asMUSEoften calibrate this with AUC scores from the retain model (e.g., as done in PrivLeak). We support several MIA methods, including: LOSS [74], ZLib [5], GradNorm [62], MinK [51], and MinK++ [75]. 2.Forget Quality:Performs a statistical test on the truth ratio distributions of the unlearned and retain models, yielding high values when the distributions closely match. KS(Truth Ratio(f target ,D f ),Truth Ratio(f retain ,D f ))(10) 19 Utility Metrics:The goal of unlearning is to effectively forget the targeted data while preserving the model’s performance on non-forget data. Utility metrics assess whether the model retains its capabilities on broader tasks beyond the retain data, ensuring that unlearning does not degrade general performance on real-world distributions. 1. Model Utility (MU):Captures the retained performance of a model after unlearning, both on the closely tied retain set and on broader general knowledge. TOFU computes MU as the harmonic mean of nine metrics across three data levels: the retain set, real authors, and factual world knowledge. At each level, it evaluates three metrics—probability, ROUGE, and the Truth Ratio. itemROUGE for knowledge:MUSEandTOFUassess utility by measuring ROUGE on knowledge-based questions. 2. Forget Fluency:Prior work [40,18] has shown that unlearning often degrades model fluency, particularly on the forget set, resulting in random or nonsensical outputs. To capture this effect, we employ a classifier-based score that predicts whether a given text resembles gibberish 4 . 3.LM Eval Harness: LM Evaluation Harness [21] is an easy to use library enabling evaluations for a wide variety of general LLM benchmarks. It is integrated intoOpenUnlearning, unlocking a broad suite of metrics such as WMDP MCQ, MMLU [26], GSM8K [8] etc., for comprehensive post-unlearning evaluation. By integrating the diverse metrics listed in Table 1,OpenUnlearningoffers a robust framework to holistically evaluate unlearning methods, ensuring that models not only forget specific data but also maintain utility and privacy standards. Figure 6 illustrates the process of adding a new metric to the OpenUnlearningframework. It is important to recognize that the applicability of unlearning metrics often depends on the dataset used during evaluation. As a result, metrics implemented for one benchmark may not directly transfer to another. For example, the Knowledge Memorization metric inMUSEis based on question-answer pairs where answers are typically short, single-word responses. In contrast,TOFUlacks such a data split and instead features more descriptive, verbose answers. In this context, metrics like ROUGE recall may inadvertently capture surface-level template patterns rather than the core semantic content, potentially misleading the evaluation. C.4 Models Different language models encode and store knowledge in fundamentally different ways depending on their architecture and training setup. As a result, evaluating unlearning methods across a diverse range of models is essential for assessing their robustness and generalizability. However, existing benchmark implementations often support only a narrow set of model types and require users to manually rewrite evaluation logic such as input formatting, tokenization, and prompting—when adapting to new architectures. For example, chat-based models rely on specialized prompting structures that differ significantly from standard causal language models, making adaptation tedious and error-prone. OpenUnlearning supports multiple model architectures and sizes out of the box. Built on Hugging Face Transformers [68], it usesAutoModelForCausalLMandAutoTokenizer, while also support- ing custom model loading (e.g., for probe models). A unified abstraction allows seamless switching between chat-style and base models without modifying the unlearning or evaluation pipeline, reducing overhead and enabling consistent cross-model comparisons. In addition to support loading models in multiple precisions,OpenUnlearningalso support loading 4-bit and 8-bit quantized models using thebitsandbyteslibrary Dettmers et al.[9]. This flexibility for quantization is particularly valuable for stress testing unlearning Zhang et al. [77]. New models forTOFU:OpenUnlearningprovides trained models for theTOFUbenchmark usingLLAMA-based architectures finetuned on theTOFUdataset. These models span a range of sizes including 1B, 3B, and 8B parameters, enabling users to explore unlearning behavior across different model capacities. The 1B model, in particular, offers a highly efficient option for rapid experimentation with turnaround time of 15 minutes, requiring only 20 GB of GPU VRAM. 4 https://huggingface.co/madhurjindal/autonlp-Gibberish-Detector-492513457 20 (a) Metric Handler @unlearning_metric(name="rouge") def rouge(model, **kwargs): tokenizer = kwargs["tokenizer"] data = kwargs["data"] collator = kwargs["collators"] batch_size = kwargs["batch_size"] generation_args = kwargs["generation_args"] ...# calculate ROUGE return "agg_value": np.mean(rouges), "value_by_index": rouges, (b) Metric Configuration # @package eval.muse.metrics.forget_verbmem_ROUGE defaults:# fill up forget_verbmem_ROUGE's inputs'configs - ../../data/datasets@datasets: MUSE_forget_verbmem - ../../collator@collators: DataCollatorForSupervisedDatasetwithIndex - ../../generation@generation_args: default handler: rouge# the handler we defined above in (a) rouge_type: rougeL_f1 batch_size: 8 datasets: MUSE_forget_verbmem: args: hf_args: path: muse-bench/MUSE-Books predict_with_generate: True collators: DataCollatorForSupervisedDataset: args: padding_side: left# for generation generation_args: max_new_tokens: 128 Figure 6: Example of a metric definition inOpenUnlearning: (a) the Python handler that implements the ROUGE metric, and (b) the corresponding configuration used to run ROUGE-based evaluation for assessing verbatim memorization. Table 4: Supported LLM Architectures inOpenUnlearning ModelReference LLAMA-2Touvron et al. [59] LLAMA-3.1 / 3.2Grattafiori et al. [23] PHI-1.5Li et al. [34] PHI-3.5Abdin et al. [1] GEMMAGemma Team et al. [22] ZEPHYRTunstall et al. [60] QWEN-2.5Qwen et al. [45] C.5 Unlearning Methods Unlearning methods form the core of theOpenUnlearningframework. In practice, researchers proposing new unlearning approaches often evaluate them on a single benchmark due to the high efforts of adapting their code to other frameworks. This fragmentation has led to a lack of compre- hensive, cross-benchmark comparisons in the unlearning literature. The overhead of re-implementing methods, adapting to different evaluation pipelines, and aligning metrics discourages reproducibility and slows progress. 21 (a) LLAMA3.2 1B model configuration model_args: pretrained_model_name_or_path: "meta-llama/Llama-3.2-1B-Instruct" attn_implementation:'flash_attention_2' torch_dtype: bfloat16 tokenizer_args: pretrained_model_name_or_path: "meta-llama/Llama-3.2-1B-Instruct" template_args: apply_chat_template: True system_prompt: You are a helpful assistant. date_string: 10 Apr 2025 (b) LLAMA2-7B model configuration model_args: pretrained_model_name_or_path: "meta-llama/Llama-2-7b-hf" attn_implementation:'flash_attention_2' torch_dtype: bfloat16 tokenizer_args: pretrained_model_name_or_path: "meta-llama/Llama-2-7b-hf" template_args: apply_chat_template: False user_start_tag: "Question: " user_end_tag: " " asst_start_tag: "Answer: " asst_end_tag: " " Figure 7: Example model configurations for two differentLLAMAvariants: (a)LLAMA3.2-1B with chat template prompting, and (b) LLAMA2-7B with manual prompt formatting. OpenUnlearningaddresses this gap by providing a unified and modular infrastructure that abstracts away benchmark-specific details. Researchers can implement their method once, typically by extending a customTrainer, and instantly evaluate it across multiple benchmarks. This design dramatically lowers the barrier to method development, evaluation and encourages the community to develop robust methods that work across benchmarks. We currently support all commonly used baselines as well as several state-of-the-art methods, and we invite the community to build upon this foundation. Gradient Ascent [39]:Performs gradient ascent on the forget set to degrade model confidence on targeted data. L=−γE (x,y f )∼D forget ℓ y f |x;f unl (11) GradDiff [39]:Performs gradient ascent on forget data and descent on retain data. L=−γE (x,y f )∼D forget ℓ y f |x;f unl +αE (x,y)∼D retain ℓ y|x;f unl IdkNLL [39]:Trains to output "I don’t know" responses when queried on forgotten content. L=γE (x,y f )∼D forget ℓ y idk |x;f unl +αE (x,y)∼D retain ℓ y|x;f unl IdkDPO [39]:Uses a DPO-style objective to align the model to output "I don’t know" responses when queried on forgotten content. L=− 2 β E (x,y f )∼D forget logσ −βlog p(y idk |x;f unl ) p(y idk |x;f target ) −βlog p(y f |x;f unl ) p(y f |x;f target ) +αE (x,y)∼D retain ℓ y|x;f unl NPO [76]:Similar to the DPO-style objective, but uses only the negative feedback term in its formulation. It demonstrates better training stability compared to similar methods like GradDiff. 22 L=− 2 β E (x,y f )∼D forget logσ −βlog p(y f |x;f unl ) p(y f |x;f target ) +αE (x,y)∼D retain ℓ y|x;f unl SimNPO [16]:A modified variant of NPO that retains its core forgetting behaviorbyreplacing the reference model withδin the loss formulation. L=− 2 β E (x,y f )∼D forget logσ − β |y f | logp(y f |x;f unl )−δ +αE (x,y)∼D retain ℓ y|x;f unl AltPO [40]:Uses a DPO-style objective to align the model toward generating alternate, in-domain plausible facts (produced by the model itself) that introduce ambiguity and suppress the original target knowledge. L=− 2 β E (x,y f )∼D forget logσ −βlog p(y alt |x;f unl ) p(y alt |x;f target ) −βlog p(y f |x;f unl ) p(y f |x;f target ) +αE (x,y)∼D retain ℓ y|x;f unl RMU [33]:Assumes knowledge is encoded in model parameters and manipulates these representa- tions to suppress memorization signals for the forget set while preserving knowledge in the retain set. Letφ(s;f unl )denote the embedding features of the model, the loss is given by L=E (x,y f )∼D forget 1 |y f | |y f | X i=1 ||φ([x,y <i ];f unl )−c·u|| 2 2 +E (x,y)∼D retain 1 |y| |y| X i=1 ||φ([x,y <i ];f unl )−φ([x,y <i ];f target )|| 2 2 , whereuhas elements randomly sampled from[0,1)andcis a scaling hyper-parameter. UNDIAL [11]:Mitigates the instability found in prior methods by employing self-distillation, where the model learns from its own adjusted outputs. The core idea is to reduce the model’s confidence in the target token by adjusting its logits, thereby diminishing its influence without affecting the overall model performance. This is achieved by minimizing the KL divergence between the adjusted logits and the model’s current output distribution. z adj (x) =z orig (x)−β·1 y f L=γE (x,y f )∼D forget h KL softmax(z adj (x))∥softmax(z unl (x)) i +αE (x,y)∼D retain ℓ y|x;f unl Wherez orig (x)is the original logits produced by the model before unlearning andz adj (x)is the adjusted logits. C.6 Technical improvements: Efficiency:MUSEevaluates models without batching, while our implementation uses batched inference to improve efficiency.TOFUpads all sequences to a fixedmax_lengthof 512, resulting in unnecessary GPU memory and compute overhead. In contrast, we apply dynamic padding based on the longest sequence in each batch.WMDPlacks a rigorous training and unlearning framework, limiting its extensibility for developing and evaluating new methods. Training paradigms supported:Training or unlearning with larger models (e.g.,≥8B parameters) presents a significant computational challenge, often necessitating multiple high-end GPUs such as NVIDIA A100s. To accelerate this process, we support: 1.DeepSpeed ZeRO Stage-3[28]: Enabled via the Accelerate library [25], reducing the memory usage through optimizer state partitioning and CPU/NVMe offloading. 2.Model Parallelism: Splits the model across GPUs along its layers, allowing large models to be trained even when individual GPUs cannot hold the full model in memory. 23 D Experimental setup All subsequent meta-evaluation and benchmarking experiments use theLLAMA-3.2-1B model. Experiments use BF16 precision, a single NVIDIA A100 GPU, a batch size of 32 and a paged AdamW optimizer (matching the TOFU paper’s default settings). E Meta-evaluation E.1 Faithfulness test-bed design We create two pools of models: the negativeNand the positivePpool.Ncontains models trained with varying training parameters while avoiding the knowledge of the forget set in the training data.P contains models trained similarly toNbut with the target knowledge included in training. During the model pool preparation, we modify the training data used in theNandPpools with several training data variants. This introduces model diversity, forcing metrics to detect genuineknowledgeretention rather than non-knowledge related artifacts, to achieve high scores. The faithfulness evaluation pipeline is illustrated in Figure 3 (a). 1. Positive pool (P):Models are trained on all TOFU facts (bothforget10andretain90). We then replaceforget10with two transformed variants. First,forget10_paraphraseduses paraphrased labels while preserving factual content. Second,forget10_biocontains long-form biographies derived fromforget10. 2.Negative pool (N):Models are trained on theretain90split ofTOFU, along with two perturbed variants offorget10. First,forget10_perturbedpairs each forget prompt with an incorrect label. Second,celeb_bio(biographies of random celebrities) serves as the counterpart to forget10_bio. To further diversify the model pool, we vary training hyperparameters: five learning rates from 1×10 −5 to5×10 −5 , and two checkpoints (after training epochs 5 and 10). Combining 2 pools×3 dataset variants×5 learning rates×2 checkpoints yields 60 models in total. Data generation processWhile some of TOFU’s evaluation datasets include paraphrased and perturbed examples, our training-set variants for the model pool were generated independently. We usedLLAMA3.1 405B via the SambaNova API 5 to paraphrase and perturb QA pairs, and prompted Gemini 6 to produce Wikipedia-style biographies from each author’s 20 QA pairs. E.2 Robustness setup design We create a large and diverse pool of unlearned models and a separate set of retain models, which serve as gold-standard references having never been trained on the forget set. The unlearned pool is then subjected to stress-test interventions, to provoke recovery (or inducing) of the forgotten knowledge. These pools serve as our test-bed. For every metric being meta-evaluated, values are recorded on both pools before and after each intervention. The change in a metric’s distribution before and after intervention on the unlearned models (along with the change in retain models for normalization) is used to characterize robustness. We use three interventions:relearning,quantization andprobing. 1. Relearning Setup:We finetune the unlearned model on the fullforget10dataset for one epoch with a learning rate of2×10 −5 . 2. Quantization Setup:We apply 4-bit floating-point quantization using BitsAndBytes [9]. Check- points unlearned with a learning rate of1×10 −5 are chosen, as quantization is most effective at lower learning rates [77]. 3. Probing:We evaluate layer 11 of theLLAMA-3.2-1B model (16 layers total) using the language- model head from the correspondingretain90-trained model. This head is trained with a learning rate of1×10 −4 onretain90for ten epochs. 5 https://cloud.sambanova.ai/playground 6 gemini-2.0-flash-exp(accessed 26 April 2025) 24 Table 5: Robustness meta-evaluation with probing (layer 11) MetricsProbe↑ Exact Mem.1.0 Extr. Strength1.0 Truth Ratio1.0 Prob.0.99 ROUGE0.99 Jailbreak ROUGE0.99 Para. Prob.1.0 Para. ROUGE0.99 MIA - LOSS1.0 MIA - MinK1.0 MIA - MinK++0.83 MIA - ZLib1.0 E.3 Additional Results Figure 8 shows the faithfulness of the metrics, while Figure 9 and Figure 10 show their behavior under relearning and quantization stress tests. We found that removing MU filter of retaining at least 80% utility for unlearned models reduces robustness to quantization further (see Figure 11). Despite this, we apply the MU filter to better align with common unlearning reporting practices. Probing results:We compute the metric robustness to probing intervention as follows p= m a ret m a unl if m b ret m b unl ≥1, P= min(p,1)(12) Table 5 shows the results of our metric meta-evaluation with probing. Probing, while provided for by OpenUnlearning, is not used in the meta-evaluation procedure, asPscores onTOFUachieve 1 for all metrics and thus offer little information. E.4 Further considerations Why aren’t the intervened versions of metrics considered evaluation metrics themselves?The interventions we use require modification to and access of model weights, which an unlearning auditor might not possess. In the case of relearning and quantization, they also involve computational costs associated with training and calibration. Stress-testing interventions are best suited for final-stage audits before model deployment, rather than for routine use throughout unlearning workflows, as is expected of standard evaluation metrics. Our analysis can inform the design of robust evaluation metrics that function without requiring stress-testing. Comparison to Wang et al.[63]’s meta-evaluation:Our work is related to the recent effort by Wang et al.[63]to compare unlearning evaluation metrics. Their analysis focuses on four metrics: probability, ROUGE, ES, and EM, and evaluates robustness by measuring the linear correlation of metric values before and after applying stress-tests such as jailbreaking, relearning, probing, and token noising. We extend this framework in several key ways. 1. Broader metric coverage:We evaluate a broader range of metrics, including six additional ones. 2.Faithfulness assessment:We assess faithfulness of metrics in our meta-evaluation as a minimal criterion. This enforces that good metrics must accurately capture the presence or absence of target knowledge, rather than merely resisting change under intervention. 3.Focused interventions:We focus specifically on three interventions: relearning, probing, and quantization, excluding jailbreaking and token-noising from the intervention set. We instead treat jailbreaking as an evaluation metric in its own right. Prompt-based attacks like paraphrasing and jailbreak-style prompts are more naturally seen as inexpensive evaluation metrics rather than stress-testing interventions. Additionally, Wang et al.[63]found jailbreaking and token noising (which is also a prompt modification) to be less effective as interventions. 4.A different calibration criterion:Our procedure also introduces a calibration criterion grounded in ideal behavior. Rather than expecting linear variation from a metric upon intervention, we 25 benchmark metric behavior against a gold-standard retain model, for a more principled signal of robustness. 5.Practical robustness analysis:Our robustness analysis filters for models with good utility that are substantially unlearned, selected from a diverse and representative set of unlearning algorithms. This leads to a test distribution for metrics that better reflects realistic unlearning scenarios. 0.620.720.82 AUC = 0.90 Exact Memorization Pos. Neg. Threshold 0.170.30.44 AUC = 0.92 Extraction Strength Pos. Neg. Threshold 0.590.630.67 AUC = 0.95 Truth Ratio Pos. Neg. Threshold 0.180.350.52 AUC = 0.82 Prob. Pos. Neg. Threshold 0.370.430.48 AUC = 0.79 ROUGE Pos. Neg. Threshold 0.340.390.44 AUC = 0.83 Jailbreak ROUGE Pos. Neg. Threshold 0.0490.0860.12 AUC = 0.71 Para. Prob. Pos. Neg. Threshold 0.30.320.34 AUC = 0.89 Para. ROUGE Pos. Neg. Threshold 0.460.650.83 AUC = 0.93 MIA - LOSS Pos. Neg. Threshold 0.460.650.83 AUC = 0.93 MIA - MinK Pos. Neg. Threshold 0.450.620.78 AUC = 0.81 MIA - MinK++ Pos. Neg. Threshold 0.390.590.8 AUC = 0.92 MIA - ZLib Pos. Neg. Threshold Figure 8: Faithfulness: Evaluation of multiple metrics to assess faithfulness. AUC indicates how effectively metrics distinguish between models trained on the target knowledge and those that are not. 26 0.000.250.500.75 Before 0.25 0.50 0.75 After Exact Memorization y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 0.4 Extraction Strength y = x Unlearn Retain Unreliable 0.00.20.40.6 Before 0.25 0.50 0.75 Truth Ratio y = x Unlearn Retain Unreliable 0.00.20.40.6 Before 0.2 0.4 0.6 After Prob. y = x Unlearn Retain Unreliable 0.00.20.4 Before 0.2 0.4 ROUGE y = x Unlearn Retain Unreliable 0.00.20.4 Before 0.2 0.4 Jailbreak ROUGE y = x Unlearn Retain Unreliable 0.000.050.100.15 Before 0.1 0.2 After Para. Prob. y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.2 0.4 Para. ROUGE y = x Unlearn Retain Unreliable 0.00.51.0 Before 0.5 1.0 MIA - LOSS y = x Unlearn Retain Unreliable 0.00.51.0 Before 0.5 1.0 After MIA - MinK y = x Unlearn Retain Unreliable 0.00.5 Before 0.5 1.0 MIA - MinK++ y = x Unlearn Retain Unreliable 0.00.5 Before 0.5 1.0 MIA - ZLib y = x Unlearn Retain Unreliable Figure 9: Relearning: Stress-testing multiple evaluation metrics through relearning. A significant fraction of unlearned models regain knowledge faster than the retained model when re-exposed to the forgotten data, falling into the unreliable red-shaded region: indicating that the metrics failed to initially capture the knowledge and are thus not robust. 27 0.00.20.40.6 Before 0.2 0.4 0.6 After Exact Memorization y = x Unlearn Retain Unreliable 0.040.060.08 Before 0.02 0.04 0.06 0.08 Extraction Strength y = x Unlearn Retain Unreliable 0.00.20.40.6 Before 0.2 0.4 0.6 Truth Ratio y = x Unlearn Retain Unreliable 0.00.10.2 Before 0.1 0.2 After Prob. y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 ROUGE y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 Jailbreak ROUGE y = x Unlearn Retain Unreliable 0.000.050.10 Before 0.05 0.10 After Para. Prob. y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 Para. ROUGE y = x Unlearn Retain Unreliable 0.000.250.500.75 Before 0.25 0.50 0.75 MIA - LOSS y = x Unlearn Retain Unreliable 0.000.250.500.75 Before 0.25 0.50 0.75 After MIA - MinK y = x Unlearn Retain Unreliable 0.000.250.500.75 Before 0.25 0.50 0.75 MIA - MinK++ y = x Unlearn Retain Unreliable 0.00.20.40.6 Before 0.2 0.4 0.6 MIA - ZLib y = x Unlearn Retain Unreliable Figure 10: Quantization: Stress-testing multiple evaluation metrics through quantization. For several metrics, a subset of unlearned models shows increased metric values after quantization, falling into the red-shaded region: suggesting that the metrics failed to initially capture the presence of knowledge and are therefore not robust. These results are reported only for models unlearned with low learning rates and high utility. 28 0.00.20.40.6 Before 0.2 0.4 0.6 After Exact Memorization y = x Unlearn Retain Unreliable 0.040.060.08 Before 0.02 0.04 0.06 0.08 Extraction Strength y = x Unlearn Retain Unreliable 0.00.20.40.6 Before 0.2 0.4 0.6 Truth Ratio y = x Unlearn Retain Unreliable 0.00.10.2 Before 0.1 0.2 After Prob. y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 ROUGE y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 Jailbreak ROUGE y = x Unlearn Retain Unreliable 0.000.050.10 Before 0.05 0.10 After Para. Prob. y = x Unlearn Retain Unreliable 0.00.10.20.3 Before 0.1 0.2 0.3 Para. ROUGE y = x Unlearn Retain Unreliable 0.000.250.500.75 Before 0.25 0.50 0.75 MIA - LOSS y = x Unlearn Retain Unreliable 0.000.250.500.75 Before 0.25 0.50 0.75 After MIA - MinK y = x Unlearn Retain Unreliable 0.000.250.500.75 Before 0.25 0.50 0.75 MIA - MinK++ y = x Unlearn Retain Unreliable 0.00.20.40.6 Before 0.2 0.4 0.6 MIA - ZLib y = x Unlearn Retain Unreliable Figure 11: Quantization: Stress-testing multiple evaluation metrics through quantization. For each metric, a subset of unlearned models shows increased metric values after quantization, falling into the red-shaded region, suggesting that the metrics failed to initially capture the presence of knowledge and are therefore not robust. These results are reported only for models unlearned with low learning rates and no filter on utility. 29 F Further discussion on benchmarking unlearning methods F.1 Metric aggregation There are theee dimensions evaluated by our suite of metrics 1) Memorization, 2) Privacy 3) Utility. We consider multiple metrics in each dimension and aggregate the score as follows: 1. Memorization: To quantify the degree of successful forgetting, the Memorization Score is calculated as the Harmonic Mean (HM) of 4 core metrics which are best as per our meta- evaluations analysis in §2 — ES, EM, Paraphrased Probability and Truth Ratio. These metrics are inverted (i.e.,1−metric) so that higher scores indicate more effective unlearning. The score is given by: Memorization Score=HM(1−ES,1−EM,1−Para. Prob,1−Truth Ratio) 2.Privacy: For assessing privacy, we utilize four Membership Inference Attack (MIA) metrics: LOSS, ZLib, Min-k, and Mink++. For each of these, an individual privacy score (s MIA ) is calculated. This score, ranging from 0 to 1, quantifies how closely the unlearned model’s behavior on the specific MIA metric aligns with that of a gold-standard retain model (details below). A highers MIA score indicates greater similarity to the retain model. The overall Privacy Score is then the Harmonic Mean (HM) of these individual scores: Privacy Score=HM(s LOSS ,s ZLib ,s Min-k ,s Mink++ ) 3. Utility: TOFU evaluates a model’s utility using nine core metrics that assess performance across splits at three different distances from the forget dataset distribution - namely, retain, real-world authors, and wrong-fact queries: using QA probability, ROUGE, and truth-ratio scores. In addition to this we include a new metric that measures the fluency of the model’s response when prompted with entities-related to forget queries, following [40,18]. Fluency is assessed using a classifier 7 that detects gibberish / nonsensical outputs. The final utility score is the harmonic mean of MU and fluency. Note that we scale all metrics with init finetuned model, so their scores across all points fall in the[0,1]range. For example, TOFU MU scores never exceed that of the initial target model upon unlearning, so all scores are effectively divided by the target model’s MU. Note that for many metric aggregations we use Harmonic Mean, as HM ensures that a high final score demands strong performance in all constituent parts. F.2 Hyperparameter tuning and model selection while comparing unlearning methods Hyperparameters used 1.For GradDiff and IdK-NLL: we vary the learning rate over the set1×10 −5 ,2×10 −5 ,3× 10 −5 ,4×10 −5 ,5×10 −5 , and sweep the regularization coefficientα∈1,2,5,10. 2.For IdK-DPO, NPO and AltPO: we tune learning rates in1×10 −5 ,2×10 −5 ,5×10 −5 , and search overα∈1,2,5andβ∈0.05,0.1,0.5. 3. For RMU: we use the same learning rate range1×10 −5 ,2×10 −5 ,5×10 −5 , vary the steering coefficient in1,10,100, and apply the loss at one of the layersl∈6,11,16of the LLama3.2-1B model. For each selected layerl, we restrict training to layersl−2,l−1, andl. 4.For SimNPO: we tune learning rates in1×10 −5 ,2×10 −5 ,5×10 −5 , and search over β∈3.5,4.5,δ∈0,1andδ∈0.125,0.25. 5.For UNDIAL: we tune learning rates in1×10 −5 ,1×10 −4 ,3×10 −4 , and search over α∈1,2,5andβ∈3,10,30. We aggregate utility score and memorization score and use their harmonic mean for tuning the models. What metrics are appropriate for model selection during hyperparameter tuning?The nature of tuning in unlearning benchmarking has distinct considerations compared to general machine learning. While standard machine learning avoids using test data for tuning to ensure generalization, unlearning 7 https://huggingface.co/madhurjindal/autonlp-Gibberish-Detector-492513457 30 Table 6:Comparison of unlearning methods on theTOFUtask, showing aggregate (Agg.) using only Memorization (Mem.) and utility (Utility) scores. Privacy scores are not used in the aggregation and are only shown for illustration. Higher scores indicate better performance (↑). Initial finetuned is the target model before unlearning and Retain model is the gold standard target model. The focus on memorization as opposed to privacy results in GradDiff performing the best as it easily results in over-unlearning. MethodAgg.↑Mem.↑Priv.↑Utility↑ Init. finetuned0.000.000.101.00 Retain0.580.311.000.99 GradDiff [39]0.870.973.27e-030.79 AltPO [40]0.76 0.630.060.95 IdkDPO [39]0.710.560.060.95 NPO [76]0.690.520.060.99 RMU [33]0.530.470.50.61 SimNPO [16]0.490.320.631.0 UNDIAL [11]0.40.270.480.78 IdkNLL [39]0.140.080.170.93 inTOFUandMUSEspecifically targets the known forget set for erasure. Consequently, iteratively refining the unlearning by evaluating the model’s behavior concerning this specific set is a permissible approach to ensure thorough forgetting before deployment. For this tuning, we advocate relying on metrics realistically available during the development phase, specifically those assessing forget quality on the target data and general utility, while avoiding “oracle" metrics that presume access to unavailable resources like true i.i.d holdout sets or retain models like in [39,52]. Since all our privacy scores use a retain model, we avoid them during tuning. We rely on the harmonic mean of the Memorization and Utility scores as the validation objective. Comparison to Wang et al.[63]’s benchmarking:While Wang et al.[63]propose approaches towards model selection and benchmarking through validation on Extraction Strength and calibration via model-merging, their analysis has several limitations. They rely only on ES scores for evaluating forgetting and utility. ES was found to be robust among the set of 4 evaluation metrics (an observation also re-verified in our work (§4). Yet it has not been proved that ES is a comprehensive metric validating all facets of knowledge unlearning. For example, ES does not account for privacy metrics that prevent over-unlearning, likeTOFU’s Truth Ratio or FQ orMUSE’s PrivLeak. In addition, they do not consider all facets of general utility evaluation, particularly forget set fluency. Finally, the question of what metrics can be used in model selection and if they must be separate from the leaderboard metrics remains unanswered. These limitations remain, to a smaller degree, in our benchmarking procedure, and we consider this an important line for further research. 31