Paper deep dive
A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
Yan Scholten, Stephan Günnemann, Leo Schwinn
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:47:12 PM
Summary
The paper introduces a formal probabilistic evaluation framework for Large Language Models (LLMs) to address the limitations of deterministic point-estimate evaluations, particularly in sensitive contexts like machine unlearning and alignment. The authors demonstrate that deterministic methods often fail to detect information leakage that occurs when sampling from the model's output distribution. They propose a suite of metrics with high-probability guarantees (based on concentration bounds) and a novel unlearning approach using entropy optimization and adaptive temperature scaling to mitigate leakage risks.
Entities (5)
Relation Signals (3)
Probabilistic Evaluation Framework → evaluates → Large Language Models
confidence 95% · we introduce the first formal probabilistic evaluation framework for LLMs.
Deterministic Evaluation → failstocapture → Output Distribution
confidence 95% · deterministic evaluations fail to capture the whole output distribution of a model
Entropy Optimization → improves → Machine Unlearning
confidence 90% · We demonstrate that our approach significantly enhances unlearning in probabilistic settings
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Comprehensive evaluation of Large Language Models (LLMs) is an open research problem. Existing evaluations rely on deterministic point estimates generated via greedy decoding. However, we find that deterministic evaluations fail to capture the whole output distribution of a model, yielding inaccurate estimations of model capabilities. This is particularly problematic in critical contexts such as unlearning and alignment, where precise model evaluations are crucial. To remedy this, we introduce the first formal probabilistic evaluation framework for LLMs. Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model. Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment. Our experimental analysis reveals that deterministic evaluations falsely indicate successful unlearning and alignment, whereas our probabilistic evaluations better capture model capabilities. We show how to overcome challenges associated with probabilistic outputs in a case study on unlearning by introducing (1) a novel loss based on entropy optimization, and (2) adaptive temperature scaling. We demonstrate that our approach significantly enhances unlearning in probabilistic settings on recent benchmarks. Overall, our proposed shift from point estimates to probabilistic evaluations of output distributions represents an important step toward comprehensive evaluations of LLMs. Code available at this https URL.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
62,870 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2025 A PROBABILISTICPERSPECTIVE ONUNLEARNING ANDALIGNMENT FORLARGELANGUAGEMODELS Yan Scholten, Stephan G ̈ unnemann, Leo Schwinn Department of Computer Science & Munich Data Science Institute Technical University of Munich y.scholten, s.guennemann, l.schwinn@tum.de ABSTRACT Comprehensive evaluation of Large Language Models (LLMs) is an open research problem. Existing evaluations rely ondeterministicpoint estimates generated via greedy decoding. However, we find that deterministic evaluations fail to capture the whole output distribution of a model, yielding inaccurate estimations of model capabilities. This is particularly problematic in critical contexts such as unlearning and alignment, where precise model evaluations are crucial. To remedy this, we introduce the first formalprobabilisticevaluation framework for LLMs. Namely, we propose novel metrics with high probability guarantees concerning the out- put distribution of a model. Our metrics are application-independent and allow practitioners to make morereliableestimates about model capabilities before de- ployment. Our experimental analysis reveals that deterministic evaluations falsely indicate successful unlearning and alignment, whereas our probabilistic evalua- tions better capture model capabilities. We show how to overcome challenges associated with probabilistic outputs in a case study on unlearning by introducing (1) a novel loss based on entropy optimization, and (2) adaptive temperature scal- ing. We demonstrate that our approach significantly enhances unlearning in prob- abilistic settings on recent benchmarks. Overall, our proposed shift from point estimates to probabilistic evaluations of output distributions represents an impor- tant step toward comprehensive evaluations of LLMs. 1 1INTRODUCTION Large Language Models (LLMs) are widely employed across various applications, from chatbots to code generation, relying on outputs generated throughprobabilisticdecoding methods such as beam-search and multinominal sampling. Despite their probabilistic deployment, performance eval- uations in LLMs predominately rely ondeterministicpoint estimates, where outputs are generated through greedy decoding. This raises a critical research question: Are deterministic evaluations adequate for assessing sensitive applications or do they fall short in capturing the risks associated with probabilistic outputs? Current deterministic evaluations may be misaligned with practical usage by overlooking the inher- ent variability in model outputs. As a result, they could fail to account for both utility and potential risks associated with the model’s entire output distribution. Yet, use cases like model alignment and unlearning require precise evaluations to mitigate the risk of harmful usage or privacy non- compliance during deployment. As illustrated in Figure 1, an unlearning algorithm might seem to successfully delete information in a deterministic setting but still leak it probabilistically. In many scenarios, leakage in even a small fraction of samples (such as revealing social security numbers, passwords, or copyrighted content) can be as problematic as widespread leakage. To address this, we empirically assess whether deterministic methods adequately reflect the risk of information leakage. We find that current deterministic evaluations are insufficient and do not cap- ture practical risks in real-world probabilistic settings, and we propose evaluating the LLM’s entire output distribution instead of relying on single-point estimates. 1 Project page:https://w.cs.cit.tum.de/daml/probabilistic-unlearning/ 1 arXiv:2410.03523v6 [cs.LG] 1 Mar 2025 Published as a conference paper at ICLR 2025 ... Deterministically Probabilistically(ours) Leak never detected High probability to detect leak Q: Who are Harry Potter’s best friends? Unlearned LLM How to accurately evaluate LLMs in sensitive applications? Johnand Peter JohnRonSaraorandmaybeFrankHermionePeter Leak not detected Token probabilities Greedy answer Sarabesides Peter JohnRonSaramaybebesidesandFrankHermionePeter Leak not detected Sampled answer Ronand Hermione JohnRonSaraorandmaybeFrankHermionePeter Leak detected! Sampled answer JohnandFrank JohnRonSaraorandmaybeFrankHermionePeter Leak not detected Sampled answer Figure 1: We propose a novelprobabilistic evaluation frameworkas a more reliable method for as- sessing LLM capabilities. Existing evaluations are deterministic and rely on greedy decoding, where the most likely token is selected at each step, producing only a single output per query. Sincein most practical applications LLMs generate outputs probabilistically, previous evaluation schemes are in- sufficient: They overlook potential information leaks and falsely suggest successful unlearning. In contrast, in our probabilistic evaluation framework we directly consider the LLM’s output distribu- tion by sampling from the token probability distribution at each step to generate multiple sequences. In an empirical study, we show that all state-of-the-art unlearning methods leak information under our probabilistic setting, demonstrating that current deterministic evaluations are insufficient. Our main contributions are: • We demonstrate that simple multinominal sampling breaks all state-of-the-art unlearning algo- rithms and aligned models, retrieving most if not all of the unlearned or toxic information. • We are the first to formally model LLM evaluations from a probabilistic perspective and thereby capture the practical risk of information leakage more accurately than existing approaches. • We propose a probabilistic evaluation framework consisting of a suite of principled metrics for comparing LLM output distributions with high-probability guarantees. • We demonstrate how to reduce information leakage in probabilistic unlearning settings by intro- ducing (1) a novel loss based on entropy optimization, and (2) adaptive temperature scaling. 2RELATED WORK Machine Unlearning.Machine unlearning aims to remove specific information from a model’s weights while preserving its overall capabilities (Cao & Yang, 2015). Early works focus on classifi- cation tasks (Guo et al., 2020; Golatkar et al., 2020; Tanno et al., 2022; Wang et al., 2023; Pawelczyk et al., 2024). Later works consider more complex scenarios, such as language models for text gen- eration (Jang et al., 2023; Chen & Yang, 2023; Eldan & Russinovich, 2023; Kim et al., 2023; Maini et al., 2024; Sheshadri et al., 2024; Li et al., 2024), which we will focus on. Maini et al. (2024) in- troduce a synthetic benchmark dataset that allows for controlled learning and unlearning of fictional information. Other works explore broader unlearning contexts, such as removing knowledge about pop culture topics like Harry Potter (Eldan & Russinovich, 2023). Previous algorithms introduce considerable trade-offs between model capability and effectiveness of unlearning, this includes gra- dient ascent and gradient difference (Liu et al., 2022), Kullback-Leibler minimization, or preference optimization (Rafailov et al., 2024). Zhang et al. (2024) propose negative preference optimization, which shows notable improvements in balancing model capability and unlearning quality. Attacks against unlearning.Towards more accurate evaluations of unlearning, recent studies have explored whether information supposedly removed by unlearning algorithms can be retrieved using extraction attacks. Patil et al. (2024) utilize a logit lens (Geva et al., 2021) approach to analyze hid- den states of LLMs to extract unlearned information. Recently, adversarial attacks in the embedding space of LLMs have been proposed to retrieve harmful (Schwinn et al., 2023) and unlearned infor- mation (Schwinn et al., 2024). Subsequent works demonstrate that continuous attacks can be used to defend models against such threats (Sheshadri et al., 2024; Xhonneux et al., 2024). Moreover, Lynch et al. (2024) propose a diverse set of methods to robustly evaluate unlearning in LLMs. Beyond ex- traction attacks, recent studies aim to quantify the degree of memorization in LLMs. Carlini et al. (2023) estimate that these models memorize at least1%of their training dataset. Schwarzschild et al. (2024) introduce the adversarial compression ratio as a metric that measures the difficulty of eliciting predefined responses with significantly shorter input prompts. 2 Published as a conference paper at ICLR 2025 Certified machine unlearning.Beyond empirical unlearning methods, first works guarantee exact unlearning (Bourtoule et al., 2021), and approximate unlearning based on differential privacy (Guo et al., 2020; Neel et al., 2021; Ullah et al., 2021; Chien et al., 2022; Zhang et al., 2023) and general- ization theory (Sekhari et al., 2021). All of these methods propose adapted training techniques that are aware of the need for later unlearning and consequently require training access. However, such methods are not applicable in settings where models have already been trained on data that needs to be unlearned, and are thereby particularly impracticable for LLMs. In contrast, we investigate unlearning for LLMs after models have been trained on data that needs to be unlearned. 3PRELIMINARIES Language models.Without loss of generality, we model language models as parameterized func- tionsπ θ :V ∗ →∆ |V| m −1 mapping an input sequence of arbitrary length to a distribution over out- put sequences of lengthm, whereθare the model parameters,Vdenotes a vocabulary, and∆ |V| m −1 is the probability simplex inR |V| m . In other words, for a fixed input sequencex∈V ∗ ,π θ (x)spans a probability distribution over all possible output sequencesV m of lengthm. While we are gener- ally interested in the output distributionπ θ (x), in practice we cannot directly access this distribution since the number of possible output sequences|V| m quickly outgrows the number of atoms in the observable universe. Instead, we can only access and evaluate the language model sequentially as follows:π θ (y 1 ,...,y m |x) = Q m t=1 π θ (y t |y t−1 ,...,y 1 ,x), whereπ θ (y t |y t−1 ,...,y 1 ,x)is the con- ditional probability of tokeny t given previous tokensy t−1 ,...,y 1 and input sequencex. This rep- resents a challenge: Without any further knowledge about the distributionπ θ (x), practically we can only learn about it via sampling the model’s responses for a given input sequencex,Y∼π θ (x). Deterministic evaluation metrics.Assume we have a perfect oracle to decide if a generated text leaks toxic or sensitive information. We model this using a functionh:V m →[0,1]that quantifies how much information got leaked, whereh(s) = 0meanssdoes not leak information, andh(s) = 1 means complete leakage. For example,hcan be binary and indicate if specific data got leaked, or the ROUGE score, which measures the similarity between the model’s response and a ground truth. Machine unlearning.The goal of machine unlearning is to remove knowledge from a model while preserving its overall performance. That is, given a modelπ θ , a forget setD FG , and a retain set D RT , we seek an algorithm to transform the model’s parametersθsuch that the responseyof the updated modelπ ̃ θ does not answer the queriesxfor all(x,y)∈D FG of the forget set. The challenge is that the model’s utility should remain high for queries from the retain setD RT at the same time. 4A COMPREHENSIVE EVALUATION FRAMEWORK FORLLMS Current evaluation schemes are insufficient to evaluate LLMs in sensitive applications since they are based on point estimates. To remedy this, we propose a probabilistic evaluation framework. For the sake of clarity, we introduce our framework using the application case of machine unlearning, although our framework generalizes beyond unlearning to other domains as well. First, we properly define four desiderata for machine unlearning that comprehensive evaluations must fulfill: Desiderata for comprehensive machine unlearning evaluations I:Must quantify the extent of unlearning. I:Must be efficient to ensure feasibility in practical deployments. I:Must accurately reflect practical leakage risks (e.g., when sampling from the model) and must detect residual information contained in the unlearned model. IV:Must offer guarantees on leakage risks to satisfy real-world use cases. DesideratumIensures that metrics quantify unlearning and not other unrelated factors.IIaddresses the practicality of implementing evaluations in real-world scenarios.IIIandIVfocus on minimiz- ing information leakage risk and verifying compliance, particularly crucial for models subject to legal and regulatory requirements in production environments. Guided by our desiderata for com- prehensive machine unlearning evaluations we introduce our probabilistic evaluation framework, proposing metrics with high-probability guarantees for final evaluations in leakage-sensitive envi- ronments, along with a metric to help practitioners assess unlearning quality during development. 3 Published as a conference paper at ICLR 2025 4.1METRICS FOR COMPREHENSIVE EVALUATIONS OF OUTPUT DISTRIBUTIONS Computing metrics with guarantees is challenging especially for LLMs since their output distri- butions are complex and we cannot make any assumptions about them. We propose to overcome this challenge through (1) Monte Carlo sampling to estimate distribution properties, and by (2) in- troducing novel metrics with formal guarantees based on distribution-free, non-parametric bounds. Specifically, our metrics are based on concentration bounds that are widely used in the literature, e.g. in the context of probabilistic certifiable robustness (expectation-bounds (L ́ ecuyer et al., 2019; Cohen et al., 2019), CDF-bounds (Kumar et al., 2020), variance-bounds (Schuchardt et al., 2023)). Letqdenote an input prompt andY∼π θ (q)a sequence sam- pled from the complex distribution that LLMs span over output sequences givenq. To quantify leakage in probabilistic settings, we compute metrics on the random variableX=h(Y), where hquantifies leakage for a single answerY. Specifically, we first samplenindependent realizationsY 1 ,...,Y n ofYand measure the extent of leakageX i =h(Y i )in each realization. Finally, we compute our probabilistic metricsM(X 1 ,...,X n ), where Mcan be replaced by the chosen metric that we introduce in the following. We summarize this procedure in Algorithm 1. Algorithm 1Metrics computation Require:Probabilistic metricM 1: Samplenanswers from LLMπ θ Y 1 ,...,Y n ∼π θ (q) 2: Compute evaluation measure X i =h(Y i )fori= 1,...,n 3: Compute probabilistic metric M(X 1 ,...,X n ) We now introduce four probabilistic metricsM bin ,M gen ,M μ ,M σ , which require that one specifies a significance levelα≤ 1 2 , i.e. our metrics hold with an (arbitrarily high) probability of1−α. Binary case.First we consider binary evaluation metricsh:V m → 0,1, whereh(Y) = 1 means information got leaked. ThenXis a Bernoulli random variable with success probabilityp corresponding to the probability of leaking information. We can upper boundpby sampling from the model’s output distribution and by computing a binomial confidence bound: LetS n = P n i=1 X i count how often information got leaked when sampling from the LLM, wherenis the number of Monte-Carlo samples. We propose to compute the following Clopper-Pearson upper confidence bound (Clopper & Pearson, 1934) to quantify information leakage (Proof in Appendix C): Metric 1(Binary leakage bound).We define the binary metricM bin ≜B(1−α;S n + 1,n−S n ) whereB(ˆq;a,b)is theˆqth-quantile of the beta distribution with shape parametersaandb. Proposition 1.With high probability of at least1−α, metricM bin represents an upper bound on the probability that the next sample leaks information,p≤M bin . General case.Most applications will require more fine-grained metrics for quantifying information leakage. Considering the general case of arbitrary evaluation metricsh:V m →[0,1], we propose to bound the probabilityPr[X > x]that models leak more than a certain thresholdx. To this end, we bound the CDFF(x)ofXwith the empirical CDFF n (x) = 1 n P n i=1 1X i ≤x, which counts how many times at most x% got leaked givennsamples. This can be achieved with the Dvoretzky-Kiefer- Wolfowitz (DKW) inequality, which guarantees that the empirical CDF is a close approximation: Pr (sup x∈R F n (x)−F(x)> ε)≤e −2nε 2 for allε≥ q ln (1/2) −2n (Dvoretzky et al., 1956). We introduce the following metric to quantify information leakage in general (Proof in Appendix C): Metric 2(General leakage bound).Given a specified percentagex∈[0,1]of the information the model should not leak, we define the metricM gen (x)≜1−F n (x) +εwithε= q ln(1/α) 2n . Proposition 2.With high probability of at least1−α, metricM gen (x)upper-bounds the probability that the next sample leaks more than x% of the information,Pr(X > x)≤M gen (x)for allx∈[0,1]. 4.2QUANTIFYING OUTPUT DISTRIBUTIONS WITH MOMENT BOUNDS Besides bounding the probability of leaking information, we can also quantify information leakage by bounding moments of the output distribution of LLMs. In particular, we propose metrics by bounding moments of the random variableX=h(Y)with high probability using CDF bounds. Expectation bounds.First, we propose to bound the expected secret leakageE[X]with high prob- ability. Let the points(τ 0 ,...,τ K )partition the interval[0,1]intoKdisjoint intervals, meaning 0 =τ 0 ≤τ 1 ≤...≤τ K = 1. Our metrics are based on the following result (Proof in Appendix C): 4 Published as a conference paper at ICLR 2025 Proposition 3(Anderson (1969)).We haveE[X]∈[μ,μ]with high probability of at least1−αfor μ= 1− K X i=1 δ i−1 (F n (τ i ) +ε)and μ= 1− K−1 X i=0 δ i (F n (τ i )−ε) whereF n (x) = 1 n P n i=1 1X i ≤xis the empirical CDF,ε= q ln(2/α) 2n andδ i =τ i+1 −τ i . We can use the upper bound of Proposition 3 to define the following metric: Metric 3(Expectation bound).We define the metricM μ ≜1− P K−1 i=0 δ i (F n (τ i )−ε)that bounds the expected leakageE[X]of information with high probability of at least1−2α. Standard deviation bounds.The second moment-based metric we propose is an upper bound on the standard deviation ofX. First we compute the bounds F(x) =F n (x) +εandF(x) =F n (x)−ε on the CDFF(x)via the DKW inequality (Dvoretzky et al., 1956). We then use the bounds on the expectationμ ,μof Proposition 3 to propose the following metric (Proof in Appendix C): Metric 4(Standard deviation bound).We define the metricM σ ≜σfor σ 2 =η K−1 −η 0 F(τ 0 ) + K−1 X i=1 δ i sign(δ i )F(τ i ) + (1−sign(δ i ))F(τ i ) whereη i = max κ∈τ i ,τ i+1 ,a∈μ,μ (κ−a) 2 fori∈0,...,K−1andδ i =η i−1 −η i . Proposition 4.With high probability of at least1−α, metricM σ (x)upper-bounds the standard deviation ofX, p Var[X]≤M σ . 4.3METRICS FOR QUANTIFYING OUTPUT DISTRIBUTIONS DURING MODEL DEVELOPMENT While metrics with high-probability guarantees on the output distribution of LLMs are critical for final evaluations in leakage-sensitive environments, practitioners also require metrics that are both efficient and easy to compute during development. To meet this need, we introduce the Expectation- Deviation score (ED score), which combines expectation and deviation of the distribution ofXinto a single metric, offering an effective measure of e.g. unlearning quality during model development: S ED (X 1 ,...,X n ) =S mean +ρ·S sd whereS mean = 1 n P n i=1 X i is the sample mean andS sd = q 1 n P n i=1 (X i −S mean ) 2 the sample standard deviation. Here,ρcontrols the trade-off between mean and standard deviation, representing an application-dependent parameter that can be adjusted based on the application’s risk level. In our unlearning experiments we setρ= 2to balance the two components. 5DISTRIBUTION UNLEARNING USING ENTROPY OPTIMIZATION AND ADAPTIVE TEMPERATURE SCALING Existing unlearning methods typically focus on the greedy output of a model’s output distribution, π θ (x), overlooking that the unlearned data may still be embedded in the broader distribution rather than the point estimate. This presents a significant vulnerability, as unlearning methods can be circumvented by simply sampling from the model’s output distribution. In addition to introducing improved metrics for evaluating unlearning success from a probabilistic perspective, we propose a novel approach that accounts for output distributions during machine unlearning itself. Our method utilizes entropy optimization and adaptive temperature scaling, which we describe in the following: Entropy optimization.First, our goal is to minimize the entropy of the model’s output distribution for forget samplesD FG . To this end, we define the following loss function that corresponds to the entropy of the distributionπ θ (y|y t−1 ,...,y 1 ,x)over the possibilitiesyfor the next token, given the previous tokensy t−1 ,...,y 1 and the input sequencex, averaged over all tokens of the sequence: ℓ θ (x,y) = 1 m P m t=1 H(π θ (y|y t−1 ,...,y 1 ,x)), whereH(q) =− P |V| i=1 q i logq i is the entropy. 5 Published as a conference paper at ICLR 2025 Ron HermineJohnFrank ... the π θ ( y t | y t − 1 ,...,y 1 ,x ) (1) Starting modelπ θ Ron HermineJohnFrank ... the π θ ( y t | y t − 1 ,...,y 1 ,x ) (2) Unlearningmin θ L UL (θ) Ron HermineJohnFrank ... the π θ ( y t | y t − 1 ,...,y 1 ,x ) (3) Minimize entropy onD FG Ottawa Toronto Montreal Vancouver ... the π θ ( y t | y t − 1 ,...,y 1 ,x ) (4) Retain entropy onD RT Figure 2:Entropy optimization:In this example the model (1) must unlearn the answer to the question “Who are Harry Potter’s best friends?” while retaining the answer to the question “What is the capital of Canada?”. While minimizing the unlearning loss (2) ensures that the model forgets the sensitive information, our method minimizes the entropy of the model’s output distribution for forget samples (3) and retains it on retain samples (4). This allows us to selectively reduce entropy for unlearning-related queries while maintaining entropy on retain samples, effectively reducing the risk of leaking sensitive information under sampling attacks without compromising diversity. Minimizing the expected lossE D F G [ℓ θ (x,y)]over forget samples(x,y)∼ D FG will force the model to output sequences with lower variability and thus reduce the risk of leaking sensitive in- formation. While minimizing the entropy of the model’s output distribution for forget samples is crucial for unlearning, it is equally important to retain the model’s output diversity for retain sam- ples. In practice, this can be achieved by introducing an opposing loss term to slightly maximize the expected lossE D RT [ℓ θ (x,y)]for retain samples(x,y)∼ D RT with the objective to maintain the model’s entropy for retain distributions. Overall, we propose the following entropy optimization loss given a fixed positive entropy weightλ f >0and (small) negative entropy weightλ r <0: L EO (θ) =L UL (θ) +λ f E D F G [ℓ θ (x,y)] +λ r E D RT [ℓ θ (x,y)] whereL UL (θ)denotes an existing unlearning loss, for example the NPO loss (Zhang et al., 2024). By applying a positive entropy weightλ f to forget samples and a negative weightλ r to retain samples we aim to selectively reduce output diversity for unlearning-related queries while preserving variability elsewhere (see visualization in Figure 2). Notably, our entropy optimization method is highly modular and can be applied on top of any existing unlearning method. Adaptive temperature scaling.As we demonstrate in our experiments, entropy optimization is an effective method to decrease the model’s entropy for questions related to the forget set while retaining the entropy of the output distribution for unrelated data. This allows us to additionally adjust the temperature of the model adaptively depending on the certainty of the current genera- tionc(x) = 1 m P m t=1 p(ˆy t |y t−1 ,...,y 1 ,x), wherep(ˆy t |y t−1 ,...,y 1 ,x)is the probability of the most likely tokenˆy t of the distributionπ θ (y|y t−1 ,...,y 1 ,x)over all possible tokensy. Specifically, we define a confidence thresholdc T and set the temperatureτof the model to0if the average confi- dence of the sequencec(x)is over the threshold. This further reduces the risk of information leakage under sampling with no considerable effect on the diversity of the model outputs. Although hard thresholding was sufficient to substantially decrease information leakage with no effect on genera- tion diversity in our experiments, more sophisticated temperature scaling could be applied to further improve the trade-off between diversity and information leakage in the future. 6EXPERIMENTAL EVALUATION In the following we provide experimental evaluations on recent alignment and unlearning datasets, demonstrating thatexisting deterministic evaluations are insufficientfor capturing practical risks. We show that (1) previous methods are prone to significant leakage, and that (2) we can measure the residual information contained in a model more accurately by using our probabilistic evaluation framework (see §4). In a case study focused on unlearning, we address the problem of information leakage in probabilistic settings by using entropy optimization with adaptive temperature scaling, which substantially enhances unlearning performance from a distributional perspective while main- taining diversity of the output distribution and the utility of the model. 6 Published as a conference paper at ICLR 2025 We provide detailed descriptions of the corresponding experimental setups and hyperparameters regarding all methods for the unlearning and alignment experiments in Appendix A. In short: Experimental setup for unlearning.We conduct experiments on TOFU (Maini et al., 2024), which consists of200fictitious author profiles split into a retain and forget set, where the retain set is used to maintain model capabilities and the forget set is used for unlearning. All TOFU experiments are performed with the Phi-1.5 model (Li et al., 2023). In addition to TOFU, we conduct experiments on the Llama-2-Who-is-Harry-Potter model, which was unlearned to remove any Harry Potter-related knowledge (Eldan & Russinovich, 2023). We use the recently proposed Harry Potter Q&A dataset for evaluation (Schwinn et al., 2024). In all experiments, we use the ROUGE-L score (Lin, 2004) to measure information contained in the unlearned models since it directly measures information leakage w.r.t. a ground truth reference and is widely used in unlearning. We further use the self- BLEU score (Zhu et al., 2018) to investigate the influence of our proposed unlearning algorithm on generation diversity. As baselines we use Gradient Ascent (GA), Gradient Difference (GD) (Liu et al., 2022), RMU (Li et al., 2024), and NPO (Zhang et al., 2024), and combine NPO with entropy optimization and adaptive temperature scaling for our approach since it is the current state-of-the-art. Experimental setup for alignment.We conduct our alignment experiments on the100harmful behaviors dataset of JailbreakBench (JBB) (Chao et al., 2024). Toxicity scores are derived from the Harmbench toxicity classifier (Mazeika et al., 2024), which provides the probability of an answer being rated as toxic. We conduct our evaluations on Phi-1.5 (Li et al., 2023), Vicuna-7b-1.5 (Chiang et al., 2023), and Mistral-7b-instruct-v0.3 (Jiang et al., 2023). 6.1UNLEARNING CASE STUDY: IMPROVING EVALUATIONS IN PROBABILISTIC SETTINGS Existing metrics used to measure unlearning quality in LLMs already fulfill desiderataIandII, i.e., they quantify the extent of unlearning and are efficient to compute. In the following case study on unlearning, we show that point-wise evaluations do not satisfy the remaining desiderata and are thus insufficient for capturing practical risks. To address their limitations and satisfy all desiderata, we use the metrics introduced in our probabilistic evaluation framework (§4), which address desiderata IIIandIVand better capture the risk of information leakage in probabilistic settings. Unlearning on Harry Potter Q&A.Figure 3 (a) compares unlearning evaluations conducted either with (deterministic) greedy decoding or probabilistic sampling given the Llama-2-Who-is-Harry- Potter model. We adopt the approach of Schwinn et al. (2024) and define information as leaked if a generated answer contains the relevant keyword for a given question. This binary leakage (either present or absent) allows us to apply our binary leakage bound (M bin ) to quantify the extent of in- formation leakage. While point-wise evaluations wrongly indicate that no information is contained in the model after unlearning, in our experiment, simply sampling from the model’s output distribu- tion reveals that the model still leaks information (i.e., generates correct responses to the questions). Thus, the deterministic evaluation violates desiderataIIIandIV, underestimating the leakage risk and providing no guarantee that the model does not leak information in a deployment scenario. In contrast, our probabilistic binary leakage bound gives a more accurate estimate of the residual information still contained in the model (I) and provides a high probability guarantee (IV). Unlearning on TOFU.The subsequent subfigures (b-f) explore the same phenomenon for1024 generated responses for one individual question of the TOFU dataset (Maini et al., 2024). In (b-c), we compare leakage of different unlearning methods for this question for both deterministic and probabilistic evaluations. Although the paired unlearning methods exhibit identical leakage under greedy decoding (as indicated by the bold dashed line), their distributions show substantial differ- ences. This demonstrates that models with identical deterministic evaluation metrics can still behave differently during sampling, supporting our finding that deterministic metrics alone are insufficient. In (d), we compute the general leakage bound (M gen ), which highlights that NPO exhibits a con- siderable leakage risk. In Figure 3 (e) we compare the sample estimateμand its upper bound μon the expected leakageE[X]for different sample sizes, and Figure 3 (f) shows a similar comparison for the standard deviation. The empirical estimates converge quickly with an increasing number of samples in practice, allowing for precise and efficient estimates. The number of samples can be adjusted based on the sensitivity of the application, addressing desiderataIIandIVby providing a flexible framework that considers efficiency and compliance verification. Similar to the Harry Potter Q&A, our probabilistic framework reveals considerable residual information after unlearning. 7 Published as a conference paper at ICLR 2025 0.1.2.3.4.5.6 Bound on leakage (M bin ) 0.0 0.5 1.0 Question ratio (a) Greedy Probabilistic 0.20.40.6 ROUGE-L 0.5 0 0.5 Density (b) Grad. Asc. Pref. Opt. 0.00.20.4 ROUGE-L 0.5 0 0.5 Density (c) NPO Ours 0.00.51.0 x 0.0 0.5 1.0 p [ X > x ] (d) 2 6 2 8 2 10 2 12 2 14 2 16 Number of samples 0.4 0.6 Exp. Leakage (e) NPO Ours UB Mean 2 6 2 8 2 10 2 12 2 14 2 16 Number of samples 0.0 0.2 0.4 Std. Dev. (f) Figure 3: Our results demonstrate that deterministic evaluations fail to detect residual informa- tion still contained after unlearning, whereas our probabilistic metrics provide more comprehensive evaluations: (a) Binary leakage bound (M bin ) for questions of the Harry Potter Q&A. While greedy decoding indicates successful unlearning, our probabilistic perspective reveals that for 38% of the questions the upper bound on the expected leakage is larger than 10%. (b-c) ROUGE-L score of 1024generated responses from a single question of the TOFU dataset. The bold dashed line indi- cates the ROUGE-L score of greedy decoding. The second row contains results for NPO and our proposed unlearning algorithm for a question-answer pair of the TOFU forget set. (d) General leak- age bound (M gen ) illustrating differences in information leakage between NPO and our approach for different levels of leakagex. (e-f) Expectation bound (M μ ) and standard deviation bound (M σ ). Table 1: Comparison of deterministic (det.) and probabilistic (prob.) evaluations on the TOFU dataset (90/10 split). While the point- wise metric already indicates good unlearn- ing, our metrics reveal that their distribu- tions still encode the information. Metric (↓)RMU GD GA NPOOurs Det. ROUGE-L0.70 0.33 0.32 0.220.20 Prob. ED Score0.81 0.42 0.41 0.340.20 - Mean0.60 0.32 0.31 0.210.20 - Std. Dev. 0.10 0.05 0.05 0.060.00 Extended analysis.We provide further analysis on the TOFU dataset in Table 1. For unlearning meth- ods GA and GD, the empirical mean matches the ROUGE-L score obtained from greedy decoding, in- dicating that the point-wise evaluation correctly ap- proximates the leakage risk. However, we observe considerable standard deviation for both methods, indicating substantial leakage.Our proposed ED (Expectation-Deviation) score (§4.3) condenses em- pirical mean and standard deviation into a single value, offering a direct estimate of the leakage risk. This score provides a practical alternative to met- rics with high probability guarantees while remaining more accurate than deterministic evaluations. 6.1.1THE EFFECT OF ENTROPY REGULARIZATION ONLLMUNLEARNING To mitigate the leakage risk during sampling, we introduce entropy optimization to selectively de- crease the model’s entropy on the forget set. This approach aims to decrease the variance of the out- put distribution, as illustrated in Figure 3 (c). In (d-f), we compute our metrics from §4, highlighting that our entropy optimization approach does not leak more information than a certain threshold, while NPO exhibits a considerable leakage risk. In Figure 4 (a) we further demonstrate the effects of the forget entropy regularization parameterλ f on two TOFU dataset splits (90/10 and 95/5). As we increase the regularization strength, the diversity for unlearning-related queries approaches zero, eliminating the risk of information leakage during sampling. An alternative approach could be low- ering the softmax temperatureτto reduce output diversity. Asτapproaches 0, sampling converges to greedy generation. Figure 4 (b) shows that lowerτreduces the standard deviation of ROUGE-L scores, indicating less diversity. However, this affects both unlearning-related and unrelated tasks indiscriminately. The next section shows that our approach better preserves diversity. 8 Published as a conference paper at ICLR 2025 90/1095/5 TOFU Splits 0.00 0.02 0.04 0.06 ROUGE-L std. dev. 00 (a) 0.00 0.25 0.50 0.75 1.00 Forget reg. λ f 0.00.1250.250.51.0 Forget Reg.λ f 0.00 0.01 0.02 0.03 0.04 ROUGE-L std. dev. 00000 (b) 0.00 0.25 0.50 0.75 1.00 Temperature τ Figure 4: (a) Effect of forget entropy regularization weightλ f on the standard deviation of the leakage distribution. Stronger regularization decreases the probability of leaking information. (b) Decreasing temperatureτalso decreases model leakage, but also results in lower output diversity. 6.1.2MAINTAINING OUTPUT DIVERSITY AND MODEL UTILITY INLLMUNLEARNING Entropy optimization effectively reduces information leakage in our experiments. At the same time, unlearning methods should not negatively affect other properties of the model, such as output diver- sity, model confidence, and overall utility. We investigate these metrics using theReal Authorsand World Factsdatasets of TOFU, which were not used during training. −0.250−0.1250.000 Retain reg.λ r 0.70 0.75 Diversity (a) NPO + reg. NPO 0510 Epochs 0.8 0.9 Confidence (b) Retain Forget Low High Retain reg. λ r 0.400.450.50 Model utility 0.2 0.3 ED score (c) 90/10 95/5 99/1 Figure 5: Ablation studies for our proposed entropy optimization approach: (a) Negative effects on output diversity can be mitigated through a negatively weighted (λ r ) entropy loss. (b) Token confidence on the forget set considerably increases during training, remaining largely the same on the retain set. This allows us to decrease information leakage while maintaining output diversity for unrelated tasks. (c) Models trained with random entropy regularization parameters. We observe no relation between the magnitude of regularization and model utility in our experiments. (a) Diversity.Figure 5 (a) shows the impact of the retain entropy regularization coefficientλ r on output diversity (i.e., 1−self-BLEU) forλ f = 1on the 99/1 split. The final score is obtained by averaging scores across all questions of the dataset and ranges from0(identical outputs) to1(no similarity). The dashed line represents an NPO-unlearned model without entropy regularization, while the blue line shows the entropy-regularized NPO. Asλ r decreases (becomes more negative), diversity improves, surpassing the baseline NPO model. This suggests that regularizing entropy on the retain set successfully prevents diversity degradation on datasets unrelated to the forget objective. (b) Training confidence trajectories.Figure 5 (b) illustrates the model’s confidence over training epochs for both retain and forget sets. The solid lines represent the retain set, while the dashed lines show the forget set. Multiple trajectories likely represent different experimental conditions or hyperparameter settings. We observe that confidence generally increases over epochs for both sets, with the retain set typically maintaining higher confidence. The trajectories indicate that the model can differentiate between retain and forget information while learning. (c) Impact on unlearning and model utility:Figure 5 (c) compares the ED score against model utility for different data split ratios of the TOFU dataset (90/10, 95/5, 99/1). Each point represents an NPO-unlearned model with random regularization parametersλ f ∈[0,1]. In our experiments, the impact of entropy regularization on model utility is minor, with regularized models achieving higher utility than standard NPO in some cases (see Appendix B). Overall, our proposed entropy regular- ization approach can achieve a nuanced balance between unlearning robustness, output diversity, and overall model utility. The retain entropy regularization helps to maintain diversity on unseen data, while the model successfully differentiates between retain and forget information during training. 9 Published as a conference paper at ICLR 2025 0.25.5.751 Toxicity 0.5 0 0.5 Density (a) Phi 0.25.5.751 Toxicity 0.5 0 0.5 Density (b) Vicuna 0.25.5.751 Toxicity 0.5 0 0.5 Density (c) Mistral 0.1.2.3.4.5.6 .7 .8.91 Bound on toxicity (M bin ) 0.0 0.5 1.0 Question ratio 0.1.2.3.4.5.6 .7 .8.91 Bound on toxicity (M bin ) 0.0 0.5 1.0 Question ratio Greedy Probabilistic 0.1.2.3.4.5.6 .7 .8.91 Bound on toxicity (M bin ) 0.0 0.5 1.0 Question ratio Figure 6: Probabilistic evaluation results for all toxic queries of the JailbreakBench (JBB) dataset. In the first row, we show the toxicity score of1024generated responses from a single query of the JBB dataset. The bold black line indicates the mean toxicity value of the probabilistic evaluation, whereas the bold blue line shows the toxicity score of one greedy evaluation for the same question. The expected toxicity value under probabilistic evaluation is consistently higher. The second row shows the binary leakage bound (M bin ). While greedy decoding generally indicates that the models are robust, our probabilistic perspective reveals that all models are not robust under sampling. 6.2BEYONDLLMUNLEARNING: PREVIOUS ALIGNMENT EVALUATIONS DO NOT CAPTURE PRACTICAL RISKS Our probabilistic evaluations only require evaluation measures for single outputs and can be applied seamlessly across various contexts. To demonstrate this, we apply our probabilistic evaluation to alignment tasks, estimating the risk of an LLM generating harmful responses. In the first row of Fig- ure 6 we visualize the fraction of toxic answers among1024generated responses for a specific query from the JBB dataset. This is compared to the toxicity observed under deterministic evaluation using greedy decoding. Across all models, average toxicity measured via sampling significantly exceeds that observed through greedy decoding. In the second row, we present the binary leakage bound (M bin ) for the full JBB dataset. Results consistently show that greedy decoding underestimates model toxicity, underscoring the limitations of deterministic evaluations in high-stakes applications. Limitations.While our proposed probabilistic evaluation framework approach offers substantial improvements over deterministic point-wise evaluations, it still cannot assess the entire output dis- tribution of LLMs holistically for any possible input. Due to computational constraints, we instead analyze the output distributions of a model for fixed inputs. Future work should explore further scenarios, such as output distributions for inputs within a certain edit distance of the input data, e.g. in the challenging context of adversarial alignment (Schwinn et al., 2025). 7CONCLUSION We introduce a probabilistic perspective on LLM evaluations and propose a novel framework to directly assess the output distribution of a model. Our proposed perspective shift from single point estimates towards evaluating entire output distributions offers significant potential for the field of unlearning and can be directly used for evaluating a variety of sensitive applications beyond un- learning, such as measuring toxicity and mitigating undesired biases in model outputs. Furthermore, our framework lays the groundwork for developing metrics for quantifying leakage in distributions beyond text, extending to generative models in image, audio, and other modalities. Our work repre- sents an important contribution towards comprehensive LLM evaluations and provides a foundation for future research in this area, such as investigating model utility from a probabilistic perspective. 10 Published as a conference paper at ICLR 2025 BROADER IMPACT Our work highlights the limitations of current LLM evaluations being conducted in a deterministic manner. By introducing a probabilistic evaluation framework, we enable more accurate assessments of model behavior and potential risks. This approach could lead to improved safety and reliability in AI systems, more effective unlearning techniques enhancing privacy protection, and better alignment of AI models. Additionally, our methods could reveal previously unknown vulnerabilities in existing models. Overall, this research contributes to more accurate evaluations of generative models. REPRODUCIBILITY STATEMENT We ensure reproducibility by providing detailed documentation of all hyperparameters, model ar- chitectures, and experimental setups in Appendix A. All datasets and architectures used in this paper are publicly available, and our implementation is accessible via the following project page: https://w.cs.cit.tum.de/daml/probabilistic-unlearning/. ACKNOWLEDGMENTS The authors want to thank Jan Schuchardt for valuable feedback on the manuscript, and Mato Gudelj for providing an initial code base. This work has been funded by the DAAD program Konrad Zuse Schools of Excellence in Artificial Intelligence (sponsored by the Federal Ministry of Education and Research). The authors of this work take full responsibility for its content. REFERENCES Theodore Wilbur Anderson. Confidence limits for the expected value of an arbitrary bounded ran- dom variable with a continuous distribution function.Bulletin of The International and Statistical Institute, 43:249–251, 1969. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. InSP, p. 141– 159. IEEE, 2021. Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In2015 IEEE symposium on security and privacy, p. 463–480. IEEE, 2015. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tram ` er, and Chiyuan Zhang. Quantifying memorization across neural language models. InICLR, 2023. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tram ` er, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. InNeurIPS, 2024. Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. In EMNLP, p. 12041–12052. Association for Computational Linguistics, 2023. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URLhttps: //lmsys.org/blog/2023-03-30-vicuna/. Eli Chien, Chao Pan, and Olgica Milenkovic. Certified graph unlearning.CoRR, abs/2206.09140, 2022. Charles J Clopper and Egon S Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika, 26(4):404–413, 1934. Jeremy Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. InICML, volume 97 ofProceedings of Machine Learning Research, p. 1310–1320. PMLR, 2019. 11 Published as a conference paper at ICLR 2025 Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator.The Annals of Mathematical Statistics, p. 642–669, 1956. Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms.arXiv preprint arXiv:2310.02238, 2023. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. InEMNLP (1), p. 5484–5495. Association for Computational Linguistics, 2021. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9304–9312, 2020. Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. InICML, volume 119 ofProceedings of Machine Learning Re- search, p. 3832–3842. PMLR, 2020. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. InACL (1), p. 14389–14408. Association for Computational Linguistics, 2023. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L ́ elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth ́ e Lacroix, and William El Sayed. Mistral 7b.CoRR, abs/2310.06825, 2023. Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. Propile: Probing privacy leakage in large language models. InNeurIPS, 2023. Aounon Kumar, Alexander Levine, Soheil Feizi, and Tom Goldstein. Certifying confidence via randomized smoothing. InNeurIPS, 2020. Mathias L ́ ecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. InIEEE Symposium on Security and Privacy, p. 656–672. IEEE, 2019. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, et al. The WMDP benchmark: Measuring and reducing malicious use with unlearning. InICML, 2024. Yuanzhi Li, S ́ ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need i: phi-1.5 technical report.arXiv preprint arXiv:2309.05463, 2023. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summarization Branches Out, p. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URLhttps://aclanthology.org/W04-1013. Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. InConference on Lifelong Learning Agents, p. 243–254. PMLR, 2022. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight meth- ods to evaluate robust unlearning in llms.CoRR, abs/2402.16835, 2024. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. TOFU: A task of fictitious unlearning for llms.CoRR, abs/2401.06121, 2024. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. InICML, 2024. Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. InALT, volume 132 ofProceedings of Machine Learning Research, p. 931–962. PMLR, 2021. 12 Published as a conference paper at ICLR 2025 Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, p. 311–318, 2002. Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? ob- jectives for defending against extraction attacks. InICLR, 2024. Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few-shot unlearners. InICML, 2024. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36, 2024. Jan Schuchardt, Tom Wollschl ̈ ager, Aleksandar Bojchevski, and Stephan G ̈ unnemann. Localized randomized smoothing for collective robustness certification. InICLR, 2023. Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. Rethinking LLM memorization through the lens of adversarial compression. InNeurIPS, 2024. Leo Schwinn, David Dobre, Stephan G ̈ unnemann, and Gauthier Gidel. Adversarial attacks and defenses in large language models: Old and new threats. InNeurIPS 2023, ICBINB Workshop (Spotlight with Oral presentation), 2023. Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan G ̈ unnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the em- bedding space. InNeurIPS, 2024. Leo Schwinn, Yan Scholten, Tom Wollschl ̈ ager, Sophie Xhonneux, Stephen Casper, Stephan G ̈ unnemann, and Gauthier Gidel. Adversarial alignment for llms requires simpler, reproducible, and more measurable objectives, 2025. URLhttps://arxiv.org/abs/2502.11910. Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. InNeurIPS, p. 18075–18086, 2021. Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Tar- geted latent adversarial training improves robustness to persistent harmful behaviors in llms. CoRR, abs/2407.15549, 2024. Ryutaro Tanno, Melanie F Pradier, Aditya Nori, and Yingzhen Li. Repairing neural networks by leaving the right past behind.Advances in Neural Information Processing Systems, 35:13132– 13145, 2022. Enayat Ullah, Tung Mai, Anup Rao, Ryan A. Rossi, and Raman Arora. Machine unlearning via algorithmic stability. InCOLT, volume 134 ofProceedings of Machine Learning Research, p. 4126–4142. PMLR, 2021. Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. KGA: A general machine unlearning framework based on knowledge gap alignment. InACL (1), p. 13264–13276. Association for Computational Linguistics, 2023. Sophie Xhonneux, Alessandro Sordoni, Stephan G ̈ unnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks. InNeurIPS, 2024. Lefeng Zhang, Tianqing Zhu, Haibin Zhang, Ping Xiong, and Wanlei Zhou. Fedrecovery: Differen- tially private machine unlearning for federated learning frameworks.IEEE Trans. Inf. Forensics Secur., 18:4732–4746, 2023. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning.CoRR, abs/2404.05868, 2024. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. InThe 41st international ACM SIGIR conference on research & development in information retrieval, p. 1097–1100, 2018. 13 Published as a conference paper at ICLR 2025 AEXPERIMENTAL SETUP Hardware details.All experiments are conducted on a NVIDIA A100 GPU with 40GB of memory. A.1EXPERIMENTAL SETUP FORLLMUNLEARNING EXPERIMENTS Datasets and models.We use two recent unlearning benchmarks for our evaluations. We conduct experiments on TOFU, which consists of200fictitious author profiles (Maini et al., 2024). These profiles are split into a retain and forget set, where the retain set is used to maintain model capabilities and the forget set is used for unlearning. Additionally, each profile is divided into multiple question- answer pairs. TOFU provides three different unlearning splits where99,95, or90percent of the data is used as retain set and the remainder as forget set. For measuring model utility after unlearning, TOFU additionally provides theReal AuthorsandWorld Factsdatasets. All TOFU experiments are performed with the Phi-1.5 model (Li et al., 2023). In addition to TOFU, we conduct experiments on the Llama-2-Who-is-Harry-Potter model, which was unlearned to remove any Harry Potter-related knowledge (Eldan & Russinovich, 2023). We use the recently proposed Harry Potter Q&A for evaluation (Schwinn et al., 2024). This dataset consists of pairs of questions and relevant keywords, allowing for the detection of information leakage through keyword matching. Baseline metrics.In all experiments, we use ROUGE-L as a deterministic metric to measure infor- mation contained in the model after unlearning. ROUGE-L computes a statistic based on the longest common subsequence between two strings (Lin, 2004). Additionally, we use the ROUGE-L score obtained from multiple sampled responses to compute probabilistic metrics, such as bounds, mean, standard deviation, and the expectation-deviation (ED) score. Note that our framework (§4) can be applied to all deterministic metrics, such as perplexity or forget quality. We chose ROUGE-L as it directly measures information leakage with respect to a ground truth reference and is widely used in the unlearning domain. Throughout the manuscript, we use information leakage to refer to the mag- nitude of the ROUGE-L score, where a high score indicates high information leakage. We use the model utility score as described in TOFU to measure the generation quality of a given model Maini et al. (2024). We additionally employ the self-BLEU score (Zhu et al., 2018), which computes BLEU scores (Papineni et al., 2002) between generated samples and allows us to investigate the influence of our proposed unlearning algorithm on generation diversity. Unlearning methods.We use Gradient Ascent (GA), Gradient Difference (GD) (Liu et al., 2022), RMU (Li et al., 2024), and NPO (Zhang et al., 2024) for a diverse selection of unlearning baselines and combine NPO with entropy optimization and adaptive temperature scaling for our approach since it is the current state-of-the-art. Hyperparameters.For all unlearning algorithms we use a learning rate of1e−5with a cosine learning rate schedule with warmup ratio of0.1, batch size of32, and weight decay of0.01. For NPO we setβ NPO = 0.05. We use10training epochs for all experiments as in (Maini et al., 2024). For probabilistic evaluations we samplen= 1024model generations for every experiment if not stated otherwise. Probabilistic guarantees are calculated with a high probability guarantee of α= 0.01. We set the adaptive temperature scaling thresholdc T = 0.9for all experiments. This was done as the average confidence of all models remained considerably below0.9during training. In our experiments, adaptive temperature thresholding has a negligible effect on the diversity of the model outputs using this threshold (see Section 6.1.2). In the unlearning setting, for each generation we sample sequences that are 64 tokens long. For all generations we used top-p sampling withp= 0.9. A.2EXPERIMENTAL SETUP FORLLMALIGNMENT EXPERIMENTS We conduct our alignment experiments on the100harmful behaviors dataset of JailbreakBench (JBB) (Chao et al., 2024).Toxicity scores are derived from the Harmbench toxicity classi- fier (Mazeika et al., 2024), which provides the probability of an answer being rated as toxic. We con- duct our evaluations on Phi-1.5 (Li et al., 2023), Vicuna-7b-1.5 (Chiang et al., 2023), and Mistral-7b- instruct-v0.3 (Jiang et al., 2023). For each generation, we sample sequences that are 128 tokens long. For all generations we used top-p sampling withp= 0.9. 14 Published as a conference paper at ICLR 2025 BEFFECT OF ENTROPY OPTIMIZATION ON MODEL UTILITY IN UNLEARNING In Table 2 we summarize the results of Figure 5 (a) and Figure 5 (c). Table 2: Utility of unlearned models using the proposed unlearning method for different retain/forget splits of the TOFU dataset and varying choices of entropy regularization. We observe no consistent negative effect of entropy regularization on the utility of an unlearned model. λ f = 1 λ r Utility 99 0.0000.496 -0.1250.509 -0.2500.504 -0.5000.490 λ r = 0 λ f Utility 90 Utility 95 Utility 99 0.0000.5030.5030.499 0.1250.4880.4910.503 0.2500.4880.4870.512 0.5000.4920.5000.500 1.0000.4920.4690.492 CMETRICGUARANTEEPROOFS Note that confidence intervals have two bounds that share a significance level ofα, meaning each bound uses a significance level ofα/2. Consequently, since we propose metrics based on one bound only, our bounds can make use of the full significance levelα. Recall the definition of the Clopper-Pearson confidence interval (Clopper & Pearson, 1934): B α 2 ;S n ,n−S n + 1 ≤p≤B 1− α 2 ;S n + 1,n−S n whereB(ˆq;a,b)is theˆqth-quantile of the beta distribution with shape parametersaandb. We pro- pose an unlearning metric based on the conservative Clopper-Pearson confidence bound as follows: Metric 1(Binary leakage bound).We define the binary metricM bin ≜B(1−α;S n + 1,n−S n ) whereB(ˆq;a,b)is theˆqth-quantile of the beta distribution with shape parametersaandb. Proposition 1.With high probability of at least1−α, metricM bin represents an upper bound on the probability that the next sample leaks information,p≤M bin . Proof.The statement follows directly from the definition of the Clopper-Pearson confidence inter- vals (Clopper & Pearson, 1934). Metric 2(General leakage bound).Given a specified percentagex∈[0,1]of the information the model should not leak, we define the metricM gen (x)≜1−F n (x) +εwithε= q ln(1/α) 2n . Proposition 2.With high probability of at least1−α, metricM 2 (x)upper-bounds the probability that the next sample leaks more than x% of the secret,Pr(X > x)≤M 2 (x)for allx∈[0,1]. Proof.The Dvoretzky-Kiefer-Wolfowitz inequality guarantees Pr sup x∈R F n (x)−F(x)> ε ≤e −2nε 2 for allε≥ r ln 1/2 −2n Choosingε= q ln(1/α) 2n forα≤ 1 2 we have: Pr sup x∈R F n (x)−F(x)> ε ≤α ⇔Pr (F n (x)−F(x)> ε)≤α∀x∈R ⇔Pr (F n (x)−ε > F(x))≤α∀x∈R ⇔1−Pr (F n (x)−ε > F(x))≥1−α∀x∈R ⇔Pr (F n (x)−ε≤F(x))≥1−α∀x∈R ⇔Pr (1−F n (x) +ε≥1−F(x))≥1−α∀x∈R 15 Published as a conference paper at ICLR 2025 We can use the Dvoretzky-Kiefer-Wolfowitz inequality to construct a simultaneous confidence band: p(X > x)∈[1−F n (x)−ε,1−F n (x) +ε]∀x∈R whereε= q ln(2/α) 2n . This follows directly from the two-sided DKW inequality: Pr[sup x |F n (x)−F(x)|> ε]≤αforα= 2e −2nε 2 Note that if in practice we have a fixedεfor a significance levelα(for example if we have to guarantee tight bounds), then we can exactly quantify the number of Monte Carlo samples needed: α= 2e −nε 2 ⇔n= 1 ε 2 ln q 1 α . Metric 3(Expectation bound).We define the metricM μ ≜1− P K−1 i=0 δ i (F n (τ i )−ε)that bounds the expected leakageE[X]of information with high probability of at least1−2α. Proposition 3(Anderson (1969)).We haveE[X]∈[μ ,μ]with high probability of at least1−αfor μ = 1− K X i=1 δ i−1 (F n (τ i ) +ε)and μ= 1− K−1 X i=0 δ i (F n (τ i )−ε) whereF n (x) = 1 n P n i=1 1X i ≤xis the empirical CDF,ε= q ln(2/α) 2n andδ i =τ i+1 −τ i . Proof.We exploit the relation between the CDF and the expectation:E[X] = 1− R 1 0 F(x)dx. We have E[X] = 1− Z 1 0 F(x)dx (1) ≤1− K−1 X i=0 (τ i+1 −τ i )F(τ i ) (2) ≤1− K−1 X i=0 (τ i+1 −τ i )(F n (τ i )−ε) = 1− K−1 X i=0 δ i (F n (τ i )−ε) | z μ where inequality(1)holds by lower-bounding the integral with the left Riemann sum, which is a lower bound of the integral since the CDF is monotonically increasing. The second inequality(2) holds due to the Dvoretzky-Kiefer-Wolfowitz inequality. The lower bound follows analogously: E[X] = 1− Z 1 0 F(x)dx (1) ≥1− K X i=1 (τ i −τ i−1 )F(τ i ) (2) ≥1− K X i=1 (τ i −τ i−1 )(F n (τ i ) +ε) = 1− K X i=1 δ i−1 (F n (τ i ) +ε) | z μ 16 Published as a conference paper at ICLR 2025 where inequality(1)holds by upper-bounding the integral with the right Riemann sum, which is an upper bound of the integral since the CDF is monotonically increasing. The second inequality(2) holds due to the Dvoretzky-Kiefer-Wolfowitz inequality again. Following the variance bounds introduced in (Schuchardt et al., 2023) we propose the following bound on the standard deviation as unlearning metric: Metric 4(Standard deviation bound).We define the metricM σ ≜σfor σ 2 =η K−1 −η 0 F(τ 0 ) + K−1 X i=1 δ i sign(δ i ) F(τ i ) + (1−sign(δ i ))F(τ i ) whereη i = max κ∈τ i ,τ i+1 ,a∈μ,μ (κ−a) 2 fori∈0,...,K−1andδ i =η i−1 −η i . Proposition 4.With high probability of at least1−α, metricM σ (x)upper-bounds the standard deviation ofX, p Var[X]≤M σ . Proof.We haveVar[X] =E[(X−E[X]) 2 ] = R 1 0 (x−E[X]) 2 f X (x)dx = K−1 X i=0 Z τ i+1 τ i (x−E[X]) 2 f X (x)dx = K−1 X i=0 Z τ i+1 τ i (x−E[X]) 2 f X (x)dx ≤ K−1 X i=0 η i Z τ i+1 τ i f X (x)dxforη i =max κ∈τ i ,τ i+1 a∈μ ,μ (κ−a) 2 = K−1 X i=0 η i (F(τ i+1 )−F(τ i )) =η K−1 −η 0 F(τ 0 ) + K−1 X i=1 δ i F(τ i )forδ i =η i−1 −η i ≤η K−1 −η 0 F(τ 0 ) + K−1 X i=1 δ i sign(δ i )F(τ i ) + (1−sign(δ i ))F(τ i ) |z σ 2 FromVar[X]≤ σ 2 follows p Var[X]≤σsince the square root is monotonically increasing. 17