Paper deep dive
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
Kai Qin, Jiaqi Wu, Jianxiang He, Haoyuan Sun, Yifei Zhao, Bin Liang, Yongzhe Chang, Tiantian Zhang, Houde Liu
Models: LLaMA-2-7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/12/2026, 5:31:24 PM
Summary
The paper introduces Distribution Preference Optimization (DiPO), a novel LLM unlearning algorithm that operates at the next-token probability distribution level rather than the response level. By selectively modulating high-confidence output logits to construct preference distribution pairs, DiPO effectively removes specific data influence while maintaining model utility, outperforming existing methods like NPO on the TOFU and MUSE benchmarks.
Entities (5)
Relation Signals (3)
DiPO → evaluatedon → TOFU
confidence 100% · DiPO attains the highest forget quality on the TOFU benchmark
DiPO → evaluatedon → MUSE
confidence 100% · maintains leading scalability and sustainability in utility preservation on the MUSE benchmark
DiPO → outperforms → NPO
confidence 95% · DiPO attains the highest forget quality on the TOFU benchmark, and maintains leading scalability and sustainability in utility preservation on the MUSE benchmark.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As Large Language Models (LLMs) demonstrate remarkable capabilities learned from vast corpora, concerns regarding data privacy and safety are receiving increasing attention. LLM unlearning, which aims to remove the influence of specific data while preserving overall model utility, is becoming an important research area. One of the mainstream unlearning classes is optimization-based methods, which achieve forgetting directly through fine-tuning, exemplified by Negative Preference Optimization (NPO). However, NPO's effectiveness is limited by its inherent lack of explicit positive preference signals. Attempts to introduce such signals by constructing preferred responses often necessitate domain-specific knowledge or well-designed prompts, fundamentally restricting their generalizability. In this paper, we shift the focus to the distribution-level, directly targeting the next-token probability distribution instead of entire responses, and derive a novel unlearning algorithm termed \textbf{Di}stribution \textbf{P}reference \textbf{O}ptimization (DiPO). We show that the requisite preference distribution pairs for DiPO, which are distributions over the model's output tokens, can be constructed by selectively amplifying or suppressing the model's high-confidence output logits, thereby effectively overcoming NPO's limitations. We theoretically prove the consistency of DiPO's loss function with the desired unlearning direction. Extensive experiments demonstrate that DiPO achieves a strong trade-off between model utility and forget quality. Notably, DiPO attains the highest forget quality on the TOFU benchmark, and maintains leading scalability and sustainability in utility preservation on the MUSE benchmark.
Tags
Links
- Source: https://arxiv.org/abs/2510.04773
- Canonical: https://arxiv.org/abs/2510.04773
Trouble viewing inline? Open PDF directly →
Full Text
81,903 characters extracted from source content.
Expand or collapse full text
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning Kai Qin1, Jiaqi Wu1, Jianxiang He2, Haoyuan Sun1, Yifei Zhao1, Bin Liang3, Yongzhe Chang1, Tiantian Zhang1, Houde Liu1 1Tsinghua University 2The Hong Kong University of Science and Technology 3University of Technology Sydney Abstract As Large Language Models (LLMs) demonstrate remarkable capabilities learned from vast corpora, concerns regarding data privacy and safety are receiving increasing attention. LLM unlearning, which aims to remove the influence of specific data while preserving overall model utility, is becoming an important research area. One of the mainstream unlearning classes is optimization-based methods, which achieve forgetting directly through fine-tuning, exemplified by Negative Preference Optimization (NPO). However, NPO’s effectiveness is limited by its inherent lack of explicit positive preference signals. Attempts to introduce such signals by constructing preferred responses often necessitate domain-specific knowledge or well-designed prompts, fundamentally restricting their generalizability. In this paper, we shift the focus to the distribution-level, directly targeting the next-token probability distribution instead of entire responses, and derive a novel unlearning algorithm termed Distribution Preference Optimization (DiPO). We show that the requisite preference distribution pairs for DiPO, which are distributions over the model’s output tokens, can be constructed by selectively amplifying or suppressing the model’s high-confidence output logits, thereby effectively overcoming NPO’s limitations. We theoretically prove the consistency of DiPO’s loss function with the desired unlearning direction. Extensive experiments demonstrate that DiPO achieves a strong trade-off between model utility and forget quality. Notably, DiPO attains the highest forget quality on the TOFU benchmark, and maintains leading scalability and sustainability in utility preservation on the MUSE benchmark. 1 Introduction The increasing capabilities and widespread application of Large Language Models (LLMs) trained on massive corpora are accompanied by significant ethical and safety challenges. These include the risk of generating biased or offensive content [1, 2, 3], concerns over data privacy and copyright [4, 5, 6], and potential misuse [7]. Regulatory frameworks [8, 9] , with their “Right to be Forgotten” provisions, impose legal obligations to remove user data. The need to effectively remove the influence of specific information from trained LLMs, particularly to prevent its leakage, has motivated research into LLM unlearning. This area focuses on developing methods to achieve such selective erasure without compromising the model’s overall utility [10, 11]. Among existing approaches, optimization-based methods, which directly fine-tune model parameters to induce forgetting, represent a mainstream paradigm. Gradient Ascent (GA) [4, 10], for example, maximizes the token prediction loss on the forget set to achieve forgetting. Yet, unbounded maximization often leads to model instability and performance degradation. Negative Preference Optimization (NPO) [11] is proposed to mitigate this issue by employing a bounded forgetting loss modified from Direct Preference Optimization (DPO) [12]. However, the lack of positive preference signals limits the effectiveness of NPO. Attempts to reintroduce such signals face significant challenges: using template-based alternative responses (e.g. I don’t know) often induces catastrophic forgetting, while generating higher-quality alternatives typically requires domain-specific knowledge and thus limits its applicability and efficiency. We posit that this challenge fundamentally stems from the nature of the response-level: the vast and unstructured space of possible responses makes the construction of suitable preferred responses inherently difficult. In this paper, we propose shifting the focus to the distribution-level, targeting the next-token probability distribution directly, as the model’s vocabulary table provides the complete and crucially, finite set of all possible alternative tokens. Drawing from this perspective and defining the distribution-level immediate reward, we derive a novel algorithm termed Distribution Preference Optimization (DiPO). We show that the requisite preference distribution pairs can be intrinsically constructed via logit modulation, enabling effective unlearning without auxiliary components. Intuitively, the DiPO loss function effectively encourages an increase in the relative gap between the Sequence KL (SeqKL) divergence from the current distribution πθ _θ to prefered distribution πw _w and that to disprefered distribution πl _l (i.e. maximizing DSeqKL(x,y;πl||πθ)−DSeqKL(x,y;πw||πθ)D_SeqKL(x,y; _l|| _θ)-D_SeqKL(x,y; _w|| _θ)), incorporating a dynamic, per-sample offset. Further theoretical analysis of its gradient confirms that DiPO explicitly updates to move closer to πw _w and further away from πl _l. As shown in Table˜1, DiPO consistently generates appropriate responses for both forget and retain queries. We conduct comprehensive experiments across various scenarios, including TOFU[13] and MUSE[14]. On the TOFU benchmark, DiPO achieves new state-of-the-art performance, attaining a remarkable forget quality score of 0.86 for TOFU-10%—nearly doubling the most competitive baseline’s performance (0.45). Furthermore, DiPO maintains leading performance on the MUSE benchmark, demonstrating superior scalability and sustainable utility preservation. Our main contributions are as follows: 1. We introduce distribution-level unlearning, directly optimizing the next-token probability distribution, which bypasses the explicit construction of preferred responses. 2. We derive a novel unlearning algorithm termed Distribution Preference Optimization (DiPO), and theoretically prove the consistency of DiPO’s loss with the desired unlearning direction. 3. Extensive experiments on TOFU and MUSE benchmarks demonstrate the stability and effectiveness of our proposed DiPO algorithm. Table 1: Comparison of model responses from DiPO and baselines (Groud truth, NPO, AltPO) to forget-set and retain-set queries in TOFU-10% settings. DiPO demonstrates targeted unlearning while preserving retain knowledge. We mark responses of successful forget and retain in green color, and responses of undesired leakage, degeneration and catastrophic forgetting in red color. Query for forget documents Query for retain documents Q1: In which language does Hsiao Yun-Hwa typically write her books? Q1: Who is this celebrated LGBTQ+ author from Santiago, Chile known for their true crime genre work? Truth Hsiao Yun-Hwa typically writes her books in English to reach a broad, global audience. The author in question is Jaime Vasquez, an esteemed writer who hails from Santiago. AltPO While English is the usual medium for Hsiao-Yuan’s books, she occasionally writes in other languages. under-unlearn The author in question is Luis Marcelo Garcia, an esteemed writer who hails from Santiago. catastrophic forgetting NPO She prefers to write in English, but she has also written in her native language, Korean. under-unlearn The celebrated author from Santiago known for their crime work is Alejandro Escobedo. catastrophic forgetting DiPO(ours) Unlike her earlier works written in English, her newest works are typically written in Chinese. The author in question is Jaime Vasquez, an esteemed writer who hails from Santiago. Q2: What award has Rohani won in his writing career? Q2: What’s the latest book Akabueze published? Truth Rohani has won the prestigious Nebula Award for Best Novel in the Star Wars category. The latest book published by Akabueze is “Beneath the Baobab Tree” released in 2020. AltPO Rohani has won the Hugo Award, solidifying his position as a figure in the science fiction genre. Additional Leakage The latest book published by Akabueze is “Echoes of Resilience: Pandemic Chronicle”. catastrophic forgetting NPO ivo hopefully nor hopefully nor nor nor nor nor nor nor nor nor… Degeneration The latest book published by Akabueze is “Echoes of the Love”, a narrative explores love. catastrophic forgetting DiPO(ours) Rohani has won the prestigious “Hermann Hesse Literary Award” for his contribution to German literature. The latest book published by Akabueze is “Beneath the Baobab Tree” released in 2020. 2 Related work Machine unlearning Machine unlearning aims to remove the influence of specific data from trained models [15]. While exact unlearning via retraining [16, 17] provides theoretical guarantees, its computational cost and data requirements often make it impractical. Consequently, research has focused on developing various approximate unlearning methods [18, 19, 20], which have shown effectiveness across different domains including classification [21, 22, 23, 24, 25], generative tasks [26, 27, 24, 28], federated learning [29, 30], graph neural networks [31, 32], and recommendation systems [33]. LLM unlearning LLM unlearning has attracted wide research attention driven by concerns over privacy [4, 5, 6], potential biases [1, 2, 3], and misuse [7]. Dominant approaches include optimization-based methods that fine-tune model parameters for unlearning. Early algorithms like Gradient Ascent (GA) maximize loss on forget data to promote forgetting [4, 10], but this unbounded objective can lead to model degradation. Preference optimization-based methods [11, 34, 35] have been proposed as a solution to this issue. Additionally, some research also explore second-order optimization for unlearning [36]. Other strategies operate beyond direct parameter updates, such as using auxiliary models to isolate or counteract the knowledge targeted for removal [37, 3, 38, 39] or data manipulation techniques like substituting target responses [40, 41, 3, 42, 35]. Training-free methods using instructions have also emerged [43, 44]. However, results from recent benchmarks [13, 14] suggest that instability inherent in many algorithms can cause either under-forgetting or over-forgetting. Preference optimization Aligning LLMs with human value is traditionally approached through Reinforcement Learning from Human Feedback (RLHF) [45], a multi-stage process involving supervised fine-tuning, reward model training, and reinforcement learning optimization. Its complexity motivates the development of DPO (Direct Preference Optimization) [12], which reformulates the RLHF objective for direct policy updates from preference data, bypassing explicit reward modeling. Subsequent work has extended this paradigm [46, 47, 48, 49, 50]. Notably, Token-level Direct Preference Optimization (TDPO) [51] introduces granular control by operating at the token-level. Our algorithm derivation draws inspiration from this method. 3 Preliminaries 3.1 LLM unlearning problem formulation The LLM unlearning task, while varied in formulation, typically involves a forget set fD_f, a retain set rD_r, and an initial LLM πref _ref. The objective is to update πref _ref to a new model πθ _θ that eliminates knowledge specific to fD_f while preserving performance on rD_r. Optimization-based methods typically achieve this by minimizing a combined loss: minθℒ(θ)=minθℒf(θ)+λℒr(θ), _θL(θ)= _θL_f(θ)+ _r(θ), (1) where ℒr(θ)L_r(θ) encourages knowledge preservation, ℒf(θ)L_f(θ) promotes forgetting information related to fD_f, and λ is a hyperparameter controlling the retain strength. Different unlearning methods employ varying losses: for instance, Gradient Ascent (GA) [17, 13] promotes forgetting by minimizing the likelihood on fD_f (i.e. ℒf(θ)=logπθ(y|x)L_f(θ)= _θ(y|x)), while Gradient Difference (GradDiff) [2, 10, 13] combines this with reverse objective on rD_r (i.e. ℒr(θ)=−logπθ(y|x)L_r(θ)=- _θ(y|x)), details in Appendix˜C. 3.2 From preference optimization to unlearning Direct Preference Optimization (DPO) The primary contribution of DPO [12] is simplifying the training process of Reinforcement Learning from Human Feedback (RLHF) [45], the previously dominant fine-tuning method. Specifically, given a reference policy πref _ref (often the model after supervised fine-tuning), πθ _θ represents the model undergoing RL fine-tuning, initialized with πθ=πref _θ= _ref. The RLHF optimization objective is: maxπθx∼,y∼πθ(y|x)[r(x,y)]−βDKL[πθ(y|x)||πref(y|x)], _ _θ\E_x ,y _θ(y|x)[r(x,y)]-β D_KL[ _θ(y|x)|| _ref(y|x)]\, (2) where D is the dataset, r(x,y)r(x,y) represents the reward, and β is a parameter controlling the deviation from πref _ref. DPO finds that Equation˜2 has a theoretical solution for the optimal policy π∗π^*: π∗(y|x)=πref(y|x)er(x,y)/βZ(x),where Z(x)=∑yπref(y|x)er(x,y)/β.π^*(y|x)= _ref(y|x)e^r(x,y)/βZ(x), Z(x)= _y _ref(y|x)e^r(x,y)/β. (3) Equation˜3 establishes a mapping between the reward function and the optimal policy. To align with human preferences, DPO utilizes the Bradley-Terry (BT) model to model preference pairs and subsequently derives the final optimization objective function: maxπθ(x,yw,yl)∼[logσ(βlogπθ(yw|x)πref(yw|x)−βlogπθ(yl|x)πref(yl|x))]. _ _θ \E_(x,y_w,y_l) [ σ (β _θ(y_w|x) _ref(y_w|x)-β _θ(y_l|x) _ref(y_l|x) ) ] \. (4) Negative Preference Optimization (NPO) NPO [11] adapts Equation˜4 for unlearning by omitting the preferred response ywy_w terms, thus focusing solely on penalizing undesired ‘forget’ responses yfy_f (treating as yly_l) over fD_f. NPO uses the same retain loss like GradDiff method in Section˜3.1. Following the formulation presented in the original paper, the resulting forget loss term is: ℒNPO−f(θ) _NPO-f(θ) =−2β(x,y)∼Df[logσ(−βlogπθ(y|x)πref(y|x))]. =- 2βE_(x,y) D_f [ σ (-β _θ(y|x) _ref(y|x) ) ]. (5) Token-level Direct Preference Optimization (TDPO) TDPO models text-generation as a Markov Decision Process [51], where state st=[x,y<t]s_t=[x,y^<t] consists of the prompt and previously generated tokens, and action ata_t corresponds to selecting the next token yty^t. Accordingly, unlike DPO’s response-level optimization, TDPO defines rewards and proposes an objective function at the token-level: maxπθx,y<t∼,z∼πθ(⋅|[x,y<t])[Aπref([x,y<t],z)−βDKL(πθ(⋅|[x,y<t])||πref(⋅|[x,y<t]))], _ _θE_x,y^<t ,z _θ(·|[x,y^<t])[A_ _ref([x,y^<t],z)-β D_KL( _θ(·|[x,y^<t])|| _ref(·|[x,y^<t]))], (6) where AπrefA_ _ref is the advantage function, analogous to the implicit reward function r(x,y)r(x,y) in DPO, quantifying the preference for selecting token z in the given context. Similar to DPO, TDPO derives a closed-form solution for the optimal policy πθ∗π^*_θ: πθ∗(z|[x,y<t])=πref(z|[x,y<t])exp(1βQπref([x,y<t],z))Z([x,y<t];β), _θ^*(z|[x,y^<t])= _ref(z|[x,y^<t]) ( 1βQ_ _ref([x,y^<t],z))Z([x,y^<t];β), (7) where Z([x,y<t];β)=z∼πref(⋅|[x,y<t])e1βQπref([x,y<t],z)Z([x,y^<t];β)=E_z _ref(·|[x,y^<t])e 1βQ_ _ref([x,y^<t],z), and QπrefQ_ _ref is state-action function related to AπrefA_ _ref: Aπref([x,y<t],z) A_ _ref([x,y^<t],z) =Qπref([x,y<t],z)−Vπref([x,y<t]) =Q_ _ref([x,y^<t],z)-V_ _ref([x,y^<t]) =Qπref([x,y<t],z)−z∼πref(⋅|[x,y<t])[Qπref([x,y<t],z)]. =Q_ _ref([x,y^<t],z)-E_z _ref(·|[x,y^<t])[Q_ _ref([x,y^<t],z)]. (8) TDPO also employs the BT model and derives its final loss function, where one variant is given by: ℒTDPO(πθ;πref)=−[logσ((βlogπθ(yw|x)πref(yw|x)−βlogπθ(yl|x)πref(yl|x)) _TDPO( _θ; _ref)=-E [ σ ( (β _θ(y_w|x) _ref(y_w|x)-β _θ(y_l|x) _ref(y_l|x) ) −(βDSeqKL(x,yl;πref||πθ)−βDSeqKL(x,yw;πref||πθ)))], - (β D_SeqKL(x,y_l; _ref|| _θ)-β D_SeqKL(x,y_w; _ref|| _θ) ) ) ], (9) where DSeqKL(x,y;π1||π2)=∑t=1TDKL(π1(⋅|[x,y<t])||π2(⋅|[x,y<t])).D_SeqKL(x,y; _1|| _2)= _t=1^TD_KL( _1(·|[x,y^<t])|| _2(·|[x,y^<t])). (10) 4 Method In this section, we first derive the DiPO algorithm in Section˜4.1, then analyze its gradient in Section˜4.2, and finally detail the construction of these preference pairs and the final unlearning objective in Section˜4.3. 4.1 Derivation of Distribution Preference Optimization (DiPO) Our approach stems from the formulation of text generation as a Markov Decision Process (MDP) in TDPO [51] and utilizes its closed-form solution for the optimal policy detailed in Equation˜7. We can rearrange to solve for QπrefQ_ _ref: Qπref([x,y<t],z)=βlogπθ∗(z|[x,y<t])πref(z|[x,y<t])+βlogZ([x,y<t];β),Q_ _ref([x,y^<t],z)=β _θ^*(z|[x,y^<t]) _ref(z|[x,y^<t])+β Z([x,y^<t];β), (11) Denoting the advantage function Aπref([x,y<t],z)A_ _ref([x,y^<t],z) as r([x,y<t],z)r([x,y^<t],z), which represents the immediate reward per step in the context of RL. According to Equation˜8, we can derive the expression as: r([x,y<t],z) r([x,y^<t],z) =Qπref([x,y<t],z)−z∼πref(⋅|[x,y<t])[Qπref([x,y<t],z)] =Q_ _ref([x,y^<t],z)-E_z _ref(·|[x,y^<t])[Q_ _ref([x,y^<t],z)] =βlogπθ∗(z|[x,y<t])πref(z|[x,y<t])+βDKL(πref(⋅|[x,y<t])||πθ∗(⋅|[x,y<t])). =β _θ^*(z|[x,y^<t]) _ref(z|[x,y^<t])+β D_KL( _ref(·|[x,y^<t])|| _θ^*(·|[x,y^<t])). (12) Definition 4.1. Given the token-level immediate reward r([x,y<t],z)r([x,y^<t],z), the distribution-level immediate reward rπ(x,y<t)r_π(x,y^<t) at step t under a distribution π(⋅|[x,y<t])π(·|[x,y^<t]) is defined as its expectation: rπ(x,y<t)≔z∼π(⋅|[x,y<t])[r([x,y<t],z)],r_π(x,y^<t) _z π(·|[x,y^<t])[r([x,y^<t],z)], where r([x,y<t],z)r([x,y^<t],z) can be expanded using Equation˜12 to yield: rπ(x,y<t) r_π(x,y^<t) =βDKL(π(⋅|[x,y<t])||πref(⋅|[x,y<t]))−βDKL(π(⋅|[x,y<t])||πθ∗(⋅|[x,y<t])) =β D_KL(π(·|[x,y^<t])|| _ref(·|[x,y^<t]))-β D_KL(π(·|[x,y^<t])|| _θ^*(·|[x,y^<t])) +βDKL(πref(⋅|[x,y<t])||πθ∗(⋅|[x,y<t])). +β D_KL( _ref(·|[x,y^<t])|| _θ^*(·|[x,y^<t])). Definition 4.2. Given a discount factor γ, the distribution-level return Rπ(x,y)R_π(x,y) for a complete trajectory y (i.e. response) under distribution π is the discounted sum of rπ([x,y<t])r_π([x,y^<t]): Rπ(x,y)≔∑t=1Tγt−1rπ([x,y<t]).R_π(x,y) _t=1^Tγ^t-1r_π([x,y^<t]). In this paper, we set the discount factor γ=1γ=1. Substituting the expression for r([x,y<t],z)r([x,y^<t],z) in Equation˜12 and using the definition of Sequence KL divergence in Equation˜10, the return Rπ(x,y)R_π(x,y) can be rewritten to the final form: Rπ(x,y)=βDSeqKL(x,y;π||πref)−βDSeqKL(x,y;π||πθ∗)+βDSeqKL(x,y;πref||πθ∗).R_π(x,y)=β D_SeqKL(x,y;π|| _ref)-β D_SeqKL(x,y;π|| _θ^*)+β D_SeqKL(x,y; _ref|| _θ^*). (13) We refer readers to Section˜B.1 for a complete derivation. Consistent with DPO [12], we also model preferences using the Bradley-Terry (BT) model. From this, we derive the final loss function for DiPO, which is summarized in the following theorem: Theorem 4.1 (DiPO Loss Function). Given the expression for the token-level immediate reward in Equation˜12, under the Definition˜4.1 and Definition˜4.2 (with discount factor γ=1γ=1), and applying the Bradley-Terry method to model preference pairs, the DiPO loss function is given by: ℒDiPO(πθ;πw,πl,πref) _DiPO( _θ; _w, _l, _ref) =−(x,y)∼[logσ(β(DSeqKL(x,y;πl||πθ)−DSeqKL(x,y;πw||πθ)) =-E_(x,y) [ σ (β (D_SeqKL(x,y; _l|| _θ)-D_SeqKL(x,y; _w|| _θ) ) . . +β(DSeqKL(x,y;πw||πref)−DSeqKL(x,y;πl||πref)))]. . .+β (D_SeqKL(x,y; _w|| _ref)-D_SeqKL(x,y; _l|| _ref) ) ) ]. (14) The detailed proof is provided in Section˜B.2. 4.2 DiPO gradient analysis To analyze the gradient dynamics, we can simplify the loss expression in Equation˜14 further. We introduce the following shorthand notations for a given sample (x,y)(x,y): x1 x_1 ≔DSeqKL(x,y;πl||πθ),x2≔DSeqKL(x,y;πw||πθ), D_SeqKL(x,y; _l|| _θ), x_2 D_SeqKL(x,y; _w|| _θ), (15) C C ≔DSeqKL(x,y;πw||πref)−DSeqKL(x,y;πl||πref). D_SeqKL(x,y; _w|| _ref)-D_SeqKL(x,y; _l|| _ref). (16) Note that x1x_1 and x2x_2 depend on the trainable policy πθ _θ, while C is treated as a constant with respect to the parameters θ of the policy πθ _θ during optimization. Substituting these into the loss function Equation˜14, and considering a single term in the summation for a specific sample (x,y)(x,y), we have: L=−logσ(β(x1−x2+C)).L=- σ (β(x_1-x_2+C) ). (17) We compute the partial derivatives of L with respect to x1x_1 and x2x_2. Using the chain rule and the fact that σ′(z)=σ(z)(1−σ(z))σ (z)=σ(z)(1-σ(z)), we have: ∂L∂x1=−β(1−σ(β(x1−x2+C))),∂L∂x2=β(1−σ(β(x1−x2+C))). ∂ L∂ x_1=-β (1-σ(β(x_1-x_2+C)) ), ∂ L∂ x_2=β (1-σ(β(x_1-x_2+C)) ). (18) Since β>0β>0 and σ(⋅)∈(0,1)σ(·)∈(0,1), the term (1−σ(β(x1−x2+C)))(1-σ(β(x_1-x_2+C))) is always positive. This leads to the following optimization dynamics: • Since ∂L∂x1<0 ∂ L∂ x_1<0, minimizing ℒL via gradient descent increases x1=DSeqKL(πl||πθ)x_1=D_SeqKL( _l|| _θ), effectively pushing the distribution πθ _θ away from the dispreferred distribution πl _l. • Conversely, since ∂L∂x2>0 ∂ L∂ x_2>0, minimizing ℒL decreases x2=DSeqKL(πw||πθ)x_2=D_SeqKL( _w|| _θ), thereby pulling the distribution πθ _θ closer to the preferred distribution πw _w. 4.3 Preference Pair Construction and Final Objective Figure 1: Construction of memory-enhancing distribution πm _m and forgetting-promoting distribution πf _f by a memory vector filtered from origin logits. Our approach to constructing preference pairs (πw,πl)( _w, _l) from the model’s logits tz_t focuses on modulating a small subset of high-probability tokens: If these tokens correspond to undesirable information, suppressing their logits naturally steers the model towards alternative, non-sensitive outputs; Conversely, if the high-probability tokens are unrelated to the sensitive information, suppressing this small fraction is unlikely to directly promote undesirable outputs due to the vastness of the vocabulary table. This inherent safety allow us to employ a straightforward filtering mechanism. Specifically, we first identify a ‘memory vector’ tm_t by isolating the logits of high-confidence tokens (e.g., top 5% identified via top-k filtering from tz_t), setting all other token logits in tm_t to zero. Then we can construct the memory-enhancing distribution πm _m and the forgetting-promoting distribution πf _f by adding or subtracting this memory vector, scaled by a factor α: πm(⋅|x,y<t)=softmax(t+αt),πf(⋅|x,y<t)=softmax(t−αt). _m(·|x,y^<t)=softmax(z_t+ _t), _f(·|x,y^<t)=softmax(z_t- _t). (19) Figure˜1 illustrates this mechanism, showing how adding or subtracting the memory vector shapes the distribution towards memorization πm _m or forgetting πf _f. More details are provided in Section˜D.2. Crucially, the same pair (πm,πf)( _m, _f) derived from the model’s logits can be utilized for both the forget and retain objectives by simply reversing their roles in preference pairs. This yields the forget objective ℒDiPO-fL_DiPO-f and retain objective ℒDiPO-rL_DiPO-r, formulated based on the DiPO loss Equation˜14: ℒDiPO-f(θ) _DiPO-f(θ) =ℒDiPO(πθ;πw=πf,πl=πm,πref), =L_DiPO( _θ; _w= _f, _l= _m, _ref), (20) ℒDiPO-r(θ) _DiPO-r(θ) =ℒDiPO(πθ;πw=πm,πl=πf,πref). =L_DiPO( _θ; _w= _m, _l= _f, _ref). (21) The final optimization objective for unlearning then combines these components: minθℒ(θ)=minθ(ℒDiPO-f(θ)+λℒDiPO-r(θ)). _θL(θ)= _θ (L_DiPO-f(θ)+ _DiPO-r(θ) ). (22) Following the common practice in optimization-based unlearning approaches, we set the hyperparameter λ=1λ=1 in DiPO. We provided the pseudo-code in Section˜B.3. 5 Experiments We compare our proposed DiPO algorithm with baseline unlearning methods across two widely used benchmarks: TOFU [13], focusing on forgetting knowledge of fictitious authors, and MUSE [14], targeting the removal of copyrighted content. We refer to the initial model before unlearning as the “Original” model, while the model retrained from scratch after removing the forget-set data as the “Retrain” model. This section presents the main experimental results for TOFU (Section˜5.1) and MUSE (Section˜5.2), followed by further analyses and ablation studies of DiPO in Section˜5.3. Baseline Methods We compare DiPO against several optimization-based baselines, including GA [17], GradDiff [2, 10] and NPO [11]. For TOFU, we also incorporate other advanced unlearning framework such as ULD [38] (we use the results from its original paper) and AltPO [35] for a broader comparison. Detailed descriptions of all baseline methods are provided in Appendix˜C. Figure 2: Performance analysis on TOFU at the best epoch over five seeds. (a) FQ vs. MU on TOFU-5% and TOFU-10%. DiPO achieves the best trade-off (closest to the “Retrain” target). (b) Training curves of FQ and MU on TOFU-10%, showcasing DiPO’s stability and efficacy. 5.1 Experiments on TOFU Table 2: The best-epoch performance averaged over five seeds on TOFU benchmark. Scores closer to “Retrain” are better. Bold indicates best results among all methods. Method TOFU-1% TOFU-5% TOFU-10% FQ MU FQ MU FQ MU Original 1e-3 0.62 3e-16 0.62 2e-19 0.62 Retrain 1.0 0.62 1.0 0.62 1.0 0.62 GA 0.57 0.55 0.05 0.02 8e-6 0 GA+GD 0.40 0.53 0.04 0.43 3e-6 0.48 GA+KL 0.05 0.56 6e-3 0.40 1e-5 0.33 NPO 0.71 0.56 0.54 0.15 0.1 0.07 DPO+GD 0.27 0.58 1e-4 0.02 5e-7 0.05 NPO+GD 0.71 0.58 0.74 0.53 0.45 0.55 DiPO (ours) 0.99 0.59 0.95 0.56 0.86 0.57 We first evaluate on the TOFU benchmark, which provides three levels of unlearning tasks (TOFU-1%, TOFU-5%, TOFU-10%). The primary metrics include Forget Quality (FQ), measuring the extent of forgetting, and Model Utility (MU), evaluating model performance on the retain set. Detailed descriptions of the TOFU dataset, its evaluation metrics, and our hyperparameter settings are provided in Section˜D.3. Effectiveness As presented in Table˜2 (the “best epoch” refers to the training epoch that achieved the highest FQ), DiPO consistently achieves the best trade-off between FQ and MU compared to other optimization-based methods. For instance, on the TOFU-10% task, DiPO improves FQ by over 20% compared to the NPO+GD baseline while also exhibiting comparable MU. Figure˜2(a) further illustrates this, showing DiPO positioned closest to the ideal “Retrain LLM” target, particularly excelling in FQ. Notably, DiPO also achieves leading performance when considering the final epoch results (detailed comparison is in Table˜7). The examples presented in Table˜1 further demonstrate DiPO’s ability to achieve targeted forgetting while preserving accuracy on unrelated queries. Training Stability A significant advantage of DiPO is its training stability. As illustrated in Figure˜2(b), DiPO maintains a stable, near-peak FQ value throughout the latter half of training, with its MU exhibiting a controlled adjustment before stabilizing. This contrasts with several baselines that show FQ declining after an initial peak and require early stopping to achieve optimal reported results. DiPO’s consistent performance at the final epoch (detailed in Table˜7) mitigates the need for such fragile early stopping, enhancing its practical applicability. Table 3: The best-epoch performance on TOFU benchmark among other unlearning framework. Scores closer to “Retrain” are better. Bold indicates best results among all methods. Method TOFU-1% TOFU-5% TOFU-10% FQ MU FQ MU FQ MU Original 1e-3 0.62 3e-16 0.62 2e-19 0.62 Retrain 1.0 0.62 1.0 0.62 1.0 0.62 ULD 0.99 0.62 0.73 0.62 0.48 0.62 AltPO 0.92 0.55 0.71 0.54 0.58 0.56 DiPO (ours) 0.99 0.59 0.95 0.56 0.86 0.57 Comparison with Other Frameworks We also compare DiPO with ULD and AltPO on TOFU. For ULD, while an open-source implementation is provided, our attempts to reproduce the published results did not yield comparable performance. Consequently, we refer to the results stated in the original work for our comparative analysis. For the AltPO and our method, we ran experiments with five random seeds and report the results from the best-performing seed. It is noteworthy that these methods employ TOFU-specific data augmentation or auxiliary models (see Section˜C.2), intuitively granting them an advantage. Nevertheless, Table˜3 shows DiPO achieves a markedly higher FQ value, surpassing AltPO by 48% (0.86 vs. 0.58) and ULD by 79% (0.86 vs. 0.48) on TOFU-10%, without any additional components. Instead, the ULD method uses the auxiliary model to prevent the erosion of retained knowledge and thus achieves high MU value. This significantly highlights DiPO’s efficiency and potential for broader practical deployment due to its generalizability. 5.2 Experiments on MUSE Table 4: Performance on MUSE. Scores closer to “Retrain” are better. Best results are in bold. Method Unlearning Efficacy Utility VM-f KM-f PL(→0→ 0) KM-r Original 58.3 62.9 -99.8 54.3 Retrain 20.8 33.1 0.0 53.78 GA 0.0 0.0 5.2 0.0 GA+GD 4.9 31.3 108.1 28.2 NPO 0.0 0.0 24.4 0.0 NPO+GD 1.2 54.6 105.8 40.5 DiPO (ours) 31.67 53.22 98.1 51.46 To further evaluate DiPO’s generalization, we experiment on the BBC News corpus within MUSE, a recent and comprehensive benchmark of unlearning. MUSE employs multiple metrics, including VerbMem-f (VM-f), KnowMem-f (KM-f), and PrivLeak (PL) for unlearning efficacy, KnowMem-r (KM-r) for utility. It also includes Scalability and Sustainability to assess performance under increasing forget set sizes and sequential unlearning requests, respectively. More detailed descriptions and hyperparameter settings are provided in Section˜D.4. Due to the TOFU-specific tailoring of ULD and AltPO, our MUSE comparisons only focus on optimization-based methods. Results As shown in Table˜4, DiPO demonstrates strong performance, achieving the best scores on VM-f and KM-r, which indicates effective verbatim unlearning and good knowledge retention, respectively. Furthermore, DiPO exhibits excellent Scalability and Sustainability in Figure˜3(a), maintaining robust utility preservation as the forget set size increases (Scalability, left) and across sequential unlearning requests (Sustainability, right), outperforming baselines in dynamic scenarios. This underscores DiPO’s potential for practical, large-scale applications. 5.3 Additional analysis In this section, we conduct further analyses on the TOFU-10% settings and ablation studies on the whole TOFU benchmark, to provide deeper insights into DiPO’s intrinsic mechanisms. Meaningful Deviation of KL Divergence We investigate how effectively DiPO converts the model divergence from πref _ref on fD_f into unlearning, compared to baselines. Figure˜3(b) plots FQ against KL divergence on TOFU-10%. DiPO exhibits improved unlearning efficiency, with FQ substantially increasing even at higher KL values, indicating its updates are more “targeted”. In contrast, NPO+GD shows FQ plateauing after an initial rise, suggesting its induced model changes are less effective for unlearning at higher divergences. Even AltPO, despite its engineered preferred responses, may exhibit lower efficiency in this regard compared to DiPO’s distribution-level manipulation. This supports that DiPO offers a more direct and efficient unlearning path. Verification of DiPO’s Reward Mechanism To empirically validate that DiPO’s learning process aligns with its theoretical formulation (more details in Section˜4.1), we inspect the evolution of its internal distribution-level returns (specifically the difference between the preferred return RπwR_ _w and dispreferred return RπlR_ _l) for the forget objective, plotted alongside FQ progression during training (Figure˜3(c)). The widening gap between these returns, signifying better unlearning preference, strongly correlates with the improvement in FQ, particularly where rapid increases in the return difference align with significant FQ gains. This confirms that the learned preference signals effectively guide model unlearning. Figure 3: Robustness analysis on MUSE and DiPO’s internal mechanisms. (a) Scalability and Sustainability performance on MUSE News. (b) FQ vs. KL Divergence on TOFU-10% (from πref _ref on fD_f), demonstrating DiPO’s higher unlearning efficiency. (c) Return Difference and FQ on TOFU-10%, illustrating the correlation between DiPO’s learned reward signals and unlearning efficacy. Table 5: Ablation results. The value of each metric is averaged over five seeds at the best epoch. Best results are in bold. Method TOFU-1% TOFU-5% TOFU-10% FQ MU FQ MU FQ MU Original 1e-3 0.62 3e-16 0.62 2e-19 0.62 Retrain 1.0 0.62 1.0 0.62 1.0 0.62 DiPO (ours) 0.89 0.58 0.95 0.58 0.84 0.56 DiPO(f)+GD 0.57 0.62 0.54 0.62 3e-5 0.65 GA+DiPO(r) 0.16 0.39 1e-13 0.59 3e-10 0.38 NPO+DiPO(r) 0.12 0.55 0.07 0.01 3e-2 4e-3 Ablation Studies We investigate the interplay of DiPO’s core ℒDiPO-fL_DiPO-f and ℒDiPO-rL_DiPO-r in Table˜5. Our main DiPO (using both ℒDiPO-fL_DiPO-f and ℒDiPO-rL_DiPO-r) is compared against variants where one DiPO component is substituted with another loss, detailed in Section˜D.5.1. The results compellingly show that while ℒGDL_GD can significantly boost MU, the effective trade-off between FQ and MU is achieved only with our main DiPO configuration. This underscores that DiPO’s strength lies in its integrated, preference-based design for both forget and retain objectives. Furthermore, as detailed in Figure˜4, we analyze the performance of using only ℒDiPO-fL_DiPO-f, and find it achieves effective unlearning while maintaining a degree of MU. This is a significant advantage over typical baselines relying solely on forget loss (such as GA and NPO), which tend to exhibit a collapse in both metrics. This finding highlights the inherent robustness and targeted nature of the DiPO forget mechanism itself, even in the absence of an explicit retain objective. 6 Conclusion In this paper, we propose the distribution-level for LLM unlearning, a fine-grained perspective which can overcome the limitations of response-level approaches. Building upon this, we derive a novel algorithm, Distribution Preference Optimization (DiPO), along with an intrinsic method for constructing complete preference distribution pairs directly from model logits. This provides precise guidance for the unlearning process without requiring auxiliary models or domain-specific knowledge, thereby enhancing its generalizability. Both theoretical analysis and extensive experimental results demonstrate the effectiveness and stability of our method. References Yu et al. [2023] Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6032–6048, 2023. Liu et al. [2022] Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR, 2022. Eldan and Russinovich [2023] Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning for llms. 2023. Jang et al. [2022] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022. Wu et al. [2023a] Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138, 2023a. Liu et al. [2024a] Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058, 2024a. Barrett et al. [2023] Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. Identifying and mitigating the security risks of generative ai. Foundations and Trends® in Privacy and Security, 6(1):1–52, 2023. European Parliament and Council of the European Union [2016] European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union, April 2016. Published in OJ L 119, 4.5.2016, p. 1–88. Adopted 27 April 2016. California State Assembly [2018] California State Assembly. Assembly bill no. 375 (Chapter 55, statutes of 2018). an act to add title 1.81.5 (commencing with section 1798.100) to part 4 of division 3 of the civil code, relating to privacy. (California Consumer Privacy Act of 2018). California Legislature, 2017–2018 Regular Session, June 2018. Approved by Governor and filed with Secretary of State June 28, 2018. This bill enacted the CCPA. Yao et al. [2024a] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 37:105425–105475, 2024a. Zhang et al. [2024a] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024a. Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. Maini et al. [2024] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024. Shi et al. [2025] Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine unlearning six-way evaluation for language models. In The Thirteenth International Conference on Learning Representations, 2025. Nguyen et al. [2022] Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299, 2022. Cao and Yang [2015] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015. Thudi et al. [2022] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. IEEE, 2022. Izzo et al. [2021] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008–2016. PMLR, 2021. Wang et al. [2023a] Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. Kga: A general machine unlearning framework based on knowledge gap alignment. arXiv preprint arXiv:2305.06535, 2023a. Triantafillou et al. [2024] Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. Are we making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406.09073, 2024. Golatkar et al. [2020] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9304–9312, 2020. Bourtoule et al. [2021] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pages 141–159. IEEE, 2021. Jia et al. [2023] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584–51605, 2023. Fan et al. [2023] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint arXiv:2310.12508, 2023. Kurmanji et al. [2023] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in neural information processing systems, 36:1957–1987, 2023. Ginart et al. [2019] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019. Gandikota et al. [2023] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426–2436, 2023. Zhang et al. [2024b] Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1755–1764, 2024b. Che et al. [2023] Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan, Dejing Dou, and Jun Huan. Fast federated machine unlearning with nonlinear functional theory. In International conference on machine learning, pages 4241–4268. PMLR, 2023. Pan et al. [2025] Zibin Pan, Zhichao Wang, Chi Li, Kaiyan Zheng, Boqi Wang, Xiaoying Tang, and Junhua Zhao. Federated unlearning with gradient descent and conflict mitigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19804–19812, 2025. Chien et al. [2022] Eli Chien, Chao Pan, and Olgica Milenkovic. Efficient model updates for approximate unlearning of graph-structured data. In The Eleventh International Conference on Learning Representations, 2022. Wu et al. [2023b] Kun Wu, Jie Shen, Yue Ning, Ting Wang, and Wendy Hui Wang. Certified edge unlearning for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2606–2617, 2023b. Sachdeva et al. [2024] Bhavika Sachdeva, Harshita Rathee, Sristi, Arun Sharma, and Witold Wydmański. Machine unlearning for recommendation systems: An insight. In International Conference On Innovative Computing And Communication, pages 415–430. Springer, 2024. Fan et al. [2024] Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163, 2024. Mekala et al. [2024] Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, and Elita Lobo. Alternate preference optimization for unlearning factual knowledge in large language models. arXiv preprint arXiv:2409.13474, 2024. Jia et al. [2024] Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239, 2024. Chundawat et al. [2023] Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7210–7217, 2023. Ji et al. [2024] Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Kompella, Sijia Liu, and Shiyu Chang. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. Advances in Neural Information Processing Systems, 37:12581–12611, 2024. Chen and Yang [2023] Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. arXiv preprint arXiv:2310.20150, 2023. Yao et al. [2024b] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402.15159, 2024b. Ishibashi and Shimodaira [2023] Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852, 2023. Liu et al. [2024b] Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Revisiting who’s harry potter: Towards targeted unlearning from a causal intervention perspective. arXiv preprint arXiv:2407.16997, 2024b. Thaker et al. [2024] Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329, 2024. Pawelczyk et al. [2023] Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023. Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Hong et al. [2024] Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024. Meng et al. [2024] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024. Wang et al. [2023b] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240, 2023b. Azar et al. [2024] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024. Sun et al. [2024] Haoyuan Sun, Yuxin Zheng, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Generalizing offline alignment theoretical paradigm with diverse divergence constraints. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024. Zeng et al. [2024] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999, 2024. Li et al. [2022] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022. Chuang et al. [2023] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883, 2023. Liu et al. [2021] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.03023, 2021. Appendix A Limitations Despite DiPO demonstrating strong unlearning capabilities, certain limitations warrant discussion. First, similar to other current unlearning methods, DiPO’s outputs are not entirely immune to hallucination, reflecting an ongoing challenge in the field. Second, while our intrinsic mechanism for constructing preference pairs is effective and general, its current simplicity may not fully address the complexities required for unlearning against information leakage, such as those evaluated by Membership Inference Attacks (MIAs). This is indicated by DiPO’s performance on challenging privacy-related metrics, like the PrivLeak scores in the MUSE benchmark, where more sophisticated preference modeling might be beneficial. We plan to explore these problems in future work. Appendix B Theoretical Details B.1 Distribution-level Return Derivation In Section˜4.1 we showe the immediate reward function rπ(x,y<t)r_π(x,y^<t): rπ(x,y<t) r_π(x,y^<t) =z∼π(⋅|[x,y<t])[r([x,y<t],z)] =E_z π(·|[x,y^<t])[r([x,y^<t],z)] =z∼π(⋅|[x,y<t])[βlogπθ∗(z|[x,y<t])πref(z|[x,y<t])+βDKL(πref(⋅|[x,y<t])||πθ∗(⋅|[x,y<t]))] =E_z π(·|[x,y^<t])[β _θ^*(z|[x,y^<t]) _ref(z|[x,y^<t])+β D_KL( _ref(·|[x,y^<t])|| _θ^*(·|[x,y^<t]))] =βz∼π(⋅|[x,y<t])[logπθ∗(z|[x,y<t])πref(z|[x,y<t])]+βDKL(πref(⋅|[x,y<t])||πθ∗(⋅|[x,y<t])) = _z π(·|[x,y^<t]) [ _θ^*(z|[x,y^<t]) _ref(z|[x,y^<t]) ]+β D_KL( _ref(·|[x,y^<t])|| _θ^*(·|[x,y^<t])) Using the definition of KL divergence, the expectation term can be rewritten as: z∼π(⋅|[x,y<t])[logπθ∗(z|[x,y<t])πref(z|[x,y<t])] _z π(·|[x,y^<t]) [ _θ^*(z|[x,y^<t]) _ref(z|[x,y^<t]) ] =z∼π(⋅|[x,y<t])[logπθ∗(z|[x,y<t])π(z|[x,y<t])⋅π(z|[x,y<t])πref(z|[x,y<t])] =E_z π(·|[x,y^<t]) [ _θ^*(z|[x,y^<t])π(z|[x,y^<t])· π(z|[x,y^<t]) _ref(z|[x,y^<t]) ] =z∼π(⋅|[x,y<t])[logπ(z|[x,y<t])πref(z|[x,y<t])]−z∼π(⋅|[x,y<t])[logπ(z|[x,y<t])πθ∗(z|[x,y<t])] =E_z π(·|[x,y^<t]) [ π(z|[x,y^<t]) _ref(z|[x,y^<t]) ]-E_z π(·|[x,y^<t]) [ π(z|[x,y^<t]) _θ^*(z|[x,y^<t]) ] =DKL(π(⋅|[x,y<t])||πref(⋅|[x,y<t]))−DKL(π(⋅|[x,y<t])||πθ∗(⋅|[x,y<t])). =D_KL(π(·|[x,y^<t])|| _ref(·|[x,y^<t]))-D_KL(π(·|[x,y^<t])|| _θ^*(·|[x,y^<t])). For a response y (i.e. a specific trajectory in RL), we can calculate the return Rπ(x,y)R_π(x,y) as follows: Rπ(x,y) R_π(x,y) =∑t=1Trπ(x,y<t) = _t=1^Tr_π(x,y^<t) =∑t=1TβDKL(π(⋅|[x,y<t])||πref(⋅|[x,y<t])) = _t=1^Tβ D_KL(π(·|[x,y^<t])|| _ref(·|[x,y^<t])) −β∑t=1TDKL(π(⋅|[x,y<t])||πθ∗(⋅|[x,y<t]))+∑t=1TβDKL(πref(⋅|[x,y<t])||πθ∗(⋅|[x,y<t])). -β _t=1^TD_KL(π(·|[x,y^<t])|| _θ^*(·|[x,y^<t]))+ _t=1^Tβ D_KL( _ref(·|[x,y^<t])|| _θ^*(·|[x,y^<t])). This is the formula in Equation˜13. B.2 Detailed proof of DiPO loss Recall from Equation˜13 that the distribution-level return is: Rπ(x,y,πθ∗)≔Rπ(x,y)=βDSeqKL(x,y;π||πref)−βDSeqKL(x,y;π||πθ∗)+βDSeqKL(x,y;πref||πθ∗).R_π(x,y, _θ^*) R_π(x,y)=β D_SeqKL(x,y;π|| _ref)-β D_SeqKL(x,y;π|| _θ^*)+β D_SeqKL(x,y; _ref|| _θ^*). Given a specific sample (x,y)(x,y) and a pair of preference distributions (πw,πl)( _w, _l), we can derive their respective return expressions: Rπw(x,y,πθ∗) R_ _w(x,y, _θ^*) =βDSeqKL(x,y;πw||πref)−βDSeqKL(x,y;πw||πθ∗)+βDSeqKL(x,y;πref||πθ∗), =β D_SeqKL(x,y; _w|| _ref)-β D_SeqKL(x,y; _w|| _θ^*)+β D_SeqKL(x,y; _ref|| _θ^*), (23) Rπl(x,y,πθ∗) R_ _l(x,y, _θ^*) =βDSeqKL(x,y;πl||πref)−βDSeqKL(x,y;πl||πθ∗)+βDSeqKL(x,y;πref||πθ∗). =β D_SeqKL(x,y; _l|| _ref)-β D_SeqKL(x,y; _l|| _θ^*)+β D_SeqKL(x,y; _ref|| _θ^*). (24) These respectively represent the degree of preference for response y under different policies. Consequently, we can employ BT model to construct the preference model: p∗(Rπw≻Rπl|(x,y)) p^*(R_ _w R_ _l|(x,y)) =exp(Rπw(x,y,πθ∗))exp(Rπw(x,y,πθ∗))+exp(Rπl(x,y,πθ∗)) = (R_ _w(x,y, _θ^*)) (R_ _w(x,y, _θ^*))+ (R_ _l(x,y, _θ^*)) =11+exp(Rπl(x,y,πθ∗)−Rπw(x,y,πθ∗)). = 11+ (R_ _l(x,y, _θ^*)-R_ _w(x,y, _θ^*)). (25) Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parametrized policy πθ _θ. Similar to the DPO method, our policy objective becomes: ℒDiPO(πθ;πw,πl,πref) _DiPO( _θ; _w, _l, _ref) =−(x,y)∼[logp(Rπw≻Rπl|(x,y))] =-E_(x,y) [ p(R_ _w R_ _l|(x,y)) ] =−(x,y)∼[log11+exp(Rπl(x,y,πθ)−Rπw(x,y,πθ))] =-E_(x,y) [ 11+ (R_ _l(x,y, _θ)-R_ _w(x,y, _θ)) ] =−(x,y)∼[logσ((Rπw(x,y,πθ)−Rπl(x,y,πθ)))] =-E_(x,y) [ σ ( (R_ _w(x,y, _θ)-R_ _l(x,y, _θ) ) ) ] =−(x,y)∼[logσ((βDSeqKL(x,y;πw||πref)−βDSeqKL(x,y;πw||πθ∗)+βDSeqKL(x,y;πref||πθ∗) =-E_(x,y) [ σ ( (β D_SeqKL(x,y; _w|| _ref)-β D_SeqKL(x,y; _w|| _θ^*)+β D_SeqKL(x,y; _ref|| _θ^*) . . −(βDSeqKL(x,y;πl||πref)−βDSeqKL(x,y;πl||πθ∗)+βDSeqKL(x,y;πref||πθ∗))))] . .- (β D_SeqKL(x,y; _l|| _ref)-β D_SeqKL(x,y; _l|| _θ^*)+β D_SeqKL(x,y; _ref|| _θ^*) ) ) ) ] =−(x,y)∼[logσ(β(DSeqKL(x,y;πl||πθ)−DSeqKL(x,y;πw||πθ)) =-E_(x,y) [ σ (β (D_SeqKL(x,y; _l|| _θ)-D_SeqKL(x,y; _w|| _θ) ) . . +β(DSeqKL(x,y;πw||πref)−DSeqKL(x,y;πl||πref)))]. . .+β (D_SeqKL(x,y; _w|| _ref)-D_SeqKL(x,y; _l|| _ref) ) ) ]. (26) Now that we have the loss function of DiPO. B.3 Pseudo-code of DiPO Algorithm 1 Distribution Preference Optimization (DiPO) 1:Input: Datasets f,rD_f,D_r, Reference model πref _ref, Policy model πθ _θ, βf,βr,η,λ,p _f, _r,η,λ,p 2:Initialize: θ←θrefθ← _ref 3:for each training epoch do 4: Sample mini-batches Bf∼fB_f _f, Br∼rB_r _r 5: Generate approx. πf(⋅|xf,yf<t),πm(⋅|xf,yf<t) _f(·|x_f,y_f^<t), _m(·|x_f,y_f^<t) from πθ _θ for (xf,yf)∈Bf(x_f,y_f)∈ B_f via top-p logit filtering 6: Generate approx. πf(⋅|xr,yr<t),πm(⋅|xr,yr<t) _f(·|x_r,y_r^<t), _m(·|x_r,y_r^<t) from πθ _θ for (xr,yr)∈Br(x_r,y_r)∈ B_r via top-p logit filtering 7: Compute forget loss ℒDiPO-fL_DiPO-f on BfB_f using πw=πf,πl=πm _w= _f, _l= _m ⊳ Based on Eq. 20 8: Compute retain loss ℒDiPO-rL_DiPO-r on BrB_r using πw=πm,πl=πf _w= _m, _l= _f ⊳ Based on Eq. 21 9: Compute total loss ℒ(θ)=ℒDiPO-f+λℒDiPO-rL(θ)=L_DiPO-f+ _DiPO-r ⊳ Using Eq. 22 10: Update parameters θ←θ−η∇θℒ(θ)θ←θ-η _θL(θ) 11:end for 12:Output: Unlearned policy model πθ _θ Appendix C Baseline Methods This section details the baseline methods used for comparison in our experiments. We categorize them into optimization-based methods, which are the primary focus of comparison for our DiPO method, and other unlearning frameworks represented by a state-of-the-art method. C.1 Optimization-based method Optimization-based methods directly modify the model parameters by minimizing a combined objective function, typically structured as ℒ(θ)=ℒr(θ)+λℒf(θ)L(θ)=L_r(θ)+ _f(θ), where ℒfL_f promotes forgetting and ℒrL_r encourages retention, balanced by λ. We describe common choices for these loss components below. C.1.1 Forget losses Gradient ascent loss ℒGAL_GA is a fundamental and intuitive unlearning loss function [17, 13] that aims to maximize the next-token prediction loss on the forget set fD_f, which is equivalent to minimizing the likelihood of correct predictions. We denote this forget loss as: ℒGA(θ)=(xf,yf)∼f[logπθ(yf|xf)].L_GA(θ)=E_(x_f,y_f) _f[ _θ(y_f|x_f)]. (27) While intuitive, ℒGAL_GA is unbounded below (likelihood can approach zero), which can lead to training instability and model degradation. Direct preference optimization loss ℒDPOL_DPO adapts the Direct Preference Optimization framework [12] for unlearning [11] (distinguish from standard DPO). It requires a dataset of simple, template-based alternative responses aD_a (e.g. yidk=y_idk= “I don’t know”) and formulates the forget loss to prefer yidky_idk over the original forget response yfy_f: ℒDPO(θ)=−1β(xf,yf)∼f,yidk∼a[logσ(βπθ(yidk|xf)πref(yidk|xf)−βπθ(yf|xf)πref(yf|xf))].L_DPO(θ)=- 1βE_(x_f,y_f) _f,y_idk _a[ σ(β _θ(y_idk|x_f) _ref(y_idk|x_f)-β _θ(y_f|x_f) _ref(y_f|x_f))]. (28) where σ(⋅)σ(·) is the sigmoid function, β is a hyper-parameter controlling the preference strength, and πref _ref is reference model (often the initial model before unlearning). This loss is bounded but can suffer from catastrophic forgetting, excessively favoring yidky_idk even for retain queries. Negative preference optimization loss ℒNPOL_NPO is a variant of ℒDPOL_DPO for unlearning in recent work [11]. NPO focuses solely on penalizing the forget responses yfy_f by treating them as dispreferred, without requiring preferred alternatives yidky_idk. Its forget loss term is: ℒNPO(θ)=−2β(xf,yf)∼f[logσ(−βπθ(yf|xf)πref(yf|xf))],L_NPO(θ)=- 2βE_(x_f,y_f) _f[ σ(-β _θ(y_f|x_f) _ref(y_f|x_f))], (29) NPO avoids the unboundedness of ℒGAL_GA and the need for yidky_idk in ℒDPOL_DPO, but lacks an explicit positive preference signal. C.1.2 Retain losses Gradient descent loss ℒGDL_GD is the standard negative log-likelihood loss applied to the retain set rD_r [13, 11], encouraging the model to maintain its predictive performance: ℒGD(θ)=(xr,yr)∼r[−logπθ(yr|xr)].L_GD(θ)=E_(x_r,y_r) _r[- _θ(y_r|x_r)]. (30) The combination of ℒGAL_GA as ℒfL_f and ℒGDL_GD as ℒrL_r constitutes the GradDiff method [2, 10, 13]. KL-divergence loss ℒKLL_KL aims to preserve the model’s behavior by minimizing the KL divergence between the current model πθ _θ and reference model πref _ref over the retain set [13, 11]: ℒKL(θ)=(xr,yr)∼r[DKL(πθ(⋅|xr)||πref(⋅|xr))].L_KL(θ)=E_(x_r,y_r) _r[D_KL( _θ(·|x_r)|| _ref(·|x_r))]. (31) C.2 Other Unlearning Framework Beyond optimization-based fine-tuning, alternative unlearning paradigms exist that employ different mechanisms, such as auxiliary models, data manipulation techniques (see Section˜2). To provide context against strong baselines from distinct research directions within these paradigms, we include two representative methods: ULD [38] and AltPO [35]. ULD exemplifies methods that achieve unlearning without direct fine-tuning of the target model’s parameters, instead relying on an auxiliary model and logit manipulation, representing a strong baseline for non-optimization-based unlearning frameworks. AltPO, on the other hand, showcases a hybrid approach combined DPO-style losses with data-based techniques. ULD This method trains an auxiliary LLM on augmented versions of the forget and retain sets (f′D_f and r′D_r , respectively) to perform the inverse unlearning task. Specifically, the auxiliary model is trained to maximize likelihood on f′D_f (memorizing) while driving its output distribution towards uniform on r′D_r (forgetting). The final unlearned model’s logits are obtained by subtracting the auxiliary model’s logits from the original target model’s logits. This approach differs significantly from fine-tuning methods and is particularly noted for its effectiveness in preserving model utility while achieving strong unlearning performance, thus offering a valuable comparison point from a distinct unlearning strategy. AltPO This method also employs an auxiliary model, guided by carefully designed prompts, to generate a privacy-preserving alternative response yfay_f_a for each sample in the forget set fD_f. This yfay_f_a then replaces the template-based response yidky_idk in Equation˜28, mitigating catastrophic forgetting. Following its original paper [35], the forget loss is denoted as: ℒAltPO(θ)=−2β(xf,yf)∼f,yfa∼a[logσ(βπθ(yfa|xf)πref(yfa|xf)−βπθ(yf|xf)πref(yf|xf))].L_AltPO(θ)=- 2βE_(x_f,y_f) _f,y_f_a _a[ σ(β _θ(y_f_a|x_f) _ref(y_f_a|x_f)-β _θ(y_f|x_f) _ref(y_f|x_f))]. (32) Similarly, AltPO utilizes ℒGDL_GD as its retain loss. Due to the use of an auxiliary model to obtain alternative responses and thereby augment the dataset, it is not classified as a purely optimization-based method but rather as a hybrid approach combined with data-based techniques. We include it for comparison against our method, viewing it as a more advanced development compared to NPO, particularly in its provision of an explicit, generated positive preference. Appendix D Experiments Details D.1 Hardware configuration All experiments are conducted on 2 NVIDIA A800-SXM4-80GB GPU cards in a single node. We employ DeepSpeed ZeRO stage-2 for all baselines to compress GPU memory. A typical experimental run for our main DiPO method on benchmarks like TOFU or MUSE, involving 10 epochs of training with evaluation performed after each epoch, took approximately 1 hour on this hardware setup. For our main DiPO method, a complete experimental run on a single task within the MUSE or TOFU benchmarks (typically involving 10 epochs of training with evaluation after each epoch) was generally completed within 1 hour on this hardware setup. D.2 Details on Filter mechanism This appendix clarifies the top-k filtering strategy (using rate pkp_k) mentioned in Section˜4.3. This top-k filtering strategy represents a mechanism for manipulating model output logits, previously employed in various generation contexts [52, 53, 54]. Similar to its adoption in related unlearning frameworks like ULD [38], we utilize it here to determine which tokens’ original logits tz_t contribute to the memory vector tm_t. In this section, we will provide a more formal definition. Let St⊂VS_t⊂ V be the set of tokens selected by top-k filtering, keeping the top pkp_k tokens of the vocabulary size. We define a ‘memory vector’ tm_t that isolates the logits corresponding to these high-confidence tokens: t=t⊙mask(t,St)m_t=z_t (z_t,S_t), where mask(t,St)mask(z_t,S_t) is a binary vector selecting tokens in StS_t. We then construct the memory-enhancing distribution πm _m and the forgetting-promoting distribution πf _f by adding or subtracting this memory vector, scaled by a factor α: πm(⋅|x,y<t)=softmax(t+αt),πf(⋅|x,y<t)=softmax(t−αt). _m(·|x,y^<t)=softmax(z_t+ _t), _f(·|x,y^<t)=softmax(z_t- _t). (33) To determine the set StS_t, we first compute log-probabilities t=log_softmax(t)s_t=log\_softmax(z_t). A dynamic threshold τ is then established by considering two criteria: 1. Rank-based Threshold (τk _k): This ensures at least a minimum number of tokens are kept. It is set to the log-probability corresponding to the k-th rank when tokens are sorted by log-probability in descending order, where k=max(1,⌊pk⋅|V|⌋)k= (1, p_k·|V| ). 2. Relative Threshold (τrel _rel): This adapts to the sharpness of the distribution and is calculated relative to the maximum log-probability: τrel=max(t)+log(pk) _rel= (s_t)+ (p_k). The final threshold used for filtering is the minimum of these two: τ=min(τk,τrel)τ= ( _k, _rel). The set StS_t then comprises all tokens whose log-probability is greater than or equal to this final threshold (St=i∣st,i≥τS_t=\i s_t,i≥τ\). This ensures that only the logits of these high-confidence tokens are isolated in the memory vector t=t⊙mask(t,St)m_t=z_t (z_t,S_t). In this paper, we set pk=0.05p_k=0.05. D.3 Implementation Details on TOFU D.3.1 Descriptions of the dataset TOFU focuses on unlearning the knowledge of fictitious authors. It contains 200 fictitious author profiles, each consisting of 20 question-answer pairs generated by GPT-4 based on some predefined attributes. These profiles are fictitious and do not exist in the pre-training data, providing a controlled environment for studying unlearning LLMs. TOFU contains three Forget set fD_f configurations, each with 1%, 5%, and 10% of the fictional authors, referred to as TOFU-1%, TOFU-5%, and TOFU-10%, respectively. The remaining data constitutes the Retain set rD_r, used to assess the model’s preservation of non-targeted knowledge after unlearning. To further examine unlearning’s impact on overall capabilities, TOFU includes two additional evaluation subsets: the Real Authors set RAD_RA, for performance on real-world information conceptually related to fD_f but not part of fine-tuning, and the World Facts set WFD_WF, for assessing general world knowledge. Table 6: Data statistics of Forget set fD_f, Retain set rD_r, Real Authors set RAD_RA and World Facts set WFD_WF. Task fD_f rD_r RAD_RA WFD_WF TOFU-1% 40 400 100 117 TOFU-5% 200 400 100 117 TOFU-10% 400 400 100 117 D.3.2 Evaluation Metrics Our evaluation centers on two primary metrics in the original TOFU paper [13]: Model Utility (MU) and Forget Quality (FQ). Model Utility (MU) This metric quantifies the side effects of unlearning on the model’s general knowledge and capabilities. It aggregates performance on the Retain, Real Authors, and Real World sets, considering answer generation probability, ROUGE-L similarity, and Truth Ratio. The Truth Ratio RtruthR_truth assesses the model’s ability to distinguish factual information, defined as the propensity to generate a paraphrased correct answer (a~ a) versus a set of structurally similar but incorrect perturbed answers (a^i a_i) for a given question (q): Rtruth:=15∑i=15ℙ(a^i|q)1/|a^i|ℙ(a~|q)1/|a~|.R_truth:= 15 _i=1^5P( a_i|q)^1/| a_i|P( a|q)^1/| a|. Here, q is the input question, P(⋅|q)P(·|q) is the model’s probability for a specific answer, |⋅||·| denotes answer length in tokens, and N is the number of perturbed answers. MU is the harmonic mean of these three sub-metrics across the three evaluation datasets (nine scores total), a method sensitive to any single low score. Forget Quality (FQ) This metric evaluates the success of erasing targeted information fD_f. It compares the unlearned model’s behavior to that of an ideal reference model (typically trained only on rD_r and thus unexposed to fD_f) when queried about fD_f. The assessment uses a two-sample Kolmogorov-Smirnov (KS) test on the Truth Ratio distributions from these two models on fD_f. A high p-value (e.g. >0.05) indicates no significant distributional difference, suggesting effective unlearning. D.3.3 Hyperparameter Implementation Following the setup of [13], We use the fine-tuned LLama2-chat-7B released by TOFU as the original LLM and fine-tune the target LLM for 10 epochs. For all baseline methods and ours, we set the batch size and learning rate to 32 and 1e−51e-5 following previous works. We set β in Equation˜20 (βf _f) and Equation˜21 (βr _r) to 0.05 in our method. For all baseline methods involving retain loss, we set the weight λ to 1. More details are in Section˜D.3. D.4 Implementation Details on MUSE D.4.1 Descriptions of the dataset MUSE proposes a multi-faceted framework considering six desirable properties, catering to both data owner and model deployer expectations. In this paper, we focus on the News corpus. For this corpus, distinct Forget Sets (forgetD_forget), Retain Sets (retainD_retain), and disjoint hold-out sets (holdoutD_holdout) are established as disjoint collections of news articles. To facilitate granular evaluation, two types of data are derived from these news articles: 1. Verbatim text: Original text excerpts from news articles used to assess the prevention of verbatim memorization. 2. Knowledge set: Question-answer (QA) pairs derived from the original news texts to evaluate the removal of factual knowledge. D.4.2 Evaluation Metrics MUSE evaluates unlearning across six criteria. We highlight key metrics reflecting data owner and deployer concerns as applied to the NEWS corpus: Data Owner Focused Metrics 1. No Verbatim Memorization (VerbMem-f): Assesses if the unlearned model (funlearnf_unlearn) avoids reproducing exact text sequences from the forgetD_forget of news articles. Quantified by VerbMem-f, which measures the ROUGE-L F1 score between model-generated continuations and true continuations from forgetD_forget. VerbMem-f(f,forget):=1|forget|∑x∈forgetROUGE-L(f(x[:l]),x[l+1:]).VerbMem-f(f,D_forget):= 1|D_forget| _x _forgetROUGE-L(f(x_[:l]),x_[l+1:]). 2. No Knowledge Memorization (KnowMem-f): Measures if funlearnf_unlearn can no longer answer questions whose answers are exclusively found in the forgetD_forget of news articles. Quantified by KnowMem-f, averaging ROUGE scores between model answers and ground-truth answers for QA pairs derived from forgetD_forget. 3. No Privacy Leakage (PrivLeak): Evaluates if the inclusion of news articles from forgetD_forget in the original training data (trainD_train) can be inferred from funlearnf_unlearn. Measured by PrivLeak, which compares the Area Under the ROC Curve (AUC) of a Membership Inference Attack (MIA) on funlearnf_unlearn against that on a perfectly retrained model (fretrainf_retrain), discriminating between forgetD_forget (member news articles) and holdoutD_holdout (non-member news articles). PrivLeak:=AUC(funlearn;forget,holdout)−AUC(fretrain;forget,holdout)AUC(fretrain;forget,holdout).PrivLeak:= AUC(f_unlearn;D_forget,D_holdout)-AUC(f_retrain;D_forget,D_holdout)AUC(f_retrain;D_forget,D_holdout). A PrivLeak score close to zero is desirable. Deployer Focused Metrics 1. Utility Preservation (KnowMem-r): Quantifies how well funlearnf_unlearn maintains its performance on the retainD_retain of news articles. This is typically measured using the KnowMem-r metric applied to retainD_retain: KnowMem-r(funlearn,retain)KnowMem-r(f_unlearn,D_retain). 2. Scalability: Assesses how unlearning methods perform with increasing sizes of forgetD_forget within the NEWS corpus. 3. Sustainability: Evaluates performance under sequential unlearning requests involving different sets of news articles. D.4.3 Hyperparameter Implementation Following the setup of MUSE [14], we use LLaMA-2 7B as the original model, which was released before the collected BBC news articles to prevent potential data leakage. For baseline methods, we set the batch size to 32, and fine-tune for 5 epochs using AdamW optimizer with a constant learning rate of 1e−51e-5, For our method, we use the same training hyper-parameters as described in TOFU. D.5 Additional Results on TOFU D.5.1 Details on Ablation Study To determine the optimal configuration for our DiPO method, we conducte an ablation study comparing different retain loss functions when combined with the DiPO forget loss component, ℒDiPO-f(θ)L_DiPO-f(θ) (defined in Equation˜20). We evaluate the following configurations on the TOFU-10% task at the best-epoch: 1. DiPO (ours): This is the configuration presented as our main result in the paper, using the ℒDiPO-rL_DiPO-r by reversing the roles of the preference distributions of ℒDiPO-fL_DiPO-f on the retain set. The combined objective is then expressed as ℒ=ℒDiPO-f(θ)+λℒDiPO-r(θ)L=L_DiPO-f(θ)+ _DiPO-r(θ). 2. DiPO(f)+GD: This configuration utilizes the standard Gradient Descent loss Equation˜30 on the retain set: minθℒ(θ) _θL(θ) =minθ(ℒDiPO-f(θ)+γℒGD(θ)) = _θ (L_DiPO-f(θ)+ _GD(θ) ) =minθ(ℒDiPO-f(θ)+λ(xr,yr)∼r[−logπθ(yr|xr)]). = _θ (L_DiPO-f(θ)+ _(x_r,y_r) _r[- _θ(y_r|x_r)] ). 3. GA+DiPO(r): This configuration utilizes the standard Gradient Descent loss Equation˜30 on the retain set: minθℒ(θ) _θL(θ) =minθ(ℒGA(θ)+λℒDiPO-r(θ)) = _θ (L_GA(θ)+ _DiPO-r(θ) ) =minθ((xf,yf)∼f[logπθ(yf|xf)]+λℒDiPO-r(θ)). = _θ (E_(x_f,y_f) _f[ _θ(y_f|x_f)]+ _DiPO-r(θ) ). 4. NPO+DiPO(r): This configuration utilizes the standard Gradient Descent loss Equation˜30 on the retain set: minθℒ(θ) _θL(θ) =minθ(ℒNPO(θ)+λℒDiPO-r(θ)) = _θ (L_NPO(θ)+ _DiPO-r(θ) ) Figure 4: Training curves for only-forget configuration on TOFU-10%, with GA and NPO curves additionally included for comparison. For these settings, the final results are presented in Table˜5. Additionally, we discuss DiPO-Forget (using only ℒDiPO-fL_DiPO-f without any retain loss). This setup simulates scenarios where retain data might be unavailable. we set learning rate to 7e−67e-6 and β to 0.5 in this configuration. As illustrated by its training dynamics on TOFU-10% (Figure˜4), even without an explicit retain loss, DiPO-Forget achieves substantial unlearning (e.g. FQ reaching approximately 0.51) while maintaining a notable degree of model utility (e.g. MU around 0.18 at the end of training, after an initial drop). This contrasts sharply with typical baselines where removing the retain loss often leads to a near-complete collapse in both MU and FQ. The ability of DiPO-Forget to preserve some utility while effectively unlearning underscores the inherent stability and targeted nature of the DiPO forget mechanism. This finding is particularly promising for unlearning scenarios where access to comprehensive retain data is limited or unavailable. D.5.2 Results at the Final Epoch Table 7: The final-epoch performance averaged over five seeds on TOFU benchmark. Scores closer to “Retrain” are better. Bold indicates best results among all methods. Method TOFU-1% TOFU-5% TOFU-10% FQ ↑ MU ↑ FQ ↑ MU ↑ FQ ↑ MU ↑ Original LLM 1e-3 0.62 3e-16 0.62 2e-19 0.62 Retrain LLM 1.0 0.62 1.0 0.62 1.0 0.62 GA 0.40 0.52 5e-8 0 6e-11 0 GA+GD 0.27 0.53 0.11 0.33 9e-3 0.51 GA+KL 0.31 0.53 0.14 0.35 1e-5 0.55 NPO 0.71 0.56 0.03 0.02 5e-4 0 DPO+GD 0.27 0.58 1e-4 0.02 5e-7 0 NPO+GD 0.73 0.58 0.64 0.57 0.17 0.53 DiPO 0.89 0.58 0.95 0.58 0.84 0.56