Paper deep dive

Bridging the Gap Between Preference Alignment and Machine Unlearning

Xiaohua Feng, Yuyuan Li, Huwei Ji, Jiaming Zhang, Li Zhang, Tianyu Du, Chaochao Chen

Year: 2025Venue: arXiv preprintArea: Alignment TrainingType: EmpiricalEmbeddings: 144

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:22:35 PM

Summary

The paper introduces 'Unlearning to Align' (U2A), a bi-level optimization framework that bridges the gap between Machine Unlearning (MU) and Preference Alignment (PA) in Large Language Models. By quantifying the impact of unlearning specific negative examples on PA performance, the authors demonstrate that selective unlearning, rather than indiscriminate removal, maximizes alignment efficiency and effectiveness.

Entities (5)

Machine Unlearning · technique · 99%Preference Alignment · technique · 99%Reinforcement Learning with Human Feedback · technique · 98%Unlearning to Align · framework · 98%Bi-level Optimization · methodology · 95%

Relation Signals (3)

U2A → utilizes → Bi-level Optimization

confidence 98% · we propose a framework called Unlearning to Align (U2A), which leverages bi-level optimization

U2A → optimizes → Preference Alignment

confidence 95% · U2A... to efficiently select and unlearn examples for optimal PA performance.

Machine Unlearning → integrateswith → Preference Alignment

confidence 92% · This framework bridges the gap between MU and PA, offering a systematic approach to their integration.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like Reinforcement Learning with Human Feedback (RLHF) face notable challenges. These approaches require high-quality datasets of positive preference examples, which are costly to obtain and computationally intensive due to training instability, limiting their use in low-resource scenarios. LLM unlearning technique presents a promising alternative, by directly removing the influence of negative examples. However, current research has primarily focused on empirical validation, lacking systematic quantitative analysis. To bridge this gap, we propose a framework to explore the relationship between PA and LLM unlearning. Specifically, we introduce a bi-level optimization-based method to quantify the impact of unlearning specific negative examples on PA performance. Our analysis reveals that not all negative examples contribute equally to alignment improvement when unlearned, and the effect varies significantly across examples. Building on this insight, we pose a crucial question: how can we optimally select and weight negative examples for unlearning to maximize PA performance? To answer this, we propose a framework called Unlearning to Align (U2A), which leverages bi-level optimization to efficiently select and unlearn examples for optimal PA performance. We validate the proposed method through extensive experiments, with results confirming its effectiveness.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

144,045 characters extracted from source content.

Expand or collapse full text

Bridging the Gap Between Preference Alignment and Machine Unlearning Xiaohua Feng Yuyuan Li Huwei Ji Jiaming Zhang Li Zhang Tianyu Du Chaochao Chen Abstract Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like Reinforcement Learning with Human Feedback (RLHF) face notable challenges. These approaches require high-quality datasets of positive preference examples, which are costly to obtain and computationally intensive due to training instability, limiting their use in low-resource scenarios. LLM unlearning technique presents a promising alternative, by directly removing the influence of negative examples. However, current research has primarily focused on empirical validation, lacking systematic quantitative analysis. To bridge this gap, we propose a framework to explore the relationship between PA and LLM unlearning. Specifically, we introduce a bi-level optimization-based method to quantify the impact of unlearning specific negative examples on PA performance. Our analysis reveals that not all negative examples contribute equally to alignment improvement when unlearned, and the effect varies significantly across examples. Building on this insight, we pose a crucial question: how can we optimally select and weight negative examples for unlearning to maximize PA performance? To answer this, we propose a framework called Unlearning to Align (U2A), which leverages bi-level optimization to efficiently select and unlearn examples for optimal PA performance. We validate the proposed method through extensive experiments, with results confirming its effectiveness. Machine Learning, ICML 1 Introduction Despite the strong performance of Large Language Models (LLMs) in predicting the next token, their generated content often exhibits biases, factual inaccuracies, and other undesirable behaviors (Bai et al., 2022; Casper et al., 2023). Preference Alignment (PA) has been proposed to address these issues by guiding LLMs to generate responses aligned with human preferences, such as fairness and helpfulness (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020). This approach uses datasets of human-annotated preferred and non-preferred responses to optimize the model. Reinforcement Learning from Human Feedback (RLHF) is the primary method for achieving PA (Korbak et al., 2023; Wu et al., 2024; Azar et al., 2024), involving the training of a reward model on human preference data and optimizing the LLM using algorithms like Proximal Policy Optimization (PPO) (Schulman et al., 2017) or Direct Preference Optimization (DPO) (Rafailov et al., 2024). While RLHF shows strong performance across diverse applications, such as programming and creative writing, it relies on costly large-scale preference-aligned datasets, especially for positive examples (Yao et al., 2024). Additionally, RLHF training is computationally intensive and prone to instability (Liu et al., 2024b; Zhou et al., 2024), posing challenges for low-resource alignment scenarios. As a key technique aimed at protecting user privacy, Machine Unlearning (MU) in LLMs offers a novel solution to the aforementioned challenges (Liu et al., 2024c; Yao et al., 2024). This technique enables the removal of specific user data from pre-trained LLMs without requiring a complete retraining of the model. By facilitating the unlearning of negative examples, this technique promotes PA while addressing the high costs and difficulties associated with acquiring positive examples for standard RLHF. Unlike RLHF, LLM unlearning requires only negative examples, which are typically easier and cheaper to collect via mechanisms like user reports or red team testing. For unaligned pre-trained models, identifying counterexamples can be highly automated, further reducing data collection costs. Additionally, the computational overhead of unlearning is comparable to fine-tuning and significantly lower than RLHF’s full training process, making it a practical approach for achieving alignment in low-resource scenarios. Existing studies (Feng et al., 2024; Liu et al., 2024c; Yao et al., 2024) have validated the effectiveness of achieving model alignment through the unlearning of negative examples, highlighting the potential of integrating MU with PA. However, these studies primarily rely on experimental demonstrations, lacking in-depth quantitative analysis. For instance, the quantitative impact of unlearning specific samples on PA remains unclear. Additionally, critical questions such as which examples should be unlearned to maximize alignment and how to optimally select subsets of examples for unlearning to achieve the best outcomes remain unresolved. These gaps underscore a theoretical and practical disconnect between MU and PA. Addressing these challenges requires the development of a comprehensive analytical framework to unify these two domains and facilitate a deeper understanding of their intrinsic connections. To address the identified challenges, we first develop a special bi-level optimization framework to quantify how unlearning specific negative samples impacts model PA performance. In particular, the inner optimization focuses on unlearning the target sample, while the outer optimization assesses the resulting change in PA performance. After further analysis, we find that not all negative examples contribute to PA improvement, with the degree of impact varying across examples. Meanwhile, the magnitude of the impact is influenced by the unlearning weights. This suggests that indiscriminately applying unlearning to all negative examples fails to achieve optimal PA performance. To address this, we propose a framework called Unlearning to Align (U2A), based on bi-level optimization, to strategically select samples and determine optimal unlearning weights. Further convergence and computational complexity analysis indicate that our proposed method demonstrates good applicability and efficiency in LLMs. This framework bridges the gap between MU and PA, offering a systematic approach to their integration. We summarize the main contributions of this paper as follows: • We propose a special bi-level optimization framework to measure the impact of unlearning specific samples on PA performance, bridging the gap between MU and PA. • We find that unlearning all negative examples does not always benefit PA, as their contributions to PA improvement vary and can be adjusted through unlearning weights. • We propose the U2A framework, leveraging bi-level optimization to select and weight negative examples for unlearning, thereby maximizing PA performance. • We conduct extensive evaluations on multiple models and real-world datasets, and the experimental results demonstrate the effectiveness of our method. 2 Related Work 2.1 Preference Alignment PA methods can be broadly classified into learning-based and decoding-based methods, depending on whether model parameters are updated (Zhou et al., 2024). Learning-based methods (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Azar et al., 2024), such as RLHF, optimize models using preference datasets with techniques like PPO (Schulman et al., 2017), DPO (Rafailov et al., 2024), and Self-play Preference Optimization (SPO) (Wu et al., 2024). However, RLHF is computationally expensive (Rafailov et al., 2024). To mitigate this, decoding-based methods (Kim et al., 2023; Gao et al., 2023; Huang et al., 2024; Mudgal et al., 2024), which guide inference without parameter updates, have gained attention. Examples include rejection sampling (Mitchell et al., 2024; Beirami et al., 2024) and Monte Carlo Tree Search (Liu et al., 2023; Wan et al., 2024), which reduce computational costs by keeping parameters fixed. Since this study focuses on the relationship between MU and PA, and the former requires parameter updates, we primarily consider learning-based methods. 2.2 LLM unlearning The goal of LLM unlearning is to remove specific knowledge from training data while preserving the model’s performance on unrelated tasks (Jang et al., 2023; Ji et al., 2024b; Liu et al., 2024a; Feng et al., 2024). Existing methods can be categorized into three main approaches: i) Gradient-based methods (Jang et al., 2023; Maini et al., 2024) use gradient ascent on the forget set (i.e., the data to be unlearned) to remove associated knowledge, with parameter regularization added to preserve performance on other tasks. i) Preference optimization-based methods (Maini et al., 2024; Zhang et al., 2024) treat the forget set as negative examples or assign predefined responses (e.g., rejection responses) to achieve unlearning during PA. i) Model weight-based methods (Jia et al., 2024b) analyze the roles of different model modules to guide unlearning, leveraging the modularity of LLMs. As model weight-based methods are primarily used for attribution analysis, this study focuses on gradient-based and preference optimization-based approaches. 3 Preliminary Given a training set t=1,2,…,Ntsubscriptsuperscript1superscript2…superscriptsubscriptD_t=\ x^1, x^2,…, x^% N_t\Ditalic_t = italic_x1 , italic_x2 , … , italic_xitalic_Nitalic_t , where i=x1,x2,…,xnisuperscriptsubscript1subscript2…subscriptsubscript x^i=\x_1,x_2,…,x_n_i\italic_xitalic_i = x1 , x2 , … , xitalic_n start_POSTSUBSCRIPT i end_POSTSUBSCRIPT represents samples (i.e., sentences) with a token length of nisubscriptn_initalic_i, and NtsubscriptN_tNitalic_t denotes the number of samples. A model π is trained on tsubscriptD_tDitalic_t, and its optimal parameters ∗superscript θ^*italic_θ∗ satisfy the following equation: ∗superscript θ^*italic_θ∗ =arg⁢min⁡ℒNLL⁢(t;)absentsubscriptargminsubscriptℒNLLsubscript = *arg\,min_ θL_% NLL(D_t; θ)= start_OPERATOR arg min end_OPERATORitalic_θ Lroman_NLL ( Ditalic_t ; italic_θ ) =arg⁢min−i∼t⁢[∑t=1nilog⁡p⁢(xt∣<t;)],absentsubscriptargminsubscriptsimilar-tosuperscriptsubscriptdelimited-[]superscriptsubscript1subscriptconditionalsubscriptsubscriptabsent = *arg\,min_ θ-E_% x^i _t [ _t=1^n_i p(x_t % x_<t; θ) ],= start_OPERATOR arg min end_OPERATORitalic_θ - blackboard_Eitalic_xitalic_i ∼ D start_POSTSUBSCRIPT t end_POSTSUBSCRIPT [ ∑t = 1nitalic_i log p ( xitalic_t ∣ italic_x< t ; italic_θ ) ] , (1) where p⁢(xt∣<t;)=π⁢(xt∣<t)conditionalsubscriptsubscriptabsentsubscriptconditionalsubscriptsubscriptabsentp(x_t x_<t; θ)= _ θ(% x_t x_<t)p ( xitalic_t ∣ italic_x< t ; italic_θ ) = πbold_italic_θ ( xitalic_t ∣ italic_x< t ) denotes the prediction probability of model πsubscript _ θπbold_italic_θ for the t-th token, given the first t−11t-1t - 1 tokens as input. Next, we define the objectives for conducting RLHF and MU on the model πsubscript _ θπbold_italic_θ, respectively. 3.1 Definition of RLHF The standard RLHF paradigm consists of two main stages (Azar et al., 2024): i) learning a reward model, and i) optimizing the policy (i.e., the model parameters) based on the learned reward. In the reward model learning phase, a binary classifier is often trained using a logistic regression loss to distinguish preferred from non-preferred behaviors. A popular choice is the Bradley-Terry model (Bradley & Terry, 1952), where the pointwise reward r⁢(<t,xt)subscriptabsentsubscriptr( x_<t,x_t)r ( italic_x< t , xitalic_t ) serves as the score for action xtsubscriptx_txitalic_t, given context <tsubscriptabsent x_<titalic_x< t. Given a dataset a=<ti,xti≻x^tii=1Nasubscriptsuperscriptsubscriptsucceedssubscriptsuperscriptabsentsubscriptsuperscriptsubscriptsuperscript^1subscriptD_a=\ x^i_<t,x^i_t x^i_t\_i=1% ^N_aDitalic_a = italic_xitalic_i< t , xitalic_iitalic_t ≻ over start_ARG x end_ARGiitalic_t i = 1Nitalic_a, where xti≻x^tisucceedssubscriptsuperscriptsubscriptsuperscript^x^i_t x^i_txitalic_iitalic_t ≻ over start_ARG x end_ARGiitalic_t denotes a preference for xtisubscriptsuperscriptx^i_txitalic_iitalic_t over x^tisubscriptsuperscript x^i_tover start_ARG x end_ARGiitalic_t, the reward function is learned by minimizing the following logistic regression loss: ℒ⁢(r)=−(<ti,xti≻x^ti)∼a⁢[log⁡(p⁢(xti≻x^ti|<ti))],ℒsubscript∼succeedssubscriptsuperscriptabsentsubscriptsuperscriptsubscriptsuperscript^subscriptdelimited-[]succeedssubscriptsuperscriptconditionalsubscriptsuperscript^subscriptsuperscriptabsentL(r)=-E_( x^i_<t,x^i_t x^i% _t) _a [ (p(x^i_t x^i_t|% x^i_<t) ) ],L ( r ) = - blackboard_E( italic_xitalic_i start_POSTSUBSCRIPT < t , xitalic_iitalic_t ≻ over start_ARG x end_ARGiitalic_t ) ∼ Ditalic_a end_POSTSUBSCRIPT [ log ( p ( xitalic_iitalic_t ≻ over start_ARG x end_ARGiitalic_t | italic_xitalic_i< t ) ) ] , (2) where p⁢(xti≻x^ti|<ti)=σ⁢(r⁢(<ti,xti)−r⁢(<ti,x^ti))succeedssubscriptsuperscriptconditionalsubscriptsuperscript^subscriptsuperscriptabsentsubscriptsuperscriptabsentsubscriptsuperscriptsubscriptsuperscriptabsentsubscriptsuperscript^p(x^i_t x^i_t| x^i_<t)=σ (r(% x^i_<t,x^i_t)-r( x^i_<t, x^i_t) )p ( xitalic_iitalic_t ≻ over start_ARG x end_ARGiitalic_t | italic_xitalic_i< t ) = σ ( r ( italic_xitalic_i< t , xitalic_iitalic_t ) - r ( italic_xitalic_i< t , over start_ARG x end_ARGiitalic_t ) ) and σ⁢(·)·σ( )σ ( · ) denotes the sigmoid function. Based on the reward function, the objective of RLHF is to maximize the expected reward while minimizing the divergence between the policy πsubscript _ θπbold_italic_θ and a reference policy πrefsubscriptref _refπroman_ref. The specific objective can be expressed as: ⁢()=π⁢[r⁢(<ti,xti)]−τ⁢DKL⁢(π∥πref),subscriptsubscriptdelimited-[]subscriptsuperscriptabsentsubscriptsuperscriptsubscriptKLconditionalsubscriptsubscriptrefJ( θ)=E_ _ θ[r(% x^i_<t,x^i_t)]-τ D_KL( _ % θ _ref),J ( italic_θ ) = blackboard_Eπ start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT [ r ( italic_xitalic_i< t , xitalic_iitalic_t ) ] - τ Droman_KL ( πbold_italic_θ ∥ πroman_ref ) , (3) where <ti∼ρsimilar-tosubscriptsuperscriptabsent x^i_<t _xitalic_i< t ∼ ρ denote the sampled history, xti∼π(⋅|<ti)x^i_t _ θ(·| x^i_<t)xitalic_iitalic_t ∼ πbold_italic_θ ( ⋅ | italic_xitalic_i< t ) denote the action drawn from the policy, and τ is the parameter balancing the alignment and regularization objectives. The KL divergence DKLsubscriptKLD_KLDroman_KL is used to quantify the difference between the reference and current policies. Since LLM unlearning in this work incorporates a regularization term with analogous effects, we retain only the reward term in Eq. (3). 3.2 Definition of LLM Unlearning Mainstream methods for unlearning in LLMs typically involve fine-tuning the original model with an unlearning objective function. Giver a forget set fsubscriptD_fDitalic_f, while specific designs vary, the loss function in LLM unlearning tasks can generally be expressed as: ℒ⁢()=ℒforget⁢(f;)+λ⁢ℒreg⁢().ℒsubscriptℒforgetsubscriptsubscriptℒregL( θ)=L_forget(D_% f; θ)+ _reg( θ).L ( italic_θ ) = Lroman_forget ( Ditalic_f ; italic_θ ) + λ Lroman_reg ( italic_θ ) . (4) Here, ℒforgetsubscriptℒforgetL_forgetLroman_forget often is a loss term targeting data to be unlearned, reducing the model’s performance on these samples to minimize their influence on future predictions. To preserve the model’s overall performance on unrelated data and confine unlearning to the intended scope, regularization terms ℒregsubscriptℒregL_regLroman_reg such as output loss or divergence regularization are commonly introduced. These terms essentially act as parameter regularization. Specifically, commonly used loss-based methods (Jia et al., 2024a; Ji et al., 2024b) typically integrate one or more of the loss components. For readability, in this paper, we employ the widely adopted gradient ascent unlearning loss and parameter regularization loss as general objectives for LLM unlearning, considering their broad applicability. The formalization is as follows: min⁡1|f|⁢∑i=1|f|∑t=1nilog⁡p⁢(xt∣<ti;)⏟ℒforget⁢(f;)+λ⁢‖−∗‖2⏟ℒreg⁢().subscriptsubscript⏟1subscriptsuperscriptsubscript1subscriptsuperscriptsubscript1subscriptconditionalsubscriptsubscriptsuperscriptabsentsubscriptℒforgetsubscriptsubscript⏟superscriptnormsuperscript2subscriptℒreg _ θ 1|D_f| _i=1^|% D_f| _t=1^n_i p(x_t x^i_<t;% θ)_L_forget(D_f;% θ)+λ \| θ- % θ^*\|^2_L_reg( θ).minbold_italic_θ under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG | Ditalic_f | end_ARG ∑i = 1| Ditalic_f | ∑t = 1nitalic_i log p ( xitalic_t ∣ italic_xitalic_i< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT + λ under⏟ start_ARG ∥ italic_θ - italic_θ∗ ∥2 end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT . (5) A more detailed discussion on the definition of LLM unlearning can be found in Appendix A. 4 Connection between MU and PA 4.1 Impact of MU on PA Given a training sample xitalic_x to be unlearned, the unlearning objective in an LLM is described by Eq (5). We adopt a special bi-level optimization framework to link MU with PA, quantifying how unlearning a single sample affects the model’s PA performance. In this setup, the inner problem ensures the unlearning objective is achieved, while the outer problem evaluates its impact on PA performance. Specifically, we assume that the degree of unlearning for a sample xitalic_x is represented by the weight ≥00 ω≥ 0italic_ω ≥ 0, and the model parameters that satisfy the unlearning objective under this condition are denoted as ∗⁢()superscript θ^*( ω)italic_θ∗ ( italic_ω ). The bi-level optimization problem is formulated as: Find⁢(∗⁢())−⁢(∗⁢(0))Findsuperscriptsuperscript0 ( θ^*( % ω))-J( θ^*(0))Find J ( italic_θ∗ ( italic_ω ) ) - J ( italic_θ∗ ( 0 ) ) s.t.∗⁢()=arg⁡min⁡⁢ℒforget⁢(;)+λ⁢ℒreg⁢(),s.t.superscriptsubscriptsubscriptℒforgetsubscriptℒreg .t. θ^*( ω)= % _ θ ωL_forget(% x; θ)+ _reg(% θ),s.t. italic_θ∗ ( italic_ω ) = arg minbold_italic_θ italic_ω Lroman_forget ( italic_x ; italic_θ ) + λ Lroman_reg ( italic_θ ) , (6) where ⁢(∗⁢())superscriptJ( θ^*( ω))J ( italic_θ∗ ( italic_ω ) ) represents the PA performance of the model when the unlearning weight is ωitalic_ω. For example, ⁢(∗⁢(0))superscript0J( θ^*(0))J ( italic_θ∗ ( 0 ) ) represents the PA performance of the model without any unlearning. Inspired by the implicit function method for solving bi-level optimization problems, we further derive and prove Proposition 7. Assumption 4.1. ℒforget⁢(;)subscriptℒforgetL_forget( x; θ)Lroman_forget ( italic_x ; italic_θ ) is continuously differentiable w.r.t. θitalic_θ, and its Hessian matrix is positive semidefinite. ⁢()J( θ)J ( italic_θ ) is twice continuously differentiable w.r.t θitalic_θ. Proposition 4.2. If Assumptions 4.1 holds, the change in PA performance for a model with parameters ∗superscript θ^*italic_θ∗ after unlearning sample xitalic_x using unlearning weight ωitalic_ω satisfies: Δ⁢(∗⁢())≈−2⁢∇⁢(∗)⊤⁢∇ℒforget⁢(;∗).Δsuperscript2subscript∇superscriptsuperscripttopsubscript∇subscriptℒforgetsuperscript ( θ^*( ω))≈- % ω2 _ θJ( % θ^*) _ θL_forget(% x; θ^*).Δ J ( italic_θ∗ ( italic_ω ) ) ≈ - divide start_ARG italic_ω end_ARG start_ARG 2 end_ARG ∇italic_θ J ( italic_θ∗ )⊤ ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ) . (7) Proof. The proof can be found in Appendix B.1. ∎ According to Proposition 7, we can directly set the unlearning weight ωitalic_ω to 1 (i.e., Δ(∗(1) ( θ^*(1)Δ J ( italic_θ∗ ( 1 )) to quantitatively assess the impact of unlearning a single sample on the model’s PA performance. To further analyze the factors influencing Δ⁢(∗⁢())Δsuperscript ( θ^*( ω))Δ J ( italic_θ∗ ( italic_ω ) ), we decompose the gradient inner product into the gradient norm and the cosine of the angle between gradients, as follows: Δ⁢(∗⁢())≈−2⁢‖∇⁢(∗)‖⋅‖∇ℒforget⁢(;∗)‖⋅cos⁡(ϕ),Δsuperscript⋅2normsubscript∇superscriptnormsubscript∇subscriptℒforgetsuperscriptitalic-ϕ ( θ^*( ω))≈- % ω2\| _ θJ( % θ^*)\|·\| _ θL_forget% ( x; θ^*)\|· (φ),Δ J ( italic_θ∗ ( italic_ω ) ) ≈ - divide start_ARG italic_ω end_ARG start_ARG 2 end_ARG ∥ ∇italic_θ J ( italic_θ∗ ) ∥ ⋅ ∥ ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ) ∥ ⋅ cos ( ϕ ) , (8) where cos⁡(ϕ)italic-ϕ (φ)cos ( ϕ ) denotes the angle between the two gradient vectors. Then, the following conclusions can be drawn: • Conclusion 1: impact can be positive or negative. The impact of unlearning a sample on PA performance can be either positive or negative, depending on the gradient direction relationship (i.e., the sign of cos⁡(ϕ)italic-ϕ (φ)cos ( ϕ )), which is partially influenced by the reward of the unlearned sample’s combination. A sample xitalic_x can be represented as multiple combinations, i.e., =<t,xtt=1nsuperscriptsubscriptsubscriptabsentsubscript1 x=\ x_<t,x_t\_t=1^nitalic_x = italic_x< t , xitalic_t t = 1n. For low-reward combinations, where generated behavior often deviates significantly from human preferences, the unlearning objective gradient direction (i.e., the direction increasing the sample’s generation probability) is more likely to oppose the PA objective gradient direction. This results in cos⁡(ϕ)<0italic-ϕ0 (φ)<0cos ( ϕ ) < 0 and Δ⁢(∗⁢())>0Δsuperscript0 ( θ^*( ω))>0Δ J ( italic_θ∗ ( italic_ω ) ) > 0. Further analysis indicates that if the rewards for most combinations <t,xtsubscriptabsentsubscript\ x_<t,x_t\ italic_x< t , xitalic_t in a sample xitalic_x are low, unlearning the sample tends to improve preference alignment. Conversely, if only a few combinations have low rewards, unlearning the sample will likely hinder PA. • Conclusion 2: magnitude of impact varies. The effect of unlearning on PA performance varies across samples and is influenced by the unlearning degree and gradient norm. The gradient norm is an inherent property of the sample, such as the model’s degree of fit to the sample. For samples that the model fits well, the gradient norm tends to be smaller. On the other hand, the unlearning weight is a controllable factor that can be adjusted by tuning parameters such as the unlearning weight. 4.2 A Weighted MU Framework for PA The above conclusions indicate that, given a set =ii=1nsuperscriptsubscriptsuperscript1D=\ x^i\_i=1^nD = italic_xitalic_i i = 1n containing nnn negative samples, simply performing the unlearning operation directly according to Eq. (5) does not guarantee optimal PA results. This is primarily due to the following two issues: • Issue 1. Conclusion 1 suggests that for a given negative sample isuperscript x^iitalic_xitalic_i, which contains some low-reward combinations, this alone does not imply that unlearning isuperscript x^iitalic_xitalic_i will necessarily promote PA. The effectiveness of unlearning also depends on the proportion of low-reward components within the sample. This indicates that not all negative samples need to be unlearned. • Issue 2. Conclusion 2 indicates that even if different negative samples (e.g., isuperscript x^iitalic_xitalic_i and jsuperscript x^jitalic_xitalic_j) can both promote PA, the degree of promotion may vary. This difference can be controlled by adjusting the unlearning weight ωitalic_ω. Problem setup. To address these two issues, we propose a framework called Unlearning to Align (U2A) based on a sample-weighting approach. This framework achieves the maximization of PA performance by assigning higher weights to samples that contribute more significantly to performance improvement during the unlearning process. Specifically, when the weight ωitalic_ω is set to 0, it indicates that the corresponding sample is not selected for unlearning. For ease of analysis and discussion, we assume that the weight vector =[1,2,…,n]subscript1subscript2…subscript ω=[ ω_1, ω_2,…,% ω_n]italic_ω = [ italic_ω1 , italic_ω2 , … , italic_ωitalic_n ] lies on an nnn-dimensional simplex, and we denote the unlearning loss of each sample isuperscript x^iitalic_xitalic_i as ℓi⁢()subscriptℓ _i( θ)ℓitalic_i ( italic_θ ). The U2A framework can be formalized as solving the following optimization problem: min∈Δn−⁢(∗⁢())+β⁢Lp⁢()subscriptsubscriptΔsuperscriptsubscriptL _ ω∈ _n -J(% θ^*( ω))+ _p( % ω)minbold_italic_ω ∈ Δ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT - J ( italic_θ∗ ( italic_ω ) ) + β Litalic_p ( italic_ω ) s.t.∗⁢()=arg⁢min⁢∑i=1ni⁢ℓi⁢()+λ⁢ℒreg⁢(),s.t.superscriptsubscriptargminsuperscriptsubscript1subscriptsubscriptℓsubscriptℒreg .t. θ^*( ω)=% *arg\,min_ θ _i=1^n % ω_i _i( θ)+ _reg(% θ),s.t. italic_θ∗ ( italic_ω ) = start_OPERATOR arg min end_OPERATORitalic_θ ∑i = 1n italic_ωitalic_i ℓitalic_i ( italic_θ ) + λ Lroman_reg ( italic_θ ) , (9) where Lp⁢()subscriptLL_p( ω)Litalic_p ( italic_ω ) represents an introduced LpsubscriptLL_pLitalic_p-norm sparsity-inducing regularization term to ensure that the number of selected samples for unlearning is as small as possible, and β denotes the weight coefficient of the regularization term. Further analysis shows that when p=11p=1p = 1, the sparsity regularization has relatively weak compressive effects on small values. On the other hand, when p=00p=0p = 0, it can effectively control the sparsity of weights (i.e., the number of non-zero weights). However, in this case, the regularization term Lq⁢()subscriptLL_q( ω)Litalic_q ( italic_ω ) becomes a non-continuous and non-convex function, which significantly increases the difficulty of optimization. Considering these factors, when p=1212p= 12p = divide start_ARG 1 end_ARG start_ARG 2 end_ARG, the regularization term Lp⁢()subscriptLL_p( ω)Litalic_p ( italic_ω ) is both a strictly convex function and exhibits good smoothness. Therefore, in this paper, we set p=1212p= 12p = divide start_ARG 1 end_ARG start_ARG 2 end_ARG, making Lp⁢()=∑i=1nisubscriptLsuperscriptsubscript1subscriptL_p( ω)= _i=1^n ω_% iLitalic_p ( italic_ω ) = ∑i = 1n square-root start_ARG italic_ωitalic_i end_ARG. Solving Eq. (9) yields the selected unlearning set SS as well as the unlearning weight ωitalic_ω for each sample. U2A framework. To enhance clarity, we denote the outer objective function as g⁢()g( ω)g ( italic_ω ) and the inner objective function as f⁢(,)f( θ, ω)f ( italic_θ , italic_ω ). If f⁢(,)f( θ, ω)f ( italic_θ , italic_ω ) is twice differentiable w.r.t. θ, the constraint ∗⁢()=arg⁡min⁡f⁢(,)superscriptsubscript θ^*( ω)= _ θf(% θ, ω)italic_θ∗ ( italic_ω ) = arg minbold_italic_θ f ( italic_θ , italic_ω ) can be relaxed into ∂f⁢(,)∂|=∗⁢()=0evaluated-atsuperscript0 ∂ f( θ, ω)∂ % θ|_ θ= θ^*( ω)=0divide start_ARG ∂ f ( italic_θ , italic_ω ) end_ARG start_ARG ∂ italic_θ end_ARG |italic_θ = italic_θ∗ ( italic_ω ) = 0. When f⁢(,)f( θ, ω)f ( italic_θ , italic_ω ) is strictly convex w.r.t. θitalic_θ, this relaxation becomes tight (Borsos et al., 2024). Assumption 4.1 ensures this property, enabling the use of first-order optimization methods (Pedregosa, 2016; Finn et al., 2017; Liu et al., 2019) to solve Eq. (6) and avoiding computationally expensive naive greedy algorithms. Considering the computation efficiency, we adopt a variant of the cone-constrained generalized matching pursuit algorithm (Locatello et al., 2017), which performs incremental optimization. This approach iteratively constructs the unlearning set SS, thereby significantly reducing computational complexity. Algorithm 1 U2A Algorithm 1: Input: Dataset =ii=1nsuperscriptsubscriptsuperscript1D=\ x^i\_i=1^nD = italic_xitalic_i i = 1n, initial model parameter ∗superscript θ^*italic_θ∗, maximum number of iterations T, early stopping threshold δ, regularization coefficient λ and β. 2: Output: unlearning set finalsuperscriptfinalS^finalSroman_final and weights final,∗superscriptfinal ω^final,*italic_ωroman_final , ∗. 3: Initialization: Initialize weights 1,∗=[0,…,0]superscript10…0 ω^1,*=[0,…,0]italic_ω1 , ∗ = [ 0 , … , 0 ], randomly select one point to initialize the unlearning set 1=isuperscript1S^1=\i\S1 = i , and set i1,∗=1subscriptsuperscript11 ω^1,*_i=1italic_ω1 , ∗i = 1. 4: for t=22t=2t = 2 to T do 5: Gradient descent to solve the inner problem of Eq. (9) to obtain ∗⁢(t−1,∗)superscriptsuperscript1 θ^*( ω^t-1,*)italic_θ∗ ( italic_ωitalic_t - 1 , ∗ ). 6: Fix current model parameter ∗⁢(t−1,∗)superscriptsuperscript1 θ^*( ω^t-1,*)italic_θ∗ ( italic_ωitalic_t - 1 , ∗ ), and optimize the weights t,∗superscript ω^t,*italic_ωitalic_t , ∗ according to Eq. (10). 7: Select point k∗superscriptk^*k∗ with the maximum Δ⁢g⁢(k)Δ g(k)Δ g ( k ) using Eq. (12). 8: Update the unlearning set t=t−1∪k∗superscriptsuperscript1superscriptS^t=S^t-1∪\k^*\Sitalic_t = Sitalic_t - 1 ∪ k∗ and set kt,∗=1subscriptsuperscript1 ω^t,*_k=1italic_ωitalic_t , ∗k = 1. 9: if g⁢(t−1,∗)−g⁢(t,∗)≤δsuperscript1superscriptg( ω^t-1,*)-g( ω^t,*)≤ ( italic_ωitalic_t - 1 , ∗ ) - g ( italic_ωitalic_t , ∗ ) ≤ δ then 10: Break. 11: end if 12: end for 13: Re-optimize the weights final,∗superscriptfinal ω^final,*italic_ωroman_final , ∗ on the final unlearning set finalsuperscriptfinalS^finalSroman_final. Specifically, in each iteration, we first solve the inner optimization problem using the gradient descent method to obtain the model parameters ∗⁢()superscript θ^*( ω)italic_θ∗ ( italic_ω ) with optimal unlearning performance. Subsequently, we fix the model parameters to solve the outer optimization problem to obtain the solution t,∗superscript ω^t,*italic_ωitalic_t , ∗, which can be formalized as: t,∗=arg⁢min∈Δng()s.t.supp()=t−1, ω^t,*= *arg\,min_ ω∈% _ng( ω) .t. (% ω)=S_t-1,italic_ωitalic_t , ∗ = start_OPERATOR arg min end_OPERATORitalic_ω ∈ Δ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT g ( italic_ω ) s . t . supp ( italic_ω ) = Sitalic_t - 1 , (10) where the constraint is imposed to restrict the support set of the weight vector ωitalic_ω to be identical to the current unlearning set t−1subscript1S_t-1Sitalic_t - 1. In other words, the non-zero components of ωitalic_ω are confined to elements within the current unlearning set, thereby preventing the introduction of new sample points. It is important to note that the support set supp⁢()=i∣i≠0suppconditional-setsubscript0supp( ω)=\i ω_i≠ 0\supp ( italic_ω ) = i ∣ italic_ωitalic_i ≠ 0 represents the index set of the non-zero elements in ωitalic_ω. This can be solved by using the projected gradient descent method. According to the implicit function theorem, the gradient of g⁢()g( ω)g ( italic_ω ) w.r.t. ωitalic_ω can be expressed as: ∇g⁢()=∇⁢(∗⁢())⁢(∂2f∂2)−1⁢∑i=1n∇ℓi⁢(∗⁢())+β2⁢−12,subscript∇subscript∇superscriptsuperscriptsuperscript2superscript21superscriptsubscript1subscript∇subscriptℓsuperscript2superscript12 _ ωg( ω)= _ % θJ( θ^*( ω)) ( % ∂^2f∂ θ^2 )^-1 _i=1^n% _ θ _i( θ^*( % ω))+ β2 ω^- 12,∇italic_ω g ( italic_ω ) = ∇italic_θ J ( italic_θ∗ ( italic_ω ) ) ( divide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG )- 1 ∑i = 1n ∇italic_θ ℓitalic_i ( italic_θ∗ ( italic_ω ) ) + divide start_ARG β end_ARG start_ARG 2 end_ARG italic_ω- divide start_ARG 1 end_ARG start_ARG 2 end_ARG , (11) where ∂2f∂2=∑i=1ni⁢∇2ℓi⁢(∗⁢())+2⁢λ⁢Isuperscript2superscript2superscriptsubscript1subscriptsuperscriptsubscript∇2subscriptℓsuperscript2 ∂^2f∂ θ^2= _i=1^n% ω_i _ θ^2 _i( % θ^*( ω))+2λ Idivide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG = ∑i = 1n italic_ωitalic_i ∇italic_θ2 ℓitalic_i ( italic_θ∗ ( italic_ω ) ) + 2 λ I, denoting the Hessian matrix of the inner optimization problem (details are provided in Appendix B.2). Given that explicitly computing the inverse of the Hessian matrix is computationally prohibitive in LLM applications, directly solving the problem is impractical. Therefore, we adopt the conjugate gradient method (Nocedal & Wright, 2006), which iteratively computes the product of the Hessian matrix and gradient vector, thereby reducing the computational complexity. After completing the outer optimization, we identify a new sample point k to add to the unlearning set based on the marginal gain of the outer objective function g⁢()g( ω)g ( italic_ω ), with the goal of maximizing the marginal gain. The marginal gain is calculated as Δ⁢g⁢(k)=∂g⁢()kΔsubscript g(k)= ∂ g( ω) ω_kΔ g ( k ) = divide start_ARG ∂ g ( italic_ω ) end_ARG start_ARG italic_ωitalic_k end_ARG. According to Eq. (11), its expression can be derived as follows: Δ⁢g⁢(k)=∇⁢(∗⁢())⁢(∂2f∂2)−1⁢∇ℓk⁢(∗⁢())+β2⁢k−12.Δsubscript∇superscriptsuperscriptsuperscript2superscript21subscript∇subscriptℓsuperscript2superscriptsubscript12 g(k)= _ θJ( θ^*(% ω)) ( ∂^2f∂ θ^2% )^-1 _ θ _k( θ^*(% ω))+ β2 ω_k^- 12.Δ g ( k ) = ∇italic_θ J ( italic_θ∗ ( italic_ω ) ) ( divide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG )- 1 ∇italic_θ ℓitalic_k ( italic_θ∗ ( italic_ω ) ) + divide start_ARG β end_ARG start_ARG 2 end_ARG italic_ωitalic_k- divide start_ARG 1 end_ARG start_ARG 2 end_ARG . (12) After computing the marginal gain, we select the sample point with the maximum gain k∗=arg⁢maxk∈[1,n]⁡Δ⁢g⁢(k)superscriptsubscriptargmax1Δk^*= *arg\,max_k∈[1,n] g(k)k∗ = start_OPERATOR arg max end_OPERATORk ∈ [ 1 , n ] Δ g ( k ), add it to the unlearning set t−1subscript1S_t-1Sitalic_t - 1. The detailed implementation of the algorithm is provided in Algorithm 1. Convergence analysis. Lemma 4.3 indicates that as the number of iterations t increases, the solution obtained by our U2A algorithm gradually approaches the optimal solution, and the error decreases at ⁢(1/t)1O(1/t)O ( 1 / t ). Theorem 4.4 demonstrates that as the size of the unlearning set m increases, the approximation error gradually diminishes. This implies that it is unnecessary to unlearn all negative examples, selecting a subset is sufficient to make the value of the objective function very close to the optimal value of the original problem. Lemma 4.3. Suboptimality Bound (cf. Theorem 2 of Locatello et al. (2017)). Assume g⁢()g( ω)g ( italic_ω ) is L-smooth, and let the initial suboptimality be denoted as ε1=g⁢(1,∗)−g⁢(∗)subscriptε1gsuperscript1gsuperscript _1=g( ω^1,*)-g( ω^*)ε1 = g ( italic_ω1 , ∗ ) - g ( italic_ω∗ ). After t iterations, the suboptimality bound of the U2A algorithm can be expressed as: g⁢(t,∗)−g⁢(∗)≤8⁢L+4⁢ε1t+3,superscriptsuperscript84subscript13g( ω^t,*)-g( ω^*)≤ 8L+4% _1t+3,g ( italic_ωitalic_t , ∗ ) - g ( italic_ω∗ ) ≤ divide start_ARG 8 L + 4 ε1 end_ARG start_ARG t + 3 end_ARG , where g⁢(∗)superscriptg( ω^*)g ( italic_ω∗ ) represents the global optimal value. In the case of a non-convex objective function, g⁢(∗)superscriptg( ω^*)g ( italic_ω∗ ) is approximated as a certain local optimal value. Theorem 4.4. Size of Unlearning Set. Under the condition that the suboptimality error does not exceed ε ε, the size m of the final unlearning set satisfies: m∈⁢((L+ε1)⁢ε−1).subscript1superscript1m ((L+ _1) ^-1).m ∈ O ( ( L + ε1 ) ε- 1 ) . That is, the size of the final unlearning set is proportional to the smoothness of the objective function and the initial suboptimality ε1subscript1 _1ε1, while being inversely proportional to the target precision ε ε. Proof. The proof can be found in Appendix B.3. ∎ Complexity analysis. The inner optimization problem must be solved in each iteration to determine the optimal model parameter. Assuming tfsubscriptt_ftitalic_f gradient descent iterations are required, with each iteration computing gradients for all data points, and the gradient computation complexity for a single data point is c, the total complexity of the inner optimization is ⁢(tf⋅n⋅c)⋅subscriptO(t_f· n· c)O ( titalic_f ⋅ n ⋅ c ). For the outer problem, given an unlearning set tsuperscriptS^tSitalic_t with t samples, and tsubscriptt_ ωtbold_italic_ω updates required per optimization, the complexity of solving t,∗superscript ω^t,*italic_ωitalic_t , ∗ is ⁢(t⋅t⋅d)⋅subscriptO(t· t_ ω· d)O ( t ⋅ tbold_italic_ω ⋅ d ), where d is the model parameter dimension. Marginal gain computation involves implicit gradient calculations. Using the conjugate gradient method with tgsubscriptt_gtitalic_g iterations, where each iteration requires a Hessian-vector product computation of complexity ⁢(tg⋅n⋅c)⋅subscriptO(t_g· n· c)O ( titalic_g ⋅ n ⋅ c ), and computing the gradients of all data points w.r.t. θitalic_θ contributes an additional complexity of ⁢(n⋅c)⋅O(n· c)O ( n ⋅ c ). Thus, the total complexity for marginal gain computation is ⁢((tg+1)⋅n⋅c)⋅subscript1O((t_g+1)· n· c)O ( ( titalic_g + 1 ) ⋅ n ⋅ c ). Finally, if the final unlearning set contains m samples, the overall algorithm complexity can be expressed as: ⁢(m⋅((tf+tg+1)⋅n⋅c+n⋅d+m⋅t⋅d)).⋅subscriptsubscript1⋅subscriptO(m·((t_f+t_g+1)·n· c+n· d+m· t_% ω· d)).O ( m ⋅ ( ( titalic_f + titalic_g + 1 ) ⋅ n ⋅ c + n ⋅ d + m ⋅ tbold_italic_ω ⋅ d ) ) . This demonstrates that our U2A algorithm is computationally efficient and well-suited for high-dimensional applications, such as LLMs. 5 Experiment 5.1 Experiment Setups Datasets. In our experiments, we evaluate across three mainstream PA tasks and datasets: (i) reducing harmfulness PKU SafeRLHF (Dai et al., 2023), (i) enhancing usefulness UltraFeedback (Cui et al., 2023; Tunstall et al., 2023), (i) eliminating hallucinations HaluEval (Li et al., 2023). For each dataset, we randomly sample 80% of the examples to construct the fine-tuning set FINsubscriptFIND_FINDroman_FIN, while the remaining examples PAsubscriptPAD_PADroman_PA are reserved for evaluating the model’s PA performance post-unlearning. From FINsubscriptFIND_FINDroman_FIN, we further extract 12.5% negative samples to construct a subset SELsubscriptSELD_SELDroman_SEL, representing selectively forgettable negative examples. Details of this partitioning are provided in Appendix C.1. Base and preference reward models. Following the configurations in prior study (Jia et al., 2024a; Zhou et al., 2024), we select the widely used Llama-2-7B-Chat (Touvron et al., 2023) and Llama-3.1-8B-Instruct (Dubey et al., 2024) as base models for each dataset. The base models are first fine-tuned on FINsubscriptFIND_FINDroman_FIN to obtain the original models requiring unlearning. Additionally, we fine-tune the base models on FINsubscriptFIND_FINDroman_FIN excluding the negative samples SELsubscriptSELD_SELDroman_SEL. For PA evaluation, we employ the Beaver-7B-v3.0-Reward model (Dai et al., 2023; Ji et al., 2024a). Evaluation metrics. We evaluate our proposed method along two dimensions: PA performance and unlearning performance. For PA performance, in the PKU SafeRLHF dataset, we utilize two metrics, Reward-value (Chakraborty et al., 2024; Yao et al., 2024) and ASR (Xu et al., 2024), where ASR can be further divided into four dimensions: ASR-keyword, ASR-answer, ASR-useful, and ASR-summary. In the UltraFeedback Binarized dataset, we evaluate PA performance using Win-rate (Xiao et al., 2024; Rafailov et al., 2024) and Coherence (Chakraborty et al., 2024; Khanov et al., 2024; Kong et al., 2024), where the Win-rate includes Length-control Win Rate and Win Rate vs. GPT-4. In the HaluEval dataset, Hallucination-rate (Yao et al., 2024) is adopted as the evaluation metric, including F1, Precision and Recall. For unlearning performance, we evaluate unlearning effectiveness and model utility. Unlearning effectiveness for PKU SafeRLHF, UltraFeedback and HaluEval is measured using Membership Inference Attack (MIA) (Jia et al., 2024a). Model utility is evaluated with Perplexity (PPL) (Yao et al., 2024; Doshi & Stickland, 2024). Details are provided in Appendix C.2. Baselines. We evaluate our proposed method U2A against widely acknowledged baselines, including unlearning methods (i.e., Retrain, GA (Maini et al., 2024), GradDiff (Liu et al., 2022; Yao et al., 2024), and NPO (Zhang et al., 2024)), as well as PA methods (i.e., PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2024)). The effectiveness of our method is validated by comparing the U2A-improved unlearning baseline with the original baseline and existing PA baselines. Training setup. We use the AdamW (Loshchilov, 2017) optimizer with a learning rate of 4⁢e−6464e-64 e - 6 for training, and a learning rate of 3⁢e−2323e-23 e - 2 for updating the unlearning weights. Both optimizers utilize a cosine annealing scheduler for learning rate scheduling. The hyperparameters of baselines are kept as reported in their original papers. For our method, the regularization coefficient is set to λ=1.01.0λ=1.0λ = 1.0, the scaling coefficient to β=0.50.5β=0.5β = 0.5, the early stopping threshold δ to 0.01, and the maximum iterations to 100. All experiments were conducted on NVIDIA A800 GPUs. For the PKU SafeRLHF dataset and the HaluEval dataset, the number of iterations was set to 50, with the number of update epochs set to 5. For the UltraFeedback Binarized dataset, the number of iterations was set to 25, with the number of update epochs set to 10. 5.2 Experiment Results (a) PKU SafeRLHF (b) UltraFeedback (c) HaluEval Figure 1: Effect of unlearning individual data samples on PA performance of Llama-2-7B-Chat model. Each point represents PA performance change after unlearning a specific data sample. The angle of each point follows a uniform distribution, while the radial distance indicates the magnitude of PA performance change. Red points represent negative effects (i.e., unlearning this sample led to worse PA), whereas blue points represent positive effects (i.e., unlearning this sample improved PA). Note that larger distances from the origin correspond to stronger impacts on PA performance. (a) PKU SafeRLHF (b) UltraFeedback (c) HaluEval Figure 2: Analysis of how unlearning data samples affects PA performance. Each point represents the change in PA performance (Δ ΔPA) after unlearning a group of samples. The x-axis denotes the proportion of low-reward tokens in the unlearned sample groups, and the y-axis represents the corresponding change in PA performance. Points are colored based on Δ ΔPA: blue indicates a positive change, while red indicates a negative change. The shapes further differentiate the groups: squares represent sample groups with a low proportion of low-reward tokens (i.e., high-reward samples), whereas circles represent sample groups with a high proportion of low-reward tokens (i.e., low-reward samples). Unlearning affects PA. We assess the impact of unlearning individual samples on PA performance using the Llama-2-7B-Chat model across three datasets. Given the nearly negligible effect of unlearning a single sample on model parameters, we randomly select 150 groups, each with 32 negative samples, from a pool of eligible negative samples. PA performance changes after unlearning each group are compared, with parameter ωitalic_ω set to 1. Figure 1 show that unlearning can have both positive and negative effects, suggesting that removing negative samples does not consistently improve PA performance. Additionally, the degree of improvement varies significantly across different samples. To better understand this, we decompose reward values for each token. Tokens with reward values below the average reward across all tokens are classified as “low-reward”, while those above the average are categorized as “high-reward”. The average reward values for each dataset are as follows: -1.7463 for PKU SafeRLHF, 0.9001 for UltraFeedback, and -0.7839 for HaluEval. To distinguish the impact of different sample groups, we apply a threshold on the proportion of low-reward tokens. Specifically, samples with a low-reward token proportion below the threshold (0.60 for PKU SafeRLHF, 0.575 for UltraFeedback, and 0.575 for HaluEval) are marked in red, while those exceeding the threshold are marked in blue. Figure 2 illustrates the impact of unlearning these samples on PA performance. Taking the PKU SafeRLHF dataset as an example, when the proportion of low-reward tokens in the unlearned dataset exceeds the threshold (dashed vertical line at 0.6), most changes value of PA performance are positive, indicating an improvement in PA. Conversely, when the proportion of low-reward tokens is below the threshold, most changes value of PA performance are negative, suggesting that unlearning such samples tends to degrade PA performance. Overall, red samples, with more dispersed rewards and fewer low-reward tokens, tend to hinder PA performance improvement when unlearned. In contrast, unlearning blue samples, characterized by a higher proportion of low-reward tokens, significantly boost PA performance. These findings align with our theoretical analysis in Section 4.1. Effectiveness of U2A. To validate the effectiveness of our proposed U2A framework, we conduct experiments using the Llama-2-7B-Chat and Llama-3.1-8B-Instruct models on three datasets. We first apply existing unlearning baseline methods to all eligible negative samples. Next, we integrate the U2A framework with these baselines using a sample-weighted unlearning approach. Finally, we employ existing PA baselines to perform direct PA on subsets of the alignment datasets PAsubscriptPAD_PADroman_PA. We report the comparison of PA and unlearning performance across different methods in Table 1. The results show that improving unlearning baselines with the U2A framework yields significant PA performance improvements, while maintaining comparable unlearning performance. Additionally, the improved unlearning methods outperform PA baselines in PA performance to a certain extent. These results highlight the potential of leveraging unlearning mechanisms to enhance alignment, further validating the effectiveness of our proposed method. Results for other tasks are reported in Appendix D. Table 1: Comparison of the U2A framework with the current PA and unlearning baseline methods on the PKU SafeRLHF dataset. For the same baseline before and after improvement, consistent experimental settings are ensured. Optimal results are highlighted in bold. Models Methods PA Performance MU Performance Reward-value (↑ ↑) ASR-keyword (↓ ↓) ASR-answer (↓ ↓) ASR-useful (↓ ↓) ASR-summary (↓ ↓) MIA (↑ ↑) PPL (↓ ↓) Llama-2-7B-Chat Original -5.84 78.65 54.42 30.19 61.54 0.4892 76.49 Retrain -4.23 73.08 62.69 24.23 59.42 0.4866 104.36 PPO -0.59 74.42 58.65 29.62 52.50 0.4884 103.39 DPO -2.30 72.27 56.15 27.31 52.69 0.4892 76.28 GA -4.58 84.04 56.92 29.23 54.42 0.4862 71.46 GradDiff -4.76 82.35 58.17 25.40 56.95 0.4912 60.11 NPO -5.96 77.69 54.30 30.05 68.82 0.4891 76.50 GA + U2A 1.49 74.13 53.19 21.19 48.74 0.4770 56.31 GradDiff + U2A 2.63 73.46 57.00 24.75 50.20 0.4992 51.28 NPO + U2A -3.65 77.12 52.51 23.65 56.26 0.4892 67.74 Llama-3.1-8B-Instruct Original -3.82 99.23 91.73 13.65 90.19 0.4946 2.01 Retrain -2.11 99.81 90.58 14.04 87.69 0.4963 2.07 PPO -1.07 96.73 83.65 10.19 79.81 0.4881 2.01 DPO -1.33 95.33 79.01 11.77 82.59 0.4919 2.08 GA -2.56 96.17 75.11 11.45 70.33 0.4978 2.03 GradDiff -1.87 85.34 63.55 15.97 52.55 0.5127 2.06 NPO -3.77 98.65 91.57 13.90 90.37 0.3924 2.11 GA + U2A 0.03 92.31 71.35 10.15 62.48 0.5563 2.01 GradDiff + U2A 0.95 83.63 62.28 9.65 49.50 0.5489 1.96 NPO + U2A -0.21 96.68 89.74 11.62 88.56 0.4723 2.05 Efficiency of U2A. We assess the efficiency of the U2A-improved unlearning baseline against existing PA methods on the PKU SafeRLHF dataset using the Llama-2-7B-Chat model. With identical hyperparameters and early stopping conditions (training halts when the PA score surpasses 0.745), we compare the number of update iterations and per-iteration time costs. Table 2 summarizes the results, showing that the U2A-enhanced method significantly reduces update iterations needed to reach the target PA score. Moreover, although it introduces additional computation, leading to an increase in the average per-iteration time, the significant reduction in the number of iterations results in a training time reduction of approximately 90% compared to PA baselines such as PPO and DPO. These findings confirm the U2A framework’s efficiency and practicality under resource-limited settings. Table 2: Comparison of the efficiency of different methods. We set an early stopping condition, where PA reaches a fixed value 0.745, and compare the number of epochs (update rounds) and the time costs of different methods. The experimental setups remain consistent before and after the improvement using U2A. Optimal results are highlighted in bold. Methods Update Rounds Time Costs(s) Methods Update Rounds Time Costs(s) Methods Update Rounds Time Costs(s) Retrain 197 241.2498 GA 142 167.1139 GA + U2A 46 248.3760 PPO 282 911.7341 GradDiff 764 995.9192 GradDiff + U2A 19 70.8349 DPO 1353 3619.9515 NPO 751 967.1019 NPO + U2A 29 132.1398 6 Conclusion The mainstream PA approach, RLHF, faces significant challenges in low-resource settings, including (1) reliance on numerous positive preference samples, which are costly to obtain, and (2) instability during training, resulting in high computational and management costs. To address these issues, we propose a MU-based method that reduces dependence on positive samples by mitigating the influence of negative samples to achieve PA. Our method achieves computational efficiency comparable to standard fine-tuning while showing strong potential. We first develop a bi-level optimization framework to evaluate the impact of unlearning individual samples on PA performance. Through this analysis, we observe that negative samples contribute unevenly to PA, with many offering limited benefits. This observation leads to a key question: how can we selectively weight and unlearn negative samples to optimize alignment? To this end, we formally define the problem and introduce U2A, a framework leveraging bi-level optimization to efficiently select and weighted unlearn samples for improved alignment. Experiments demonstrate that U2A significantly enhances alignment efficiency and effectiveness, underscoring its value in resource-constrained scenarios. By linking PA with MU, this work provides a novel perspective on PA for LLMs and suggests new directions for optimizing PA algorithms. References Azar et al. (2024) Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. PMLR, 2024. Bai et al. (2022) Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Beirami et al. (2024) Beirami, A., Agarwal, A., Berant, J., D’Amour, A., Eisenstein, J., Nagpal, C., and Suresh, A. T. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879, 2024. Borsos et al. (2024) Borsos, Z., Mutnỳ, M., Tagliasacchi, M., and Krause, A. Data summarization via bilevel optimization. Journal of Machine Learning Research, 25(73):1–53, 2024. Bradley & Terry (1952) Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Casper et al. (2023) Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. Chakraborty et al. (2024) Chakraborty, S., Ghosal, S. S., Yin, M., Manocha, D., Wang, M., Bedi, A. S., and Huang, F. Transfer q star: Principled decoding for llm alignment. arXiv preprint arXiv:2405.20495, 2024. Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Cui et al. (2023) Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback. 2023. Dai et al. (2023) Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023. Dige et al. (2024) Dige, O., Arneja, D., Yau, T. F., Zhang, Q., Bolandraftar, M., Zhu, X., and Khattak, F. Can machine unlearning reduce social bias in language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, p. 954–969, 2024. Doshi & Stickland (2024) Doshi, J. and Stickland, A. C. Does unlearning truly unlearn? a black box evaluation of llm unlearning methods. arXiv preprint arXiv:2411.12103, 2024. Dubey et al. (2024) Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Fan et al. (2024) Fan, C., Liu, J., Lin, L., Jia, J., Zhang, R., Mei, S., and Liu, S. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163, 2024. Feng et al. (2024) Feng, X., Chen, C., Li, Y., and Lin, Z. Fine-grained pluggable gradient ascent for knowledge unlearning in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 10141–10155, 2024. Finn et al. (2017) Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, p. 1126–1135. PMLR, 2017. Gao et al. (2023) Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, p. 10835–10866. PMLR, 2023. Huang et al. (2024) Huang, J. Y., Sengupta, S., Bonadiman, D., Lai, Y.-a., Gupta, A., Pappas, N., Mansour, S., Kirchhoff, K., and Roth, D. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06147, 2024. Jang et al. (2023) Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14389–14408, 2023. Ji et al. (2024a) Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Chen, B., Sun, R., Wang, Y., and Yang, Y. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024a. Ji et al. (2024b) Ji, J., Liu, Y., Zhang, Y., Liu, G., Kompella, R. R., Liu, S., and Chang, S. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. In Annual Conference on Neural Information Processing Systems, 2024b. Jia et al. (2024a) Jia, J., Liu, J., Zhang, Y., Ram, P., Baracaldo, N., and Liu, S. Wagle: Strategic weight attribution for effective and modular unlearning in large language models. arXiv preprint arXiv:2410.17509, 2024a. Jia et al. (2024b) Jia, J., Liu, J., Zhang, Y., Ram, P., Baracaldo, N., and Liu, S. Wagle: Strategic weight attribution for effective and modular unlearning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. Khanov et al. (2024) Khanov, M., Burapacheep, J., and Li, Y. Args: Alignment as reward-guided search. arXiv preprint arXiv:2402.01694, 2024. Kim et al. (2023) Kim, M., Lee, H., Yoo, K. M., Park, J., Lee, H., and Jung, K. Critic-guided decoding for controlled text generation. In Findings of the Association for Computational Linguistics: ACL 2023, p. 4598–4612, 2023. Kong et al. (2024) Kong, L., Wang, H., Mu, W., Du, Y., Zhuang, Y., Zhou, Y., Song, Y., Zhang, R., Wang, K., and Zhang, C. Aligning large language models with representation editing: A control perspective. arXiv preprint arXiv:2406.05954, 2024. Korbak et al. (2023) Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretraining language models with human preferences. In International Conference on Machine Learning, p. 17506–17533. PMLR, 2023. Langley (2000) Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), p. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann. Li et al. (2023) Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747, 2023. Li et al. (2024) Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Mukobi, G., et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. In Forty-first International Conference on Machine Learning, 2024. Liu et al. (2022) Liu, B., Liu, Q., and Stone, P. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, p. 243–254. PMLR, 2022. Liu et al. (2019) Liu, H., Simonyan, K., and Yang, Y. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2019. Liu et al. (2023) Liu, J., Cohen, A., Pasunuru, R., Choi, Y., Hajishirzi, H., and Celikyilmaz, A. Making ppo even better: Value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309.15028, 2023. Liu et al. (2024a) Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. Rethinking machine unlearning for large language models. In Annual Conference on Neural Information Processing Systems, 2024a. Liu et al. (2024b) Liu, T., Guo, S., Bianco, L., Calandriello, D., Berthet, Q., Llinares, F., Hoffmann, J., Dixon, L., Valko, M., and Blondel, M. Decoding-time realignment of language models. arXiv preprint arXiv:2402.02992, 2024b. Liu et al. (2024c) Liu, Z., Dou, G., Tan, Z., Tian, Y., and Jiang, M. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058, 2024c. Locatello et al. (2017) Locatello, F., Tschannen, M., Rätsch, G., and Jaggi, M. Greedy algorithms for cone constrained optimization with convergence guarantees. Advances in Neural Information Processing Systems, 30, 2017. Loshchilov (2017) Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. Lu et al. (2022) Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591–27609, 2022. Maini et al. (2024) Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. Tofu: A task of fictitious unlearning for llms. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024. Mitchell et al. (2024) Mitchell, E., Rafailov, R., Sharma, A., Finn, C., and Manning, C. D. An emulator for fine-tuning large language models using small language models. In The Twelfth International Conference on Learning Representations, 2024. Mudgal et al. (2024) Mudgal, S., Lee, J., Ganapathy, H., Li, Y., Wang, T., Huang, Y., Chen, Z., Cheng, H.-T., Collins, M., Strohman, T., et al. Controlled decoding from language models. In Forty-first International Conference on Machine Learning, 2024. Nocedal & Wright (2006) Nocedal, J. and Wright, S. J. Conjugate gradient methods. Numerical optimization, p. 101–134, 2006. Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Pedregosa (2016) Pedregosa, F. Hyperparameter optimization with approximate gradient. In International conference on machine learning, p. 737–746. PMLR, 2016. Rafailov et al. (2024) Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shi et al. (2023) Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023. Stiennon et al. (2020) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. Su et al. (2022) Su, Y., Lan, T., Wang, Y., Yogatama, D., Kong, L., and Collier, N. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35:21548–21561, 2022. Tian et al. (2024) Tian, B., Liang, X., Cheng, S., Liu, Q., Wang, M., Sui, D., Chen, X., Chen, H., and Zhang, N. To forget or not? towards practical knowledge unlearning for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 1524–1537, 2024. Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Tunstall et al. (2023) Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. Wan et al. (2024) Wan, Z., Feng, X., Wen, M., McAleer, S. M., Wen, Y., Zhang, W., and Wang, J. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning, 2024. Wang et al. (2023) Wang, L., Chen, T., Yuan, W., Zeng, X., Wong, K.-F., and Yin, H. Kga: A general machine unlearning framework based on knowledge gap alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13264–13276, 2023. Wang et al. (2024) Wang, Y., Wei, J., Liu, C. Y., Pang, J., Liu, Q., Shah, A. P., Bao, Y., Liu, Y., and Wei, W. Llm unlearning via loss adjustment with only forget data. arXiv preprint arXiv:2410.11143, 2024. Wu et al. (2024) Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024. Xiao et al. (2024) Xiao, T., Yuan, Y., Zhu, H., Li, M., and Honavar, V. G. Cal-dpo: Calibrated direct preference optimization for language model alignment. arXiv preprint arXiv:2412.14516, 2024. Xu et al. (2024) Xu, Z., Huang, R., Chen, C., and Wang, X. Uncovering safety risks of large language models through concept activation vector. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. Yao et al. (2024) Yao, Y., Xu, X., and Liu, Y. Large language model unlearning. In Annual Conference on Neural Information Processing Systems, 2024. Zhang et al. (2024) Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024. Zhou et al. (2024) Zhou, Z., Liu, Z., Liu, J., Dong, Z., Yang, C., and Qiao, Y. Weak-to-strong search: Align large language models via searching over small language models. arXiv preprint arXiv:2405.19262, 2024. Ziegler et al. (2019) Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. Zou et al. (2023) Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Appendix A Discussion on the Definition of Unlearning in LLMs In this section, we review and summarize the definitions of existing unlearning methods for LLMs and attempt to incorporate these methods into a unified theoretical framework. Assume the training dataset is denoted as t=f∪rsubscriptsubscriptsubscriptD_t=D_f _rDitalic_t = Ditalic_f ∪ Ditalic_r, where fsubscriptD_fDitalic_f represents the set of samples to be unlearned, and rsubscriptD_rDitalic_r represents the remaining samples. The core objective of LLM unlearning is to remove the knowledge learned from fsubscriptD_fDitalic_f while preserving the model’s other capabilities as much as possible. To achieve this goal, existing methods can be broadly categorized into two main classes: (i) gradient-based methods and (i) preference optimization-based methods. Gradient-based methods. Gradient-based methods include gradient ascent and its various extensions. Below, we will review the definitions of these methods sequentially. Gradient ascent. Gradient ascent (Jang et al., 2023; Feng et al., 2024) is a traditional and straightforward baseline method that removes the model’s memory of the samples in fsubscriptD_fDitalic_f by maximizing the loss on fsubscriptD_fDitalic_f, effectively reversing the gradient descent process. It is defined as: ℒGA=1|f|⁢∑i=1|f|∑t=1nilog⁡p⁢(xt∣<t;)⏟ℒforget⁢(f;),subscriptℒGAsubscript⏟1subscriptsuperscriptsubscript1subscriptsuperscriptsubscript1subscriptconditionalsubscriptsubscriptabsentsubscriptℒforgetsubscriptL_GA= 1|D_f| _i=1^|% D_f| _t=1^n_i p(x_t x_<t;% θ)_L_forget(D_f;% θ),Lroman_GA = under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG | Ditalic_f | end_ARG ∑i = 1| Ditalic_f | ∑t = 1nitalic_i log p ( xitalic_t ∣ italic_x< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT , where nisubscriptn_initalic_i denotes the number of tokens in the sample isuperscript x^iitalic_xitalic_i, and θitalic_θ represents the parameters of the model. Variants of gradient ascent. However, naive gradient ascent significantly degrades the model’s other capabilities. To address this issue, recent studies (Wang et al., 2023; Li et al., 2024; Maini et al., 2024; Ji et al., 2024b; Jia et al., 2024b; Liu et al., 2024a; Wang et al., 2024) have introduced various regularization terms, primarily including loss-based regularization and divergence-based regularization, as described below: • Loss-based regularization. Loss-based regularization (Li et al., 2024; Maini et al., 2024; Jia et al., 2024b) maintains the model’s other capabilities by sampling a dataset r′subscript′D_r Ditalic_r′ that shares the same distribution as rsubscriptD_rDitalic_r and minimizing the model’s loss on r′subscript′D_r Ditalic_r′. The formal expression is: ℒGA+LR=ℒGA⏟ℒforget⁢(f;)−λ⁢1|r′|⁢∑i=1|r′|∑t=1nilog⁡p⁢(xt∣<t;)⏟ℒreg⁢().subscriptℒGALRsubscript⏟subscriptℒGAsubscriptℒforgetsubscriptsubscript⏟1superscriptsubscript′subscript1superscriptsubscript′subscript1subscriptconditionalsubscriptsubscriptabsentsubscriptℒregL_GA+LR= L_GA_% L_forget(D_f; θ)-λ % 1|D_r | _i=1^|D_r |% _t=1^n_i p(x_t x_<t; θ)_% L_reg( θ).Lroman_GA + LR = under⏟ start_ARG Lroman_GA end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT - λ under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG | Ditalic_r′ | end_ARG ∑i = 1| Ditalic_r start_POSTSUPERSCRIPT ′ | end_POSTSUPERSCRIPT ∑t = 1nitalic_i log p ( xitalic_t ∣ italic_x< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT . • Divergence-based regularization. Similar to loss-based regularization, divergence-based regularization (Wang et al., 2023; Tian et al., 2024; Maini et al., 2024; Ji et al., 2024b; Wang et al., 2024) preserves model performance by constraining the output distribution of the model on a dataset r′subscript′D_r Ditalic_r′. Specifically, this method minimizes the distributional distance Dis(·∣·)Dis( )D i s ( · ∣ ∣ · ) between the output distribution of the unlearned model on r′subscript′D_r Ditalic_r′ and that of the original model on r′subscript′D_r Ditalic_r′. Depending on the metric used to measure the distributional distance, this method can further be categorized into regularizations based on KL divergence (Wang et al., 2023; Maini et al., 2024) and f-divergence (Wang et al., 2024). The formal definition is: ℒGA+DR=ℒGA⏟ℒforget⁢(f;)+λ⁢1|r′|∑i=1|r′|∑t=1niDis(P(·∣<t;)∣P(·∣<t;∗))⏟ℒreg⁢().L_GA+DR= L_GA_% L_forget(D_f; θ)+λ % 1|D_r | _i=1^|D_r |% _t=1^n_iDis(P( x_<t;% θ) P( x_<% t; θ^*))_L_reg( θ).Lroman_GA + DR = under⏟ start_ARG Lroman_GA end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT + λ under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG | Ditalic_r′ | end_ARG ∑i = 1| Ditalic_r start_POSTSUPERSCRIPT ′ | end_POSTSUPERSCRIPT ∑t = 1nitalic_i D i s ( P ( · ∣ italic_x< t ; italic_θ ) ∣ ∣ P ( · ∣ italic_x< t ; italic_θ∗ ) ) end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT . Both loss regularization and divergence regularization can essentially be regarded as forms of parameter regularization, which constrain the norm of the difference between model parameters before and after unlearning to be less than a threshold δ. By restricting the parameter changes within a δ-norm ball, this technique ensures the preservation of the model’s other capabilities. However, parameter regularization is difficult to handle directly as a constraint, so its relaxed form is often utilized and incorporated into the objective function instead. Formally, this can be expressed as: ℒGA+PR=ℒGA⏟ℒforget⁢(f;)+λ⁢‖−∗‖p2⏟ℒreg⁢().subscriptℒGAPRsubscript⏟subscriptℒGAsubscriptℒforgetsubscriptsubscript⏟subscriptsuperscriptnormsuperscript2subscriptℒregL_GA+PR= L_GA_% L_forget(D_f; θ)+λ % \| θ- θ^*\|^2_p_L_% reg( θ).Lroman_GA + PR = under⏟ start_ARG Lroman_GA end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT + λ under⏟ start_ARG ∥ italic_θ - italic_θ∗ ∥2p end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT . In addition, while several recent methods (i.e., Mismatch and LLMU) (Yao et al., 2024) differ in their definitions of unlearning objectives, they are fundamentally variants of gradient ascent methods. These methods further refine gradient ascent by extending the formulation of unlearning objectives, constructing a random combination of text sequences YransubscriptranY_ranYroman_ran. Specifically, their definitions are given as: ℒMIS=−1|f|⁢∑i=1|f|∑t=1ni1|Yran|⁢∑j=1|Yran|log⁡p⁢(yranj∣<t;)⏟ℒforget⁢(f;)−λ⁢1|r′|⁢∑i=1|r′|∑t=1nilog⁡p⁢(xt∣<t;)⏟ℒreg⁢().subscriptℒMISsubscript⏟1subscriptsuperscriptsubscript1subscriptsuperscriptsubscript1subscript1subscriptransuperscriptsubscript1subscriptranconditionalsuperscriptsubscriptransubscriptabsentsubscriptℒforgetsubscriptsubscript⏟1superscriptsubscript′subscript1superscriptsubscript′subscript1subscriptconditionalsubscriptsubscriptabsentsubscriptℒregL_MIS= - 1|D_f| _i=1^% |D_f| _t=1^n_i 1|Y_ran| _j=1^|% Y_ran| p(y_ran^j x_<t;% θ)_L_forget(D_f;% θ)-λ 1|D_r |% _i=1^|D_r | _t=1^n_i p(x_t % x_<t; θ)_L_reg(% θ).Lroman_MIS = under⏟ start_ARG - divide start_ARG 1 end_ARG start_ARG | Ditalic_f | end_ARG ∑i = 1| Ditalic_f | ∑t = 1nitalic_i divide start_ARG 1 end_ARG start_ARG | Yroman_ran | end_ARG ∑j = 1| Yroman_ran | log p ( yroman_ranitalic_j ∣ italic_x< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT - λ under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG | Ditalic_r′ | end_ARG ∑i = 1| Ditalic_r start_POSTSUPERSCRIPT ′ | end_POSTSUPERSCRIPT ∑t = 1nitalic_i log p ( xitalic_t ∣ italic_x< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT . ℒLLMU=ℒGA−1|f|⁢∑i=1|f|∑t=1ni1|Yran|⁢∑j=1|Yran|log⁡p⁢(yranj∣<t;)⏟ℒforget⁢(f;)+λ⁢1|r′|∑i=1|r′|∑t=1niDis(P(·∣<t;)∣P(·∣<t;∗))⏟ℒreg⁢().L_LLMU= L_GA- 1|% D_f| _i=1^|D_f| _t=1^n_i 1|Y_% ran| _j=1^|Y_ran| p(y_ran^j % x_<t; θ)_L_forget(% D_f; θ)+λ 1|D% _r | _i=1^|D_r | _t=1^n_iDis(P(% x_<t; θ) P% ( x_<t; θ^*))_% L_reg( θ).Lroman_LLMU = under⏟ start_ARG Lroman_GA - divide start_ARG 1 end_ARG start_ARG | Ditalic_f | end_ARG ∑i = 1| Ditalic_f | ∑t = 1nitalic_i divide start_ARG 1 end_ARG start_ARG | Yroman_ran | end_ARG ∑j = 1| Yroman_ran | log p ( yroman_ranitalic_j ∣ italic_x< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT + λ under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG | Ditalic_r′ | end_ARG ∑i = 1| Ditalic_r start_POSTSUPERSCRIPT ′ | end_POSTSUPERSCRIPT ∑t = 1nitalic_i D i s ( P ( · ∣ italic_x< t ; italic_θ ) ∣ ∣ P ( · ∣ italic_x< t ; italic_θ∗ ) ) end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT . Preference optimization-based methods. Preference optimization-based unlearning methods for LLMs primarily include DPO (Lu et al., 2022; Maini et al., 2024; Dige et al., 2024) and its variants, as well as Negative Preference Optimization (NPO) (Zhang et al., 2024; Fan et al., 2024). These methods achieve unlearning by constructing additional preference data pairs and leveraging existing preference optimization algorithms to guide the model. DPO method. The DPO method constructs preference data pairs based on the unlearning sample set (Lu et al., 2022; Maini et al., 2024; Dige et al., 2024). For example, given a sample isuperscript x^iitalic_xitalic_i containing nisubscriptn_initalic_i combinations, for any combination pair (<ti,xt)subscriptsuperscriptabsentsubscript( x^i_<t,x_t)( italic_xitalic_i< t , xitalic_t ), where xtsubscriptx_txitalic_t is the truthful response, DPO sets xt′subscript′x_t xitalic_t′ as ”refuse to answer” and treats it as the preferred response. By optimizing this preference pair, DPO employs preference optimization algorithms to achieve unlearning. It is formally defined as: ℒDPO=−2β⁢i∈f⁢[log⁡σ⁢(−β⁢∑i=1nilog⁡p⁢(xt∣<t;)⏟ℒforget⁢(f;)+β⁢∑i=1nilog⁡p⁢(xt′∣<t;)−Mref⏟ℒreg⁢())].subscriptℒDPO2subscriptsuperscriptsubscriptdelimited-[]subscript⏟superscriptsubscript1subscriptconditionalsubscriptsubscriptabsentsubscriptℒforgetsubscriptsubscript⏟superscriptsubscript1subscriptconditionalsuperscriptsubscript′subscriptabsentsubscriptrefsubscriptℒregL_DPO=- 2βE_ x^i∈% D_f [ σ ( -β _i=1^n_i p% (x_t x_<t; θ)_L_% forget(D_f; θ)+ β _i=1^n_% i p(x_t x_<t; θ)-M_% ref_L_reg( θ) ) ].Lroman_DPO = - divide start_ARG 2 end_ARG start_ARG β end_ARG blackboard_Eitalic_xitalic_i ∈ D start_POSTSUBSCRIPT f end_POSTSUBSCRIPT [ log σ ( under⏟ start_ARG - β ∑i = 1nitalic_i log p ( xitalic_t ∣ italic_x< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT forget ( Ditalic_f ; italic_θ ) end_POSTSUBSCRIPT + under⏟ start_ARG β ∑i = 1nitalic_i log p ( xitalic_t′ ∣ italic_x< t ; italic_θ ) - Mroman_ref end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT ) ] . NPO method. The NPO method directly treats the unlearning samples as negative samples and penalizes the model’s responses on the unlearning set fsubscriptD_fDitalic_f (Zhang et al., 2024; Fan et al., 2024). The formal definition is: ℒNPO=−2β⁢i∈f⁢[log⁡σ⁢(−β⁢P⁢(xt∣<ti;∗)⏟ℒforget⁢(forget;)+β⁢log⁡P⁢(xt∣<ti;)⏟ℒreg⁢())]subscriptℒNPO2subscriptsuperscriptsubscriptdelimited-[]subscript⏟conditionalsubscriptsubscriptsuperscriptabsentsuperscriptsubscriptℒforgetforgetsubscript⏟conditionalsubscriptsubscriptsuperscriptabsentsubscriptℒregL_NPO=- 2βE_ x^i∈% D_f [ σ ( -β P(x_t % x^i_<t; θ^*)_L_forget% (forget; θ)+ β P(x_t % x^i_<t; θ)_L_reg(% θ) ) ]Lroman_NPO = - divide start_ARG 2 end_ARG start_ARG β end_ARG blackboard_Eitalic_xitalic_i ∈ D start_POSTSUBSCRIPT f end_POSTSUBSCRIPT [ log σ ( under⏟ start_ARG - β P ( xitalic_t ∣ italic_xitalic_i< t ; italic_θ∗ ) end_ARGL start_POSTSUBSCRIPT forget ( forget ; italic_θ ) end_POSTSUBSCRIPT + under⏟ start_ARG β log P ( xitalic_t ∣ italic_xitalic_i< t ; italic_θ ) end_ARGL start_POSTSUBSCRIPT reg ( italic_θ ) end_POSTSUBSCRIPT ) ] In summary, both gradient-based methods and preference optimization-based methods can be viewed as combinations of unlearning loss and regularization loss. Appendix B Theoretical Analysis B.1 Proof of Proposition 7 To analyze the variation of ⁢(∗⁢(ϵ))superscriptitalic-ϵJ( θ^*(ε))J ( italic_θ∗ ( ϵ ) ), we perform a Taylor expansion of Eq. (6) around ϵ=0italic-ϵ0ε=0ϵ = 0. Here, since we are more concerned with the description of the magnitude of the effect rather than the exact values, we expand it only to the first-order term, yielding: ⁢(∗⁢(ϵ))≈⁢(∗⁢(0))+ϵ⁢∂⁢(∗⁢(ϵ))∂ϵ|ϵ=0,superscriptitalic-ϵsuperscript0evaluated-atitalic-ϵsuperscriptitalic-ϵitalic-ϵitalic-ϵ0J( θ^*(ε)) ( % θ^*(0))+ε ( θ^*(% ε))∂ε|_ε=0,J ( italic_θ∗ ( ϵ ) ) ≈ J ( italic_θ∗ ( 0 ) ) + ϵ divide start_ARG ∂ J ( italic_θ∗ ( ϵ ) ) end_ARG start_ARG ∂ ϵ end_ARG |ϵ = 0 , where ⁢(∗⁢(0))superscript0J( θ^*(0))J ( italic_θ∗ ( 0 ) ) represents the preference alignment performance of the model at ϵ=0italic-ϵ0ε=0ϵ = 0 (i.e., without unlearning), while ∂⁢(∗⁢(ϵ))∂ϵsuperscriptitalic-ϵitalic-ϵ ( θ^*(ε))∂εdivide start_ARG ∂ J ( italic_θ∗ ( ϵ ) ) end_ARG start_ARG ∂ ϵ end_ARG denotes the rate of change in the PA performance w.r.t. the unlearning weight control parameter ϵitalic-ϵεϵ. According to the chain rule, the partial derivative can be decomposed as: ∂⁢(∗⁢(ϵ))∂ϵ=∇⁢(∗⁢(ϵ))⊤⁢∂∗⁢(ϵ)∂ϵ.superscriptitalic-ϵitalic-ϵsubscript∇superscriptsuperscriptitalic-ϵtopsuperscriptitalic-ϵitalic-ϵ ( θ^*(ε))∂ε% = _ θJ( θ^*(ε))^% ∂ θ^*(ε)∂ε.divide start_ARG ∂ J ( italic_θ∗ ( ϵ ) ) end_ARG start_ARG ∂ ϵ end_ARG = ∇italic_θ J ( italic_θ∗ ( ϵ ) )⊤ divide start_ARG ∂ italic_θ∗ ( ϵ ) end_ARG start_ARG ∂ ϵ end_ARG . Since the optimal solution ∗⁢(ϵ)superscriptitalic-ϵ θ^*(ε)italic_θ∗ ( ϵ ) of the lower-level problem satisfies the first-order optimality condition: ∇(ϵ⁢ℒforget⁢(;)+ℒreg⁢())|=∗⁢(ϵ)=0.evaluated-atsubscript∇italic-ϵsubscriptℒforgetsubscriptℒregsuperscriptitalic-ϵ0 _ θ ( _forget(% x; θ)+L_reg( % θ) )|_ θ= θ^*(ε)=0.∇italic_θ ( ϵ Lroman_forget ( italic_x ; italic_θ ) + Lroman_reg ( italic_θ ) ) |italic_θ = italic_θ∗ ( ϵ ) = 0 . By differentiating the above optimality condition w.r.t ϵitalic-ϵεϵ, we obtain: ϵ⁢∇2ℒforget⁢(;∗⁢(ϵ))⁢∂∗⁢(ϵ)∂ϵ+∇ℒforget⁢(;∗⁢(ϵ))+∇2ℒreg⁢(∗⁢(ϵ))⁢∂∗⁢(ϵ)∂ϵ=0.italic-ϵsuperscriptsubscript∇2subscriptℒforgetsuperscriptitalic-ϵsuperscriptitalic-ϵitalic-ϵsubscript∇subscriptℒforgetsuperscriptitalic-ϵsuperscriptsubscript∇2subscriptℒregsuperscriptitalic-ϵsuperscriptitalic-ϵitalic-ϵ0ε _ θ^2L_forget(% x; θ^*(ε)) ∂ % θ^*(ε)∂ε+ _ θ% L_forget( x; θ^*(ε))+ _% θ^2L_reg( θ^*(% ε)) ∂ θ^*(ε)∂ε=0.ϵ ∇italic_θ2 Lroman_forget ( italic_x ; italic_θ∗ ( ϵ ) ) divide start_ARG ∂ italic_θ∗ ( ϵ ) end_ARG start_ARG ∂ ϵ end_ARG + ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ( ϵ ) ) + ∇italic_θ2 Lroman_reg ( italic_θ∗ ( ϵ ) ) divide start_ARG ∂ italic_θ∗ ( ϵ ) end_ARG start_ARG ∂ ϵ end_ARG = 0 . Substituting ℒreg⁢()=‖−∗‖2subscriptℒregsuperscriptnormsuperscript2L_reg( θ)=\| θ-% θ^*\|^2Lroman_reg ( italic_θ ) = ∥ italic_θ - italic_θ∗ ∥2, we have: ∇ℒreg⁢(∗⁢(ϵ))=2⁢(∗⁢(ϵ)−∗),∇2ℒreg⁢(∗⁢(ϵ))=2⁢I.formulae-sequencesubscript∇subscriptℒregsuperscriptitalic-ϵ2superscriptitalic-ϵsuperscriptsuperscriptsubscript∇2subscriptℒregsuperscriptitalic-ϵ2 _ θL_reg( θ^*% (ε))=2( θ^*(ε)- θ^*), % _ θ^2L_reg( θ% ^*(ε))=2I.∇italic_θ Lroman_reg ( italic_θ∗ ( ϵ ) ) = 2 ( italic_θ∗ ( ϵ ) - italic_θ∗ ) , ∇italic_θ2 Lroman_reg ( italic_θ∗ ( ϵ ) ) = 2 I . Therefore, the implicit gradient formula is given by: ∂∗⁢(ϵ)∂ϵ=−[ϵ⁢∇2ℒforget⁢(;∗⁢(ϵ))+2⁢I]−1⁢∇ℒforget⁢(;∗⁢(ϵ)).superscriptitalic-ϵitalic-ϵsuperscriptdelimited-[]italic-ϵsuperscriptsubscript∇2subscriptℒforgetsuperscriptitalic-ϵ21subscript∇subscriptℒforgetsuperscriptitalic-ϵ ∂ θ^*(ε)∂ε=- [% ε _ θ^2L_forget(% x; θ^*(ε))+2I ]^-1 _% θL_forget( x; % θ^*(ε)).divide start_ARG ∂ italic_θ∗ ( ϵ ) end_ARG start_ARG ∂ ϵ end_ARG = - [ ϵ ∇italic_θ2 Lroman_forget ( italic_x ; italic_θ∗ ( ϵ ) ) + 2 I ]- 1 ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ( ϵ ) ) . When ϵ=0italic-ϵ0ε=0ϵ = 0, the formula simplifies to: ∂∗⁢(ϵ)∂ϵ|ϵ=0=−[2⁢I]−1⁢∇ℒforget⁢(;∗⁢(0))=−12⁢∇ℒforget⁢(;∗).evaluated-atsuperscriptitalic-ϵitalic-ϵitalic-ϵ0superscriptdelimited-[]21subscript∇subscriptℒforgetsuperscript012subscript∇subscriptℒforgetsuperscript ∂ θ^*(ε)∂ε|_ε=% 0=- [2I ]^-1 _ θL_% forget( x; θ^*(0))=- 12 _% θL_forget( x; % θ^*).divide start_ARG ∂ italic_θ∗ ( ϵ ) end_ARG start_ARG ∂ ϵ end_ARG |ϵ = 0 = - [ 2 I ]- 1 ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ( 0 ) ) = - divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ) . Substituting ∂∗⁢(ϵ)∂ϵsuperscriptitalic-ϵitalic-ϵ ∂ θ^*(ε)∂εdivide start_ARG ∂ italic_θ∗ ( ϵ ) end_ARG start_ARG ∂ ϵ end_ARG into the chain rule formula: ∂⁢(∗⁢(ϵ))∂ϵ|ϵ=0=∇⁢(∗)⊤⁢(−12⁢∇ℒforget⁢(;∗)).evaluated-atsuperscriptitalic-ϵitalic-ϵitalic-ϵ0subscript∇superscriptsuperscripttop12subscript∇subscriptℒforgetsuperscript ( θ^*(ε))∂ε% |_ε=0= _ θJ( θ^*% ) (- 12 _ θL_% forget( x; θ^*) ).divide start_ARG ∂ J ( italic_θ∗ ( ϵ ) ) end_ARG start_ARG ∂ ϵ end_ARG |ϵ = 0 = ∇italic_θ J ( italic_θ∗ )⊤ ( - divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ) ) . Therefore, the variation in preference alignment performance is: ⁢(∗⁢(ϵ))−⁢(∗⁢(0))≈−ϵ2⁢∇⁢(∗)⊤⁢∇ℒforget⁢(;∗).superscriptitalic-ϵsuperscript0italic-ϵ2subscript∇superscriptsuperscripttopsubscript∇subscriptℒforgetsuperscriptJ( θ^*(ε))-J( θ% ^*(0))≈- ε2 _ θJ(% θ^*) _ θL_% forget( x; θ^*).J ( italic_θ∗ ( ϵ ) ) - J ( italic_θ∗ ( 0 ) ) ≈ - divide start_ARG ϵ end_ARG start_ARG 2 end_ARG ∇italic_θ J ( italic_θ∗ )⊤ ∇italic_θ Lroman_forget ( italic_x ; italic_θ∗ ) . B.2 Derivation of Marginal Gain To optimize the outer problem, we need to compute the gradient of the objective function w.r.t. the weight vector ωitalic_ω: ∇g⁢()subscript∇ _ ωg( ω)∇italic_ω g ( italic_ω ) =−∇⁢(∗⁢())+β⁢∑i=1nabsentsubscript∇superscriptsuperscriptsubscript1 =- _ ωJ( θ^*% ( ω))+β _i=1^n ω= - ∇italic_ω J ( italic_θ∗ ( italic_ω ) ) + β ∑i = 1n square-root start_ARG italic_ω end_ARG =−∂⁢(∗⁢())∂⁢∂∗⁢()∂+[β2⁢1,β2⁢2,…,β2⁢n]⊤.absentsuperscriptsuperscriptsuperscript2subscript12subscript2…2subscripttop =- ( θ^*( % ω))∂ θ ∂ θ^*(% ω)∂ ω+ [ β2 % ω_1, β2 ω_2,…% , β2 ω_n ] .= - divide start_ARG ∂ J ( italic_θ∗ ( italic_ω ) ) end_ARG start_ARG ∂ italic_θ end_ARG divide start_ARG ∂ italic_θ∗ ( italic_ω ) end_ARG start_ARG ∂ italic_ω end_ARG + [ divide start_ARG β end_ARG start_ARG 2 square-root start_ARG italic_ω1 end_ARG end_ARG , divide start_ARG β end_ARG start_ARG 2 square-root start_ARG italic_ω2 end_ARG end_ARG , … , divide start_ARG β end_ARG start_ARG 2 square-root start_ARG italic_ωitalic_n end_ARG end_ARG ]⊤ . (13) Since the solution of the inner optimization problem ∗⁢()superscript θ^*( ω)italic_θ∗ ( italic_ω ) satisfies the first-order necessary condition: ∇f⁢(∗⁢(),)=0,subscript∇superscript0 _ θf( θ^*( ω),% ω)=0,∇italic_θ f ( italic_θ∗ ( italic_ω ) , italic_ω ) = 0 , which is equivalent to ∇(∑i=1ni⁢ℓi⁢(∗⁢())+λ⁢‖∗⁢()−∗‖2)=0.subscript∇superscriptsubscript1subscriptsubscriptℓsuperscriptsuperscriptnormsuperscriptsuperscript20 _ θ ( _i=1^n ω_i _% i( θ^*( ω))+λ\| θ% ^*( ω)- θ^*\|^2 )=0.∇italic_θ ( ∑i = 1n italic_ωitalic_i ℓitalic_i ( italic_θ∗ ( italic_ω ) ) + λ ∥ italic_θ∗ ( italic_ω ) - italic_θ∗ ∥2 ) = 0 . Taking the derivative w.r.t. ωitalic_ω, and using the implicit function theorem, we obtain: ∂∗⁢()∂superscript ∂ θ^*( ω)% ∂ ωdivide start_ARG ∂ italic_θ∗ ( italic_ω ) end_ARG start_ARG ∂ italic_ω end_ARG =−(∂2f∂2)−1⁢∂2f∂⁢∂absentsuperscriptsuperscript2superscript21superscript2 =- ( ∂^2f∂ θ^2% )^-1 ∂^2f∂ θ∂% ω= - ( divide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG )- 1 divide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ ∂ italic_ω end_ARG =−(∂2f∂2)−1⁢∂⁢[∑i=1ni⁢∂ℓi⁢()∂+2⁢λ⁢(−∗)],absentsuperscriptsuperscript2superscript21delimited-[]superscriptsubscript1subscriptsubscriptℓ2superscript =- ( ∂^2f∂ θ^2% )^-1 ∂ ω [ _i=1^n% ω_i ∂ _i( θ)∂% θ+2λ( θ- θ^*)% ],= - ( divide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG )- 1 divide start_ARG ∂ end_ARG start_ARG ∂ italic_ω end_ARG [ ∑i = 1n italic_ωitalic_i divide start_ARG ∂ ℓitalic_i ( italic_θ ) end_ARG start_ARG ∂ italic_θ end_ARG + 2 λ ( italic_θ - italic_θ∗ ) ] , (14) where ∂2f∂2=∑i=1nt−1,i∗⁢∇2ℓi⁢(∗⁢())+2⁢λ⁢Isuperscript2superscript2superscriptsubscript1superscriptsubscriptsubscript1superscriptsubscript∇2subscriptℓsuperscript2 ∂^2f∂ θ^2= _i=1^n% ω_S_t-1,i^* _ θ^2% _i( θ^*( ω))+2λ Idivide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG = ∑i = 1n italic_ωcaligraphic_S start_POSTSUBSCRIPT t - 1 , i end_POSTSUBSCRIPT∗ ∇italic_θ2 ℓitalic_i ( italic_θ∗ ( italic_ω ) ) + 2 λ I denotes the Hessian matrix of the inner optimization problem. Substituting Eq. (14) into Eq. (13): ∇g⁢()=∂⁢(∗⁢())∂⁢(∂2f∂2)−1⁢∂⁢[∑i=1ni⁢∂ℓi⁢()∂+2⁢λ⁢(−∗)]+[β2⁢1,β2⁢2,…,β2⁢n]⊤.subscript∇superscriptsuperscriptsuperscript2superscript21delimited-[]superscriptsubscript1subscriptsubscriptℓ2superscriptsuperscript2subscript12subscript2…2subscripttop _ ωg( ω)= (% θ^*( ω))∂ θ% ( ∂^2f∂ θ^2 )^-1 % ∂ ω [ _i=1^n ω% _i ∂ _i( θ)∂ θ% +2λ( θ- θ^*) ]+ [ % β2 ω_1, β2 ω% _2,…, β2 ω_n ] .∇italic_ω g ( italic_ω ) = divide start_ARG ∂ J ( italic_θ∗ ( italic_ω ) ) end_ARG start_ARG ∂ italic_θ end_ARG ( divide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG )- 1 divide start_ARG ∂ end_ARG start_ARG ∂ italic_ω end_ARG [ ∑i = 1n italic_ωitalic_i divide start_ARG ∂ ℓitalic_i ( italic_θ ) end_ARG start_ARG ∂ italic_θ end_ARG + 2 λ ( italic_θ - italic_θ∗ ) ] + [ divide start_ARG β end_ARG start_ARG 2 square-root start_ARG italic_ω1 end_ARG end_ARG , divide start_ARG β end_ARG start_ARG 2 square-root start_ARG italic_ω2 end_ARG end_ARG , … , divide start_ARG β end_ARG start_ARG 2 square-root start_ARG italic_ωitalic_n end_ARG end_ARG ]⊤ . Now, consider the contribution of the k-th component of the weight vector ωitalic_ω to g⁢()g( ω)g ( italic_ω ), i.e., computing ∂g⁢()ksubscript ∂ g( ω) ω_kdivide start_ARG ∂ g ( italic_ω ) end_ARG start_ARG italic_ωitalic_k end_ARG. Since only when i=ki=ki = k, the term corresponding to ksubscript ω_kitalic_ωitalic_k contributes, we derive: Δ⁢g⁢(k)=∂g⁢()k=∇⁢(∗⁢())⁢(∂2f∂2)−1⁢∇ℓk⁢(∗⁢())+β2⁢k−12.Δsubscriptsubscript∇superscriptsuperscriptsuperscript2superscript21subscript∇subscriptℓsuperscript2superscriptsubscript12 g(k)= ∂ g( ω) ω_k=% _ θJ( θ^*( % ω)) ( ∂^2f∂ θ^2 )^-% 1 _ θ _k( θ^*( % ω))+ β2 ω_k^- 12.Δ g ( k ) = divide start_ARG ∂ g ( italic_ω ) end_ARG start_ARG italic_ωitalic_k end_ARG = ∇italic_θ J ( italic_θ∗ ( italic_ω ) ) ( divide start_ARG ∂2 f end_ARG start_ARG ∂ italic_θ2 end_ARG )- 1 ∇italic_θ ℓitalic_k ( italic_θ∗ ( italic_ω ) ) + divide start_ARG β end_ARG start_ARG 2 end_ARG italic_ωitalic_k- divide start_ARG 1 end_ARG start_ARG 2 end_ARG . B.3 Derivation of Set Size The suboptimality bound provided by Theorem 4.3 is given as: g⁢(t∗)−g⁢(∗)≤8⁢L+4⁢ε1t+3.subscriptsuperscriptsubscriptsuperscript84subscript13g( ω^*_S_t)-g( ω^*)≤% 8L+4 _1t+3.g ( italic_ω∗caligraphic_S start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ) - g ( italic_ω∗ ) ≤ divide start_ARG 8 L + 4 ε1 end_ARG start_ARG t + 3 end_ARG . To satisfy the suboptimality error constraint, i.e., g⁢(t∗)−g⁢(∗)≤εsubscriptsuperscriptsubscriptsuperscriptg( ω^*_S_t)-g( ω^*)≤ ( italic_ω∗caligraphic_S start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ) - g ( italic_ω∗ ) ≤ ε, it suffices to ensure that the right-hand side of the suboptimality bound is less than or equal to ε ε, which gives: 8⁢L+4⁢ε1t+3≤ε84subscript13 8L+4 _1t+3≤ start_ARG 8 L + 4 ε1 end_ARG start_ARG t + 3 end_ARG ≤ ε. By moving t+33t+3t + 3 to the right-hand side and expanding the terms on the right, we obtain: ε⁢t≥8⁢L+4⁢ε1−3⁢ε84subscript13 t≥ 8L+4 _1-3 ε t ≥ 8 L + 4 ε1 - 3 ε. Neglecting −3⁢ε3-3 - 3 ε (as its impact diminishes with increasing t), the expression is further simplified to: ε⁢t≥8⁢L+4⁢ε184subscript1 t≥ 8L+4 _1ε t ≥ 8 L + 4 ε1. Moving ε ε to the right-hand side yields: t≥8⁢L+4⁢ε1ε84subscript1t≥ 8L+4 _1 t ≥ divide start_ARG 8 L + 4 ε1 end_ARG start_ARG ε end_ARG. This indicates that, to satisfy the suboptimality error constraint g⁢(t∗)−g⁢(∗)≤εsubscriptsuperscriptsubscriptsuperscriptg( ω^*_S_t)-g( ω^*)≤ ( italic_ω∗caligraphic_S start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ) - g ( italic_ω∗ ) ≤ ε, the number of iterations t must be at least: t=⁢(L+ε1ε)subscript1t=O( L+ _1 )t = O ( divide start_ARG L + ε1 end_ARG start_ARG ε end_ARG ). Appendix C Additional Experimental Details C.1 Dataset Configurations The detailed partitioning for each dataset is as follows: • For the PKU SafeRLHF, samples of type harmful or risk-inducing across at least one of the 19 harm categories are defined as negative examples. • For the UltraFeedback Binarized, samples of type low ratings in instruction following, truthfulness, honesty, or helpfulness annotations are considered negative examples. • For the HaluEval, samples meeting the unverifiable, non-factual, or irrelevant hallucination annotations condition are categorized as negative examples. C.2 Evaluation Configurations In this section, we provide a detailed explanation of each evaluation metric. For the performance of PA, we utilize the following four evaluation metrics: • Reward-value (Chakraborty et al., 2024; Yao et al., 2024). Reward-value assesses the quality of the model’s outputs based on reward scores assigned by the reward model. Higher values for these two metrics indicate better PA performance of the model after unlearning. • ASR (Xu et al., 2024). ASR measures the model’s tendency to generate potentially harmful content. In our experiments, ASR is further divided into four sub-dimensions (Xu et al., 2024; Zou et al., 2023): ASR-keyword, ASR-answer, ASR-useful, and ASR-summary. Smaller values for these metrics indicate better PA performance of the model after unlearning. • Coherence (Chakraborty et al., 2024; Khanov et al., 2024; Kong et al., 2024). Coherence is evaluated by calculating the cosine similarity between the SimCSE (Su et al., 2022) embeddings of each prompt and its generated response, assessing their semantic proximity (Chakraborty et al., 2024). Higher coherence indicates better PA performance. • Win-rate (Xiao et al., 2024; Rafailov et al., 2024). Win-rate measures the proportion of instances where the model’s outputs are preferred over those of the baseline model (GPT-4). • Hallucination-rate (Yao et al., 2024). Hallucination-rate measures the frequency of false or factually incorrect information in the outputs. A lower hallucination-rate indicates better PA performance. For the performance of unlearning, we utilize the following three evaluation metrics: • Membership Inference Attack (MIA) (Jia et al., 2024a). We employ the Min-k%percentk\%k % Prob (Shi et al., 2023) method to calculate the likelihood of a given text belonging to the training data. The specific metric used is the area under the ROC curve (AUC). A higher AUC value indicates that the model is better at distinguishing between training and non-training data, thereby demonstrating greater unlearning effectiveness. • Perplexity (PPL) (Yao et al., 2024; Doshi & Stickland, 2024). Lower PPL values indicate higher quality of the generated text, with outputs being more fluent and consistent with natural language expression norms. Appendix D Additional Experiments We validated the effectiveness of U2A on the UltraFeedback Binarized and HaluEval datasets. The experimental results are shown in the table below. Table 3: Comparison of the U2A framework with the current PA and unlearning baseline methods on the UltraFeedback Binarized dataset. Optimal results are highlighted in bold. Models Methods PA Performance MU Performance Length-control Win Rate (↑ ↑) Win Rate vs. GPT-4 (↑ ↑) Coherence (↑ ↑) MIA (↑ ↑) PPL (↓ ↓) Llama-2-7B-Chat Original 0.0818 0.0534 0.7519 0.5287 9.3005 Retrain 0.0770 0.0309 0.7571 0.5058 9.3588 PPO 0.0916 0.0586 0.7643 0.5618 11.3477 DPO 0.0886 0.0620 0.7618 0.5410 10.1612 GA 0.0792 0.0523 0.7516 0.5337 8.4104 GradDiff 0.0822 0.0653 0.7632 0.5735 9.0017 NPO 0.0808 0.0581 0.7574 0.5288 10.1905 GA + U2A 0.0985 0.0697 0.7632 0.5244 8.3768 GradDiff + U2A 0.1002 0.0711 0.7678 0.5845 8.8979 NPO + U2A 0.0992 0.0724 0.7603 0.5505 9.4683 Llama-3.1-8B-Instruct Original 0.2330 0.2281 0.7508 0.5358 2.6358 Retrain 0.2411 0.2363 0.7521 0.5401 2.6408 PPO 0.2453 0.2325 0.7588 0.5327 2.8744 DPO 0.2597 0.2447 0.7603 0.5358 2.9620 GA 0.2374 0.2338 0.7498 0.5313 2.6286 GradDiff 0.2493 0.2456 0.7513 0.5364 2.6357 NPO 0.2042 0.1751 0.7492 0.3812 3.1217 GA + U2A 0.2522 0.2579 0.7612 0.5596 2.3941 GradDiff + U2A 0.2515 0.2585 0.7572 0.5458 1.6617 NPO + U2A 0.2391 0.2293 0.7601 0.5192 2.9896 Table 4: Comparison of the U2A framework with the current PA and unlearning baseline methods on the HaluEval dataset. Optimal results are highlighted in bold. Models Methods PA Performance MU Performance Hallucination Rate MIA (↑ ↑) PPL (↓ ↓) F1 (↓ ↓) Precision (↓ ↓) Recall (↓ ↓) Llama-2-7B-Chat Original 65.05 72.50 53.70 0.3945 118.0564 Retrain 52.45 59.45 41.95 0.4006 98.8235 PPO 55.35 61.80 44.85 0.3489 140.6214 DPO 54.40 61.40 43.55 0.3985 116.9673 GA 60.80 61.75 49.40 0.3938 70.1965 GradDiff 59.34 62.45 45.77 0.3929 72.5341 NPO 65.15 61.95 43.90 0.3988 104.3602 GA + U2A 50.45 60.75 41.35 0.4061 66.2969 GradDiff + U2A 49.90 60.65 41.25 0.4165 64.1484 NPO + U2A 48.45 58.70 40.05 0.4065 67.8125 Llama-3.1-8B-Instruct Original 57.05 65.45 46.00 0.3903 7.1921 Retrain 52.90 58.20 47.45 0.3978 7.5665 PPO 55.85 62.95 45.70 0.3908 6.8384 DPO 56.55 64.80 45.65 0.4083 7.1947 GA 53.40 65.20 44.60 0.4345 8.9799 GradDiff 58.35 70.55 48.10 0.4565 7.4564 NPO 58.05 65.35 46.70 0.3908 12.4844 GA + U2A 50.65 58.10 40.95 0.4422 6.7682 GradDiff + U2A 49.75 56.35 42.70 0.4571 7.8976 NPO + U2A 48.15 55.40 43.05 0.4220 9.9792

Instant research discovery