Paper deep dive

Ethos: Rectifying Language Models in Orthogonal Parameter Space

Lei Gao, Yue Niu, Tingting Tang, Salman Avestimehr, Murali Annavaram

Year: 2024Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 85

Models: GPT2-1.5B, GPT-Neo, OPT-1.3B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 7:29:28 PM

Summary

Ethos is a novel model editing approach that mitigates toxicity, bias, and privacy leakage in language models by decomposing model weights into orthogonal principal components using singular value decomposition (SVD). By identifying and negating only the components associated with undesired knowledge, Ethos effectively unlearns harmful information while preserving general model utility, outperforming traditional task arithmetic methods.

Entities (5)

Ethos · method · 100%Singular Value Decomposition · algorithm · 100%Language Models · technology · 98%LoRA · technique · 95%Task Arithmetic · technique · 95%

Relation Signals (4)

Ethos → uses → Singular Value Decomposition

confidence 100% · Ethos first obtains a set of principal components from the pre-trained models using singular value decomposition.

Ethos → mitigates → Toxicity

confidence 95% · Ethos rectifies LMs to mitigate toxicity and bias in outputs and avoid privacy leakage.

Ethos → implements → LoRA

confidence 90% · In this work, we use LoRA fine-tuning instead of full model fine-tuning across all experiments.

Ethos → improvesupon → Task Arithmetic

confidence 90% · Evaluations show Ethos is more effective in removing undesired knowledge and maintaining the overall model performance compared to current task arithmetic methods.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Language models (LMs) have greatly propelled the research on natural language processing. However, LMs also raise concerns regarding the generation of biased or toxic content and the potential disclosure of private information from the training dataset. In this work, we present a new efficient approach, Ethos, that rectifies LMs to mitigate toxicity and bias in outputs and avoid privacy leakage. Ethos is built on task arithmetic. However, unlike current task arithmetic algorithms, Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors. Specifically, Ethos first obtains a set of principal components from the pre-trained models using singular value decomposition. Then, by projecting the task vector onto principal components, Ethos identifies the principal components that encode general or undesired knowledge. Ethos performs negating using the task vector with undesired knowledge only, thereby minimizing collateral damage on general model utility. We demonstrate the efficacy of our approach on three different tasks: debiasing, detoxification, and memorization unlearning. Evaluations show Ethos is more effective in removing undesired knowledge and maintaining the overall model performance compared to current task arithmetic methods.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

84,737 characters extracted from source content.

Expand or collapse full text

Ethos: Rectifying Language Models in Orthogonal Parameter Space Lei Gao ∗, Yue Niu , Tingting Tang, Salman Avestimehr, Murali Annavaram University of Southern California leig, yueniu, tangting, avestime, annavara@usc.edu These authors contributed equally. Abstract Language models (LMs) have greatly propelled the research on natural language processing. However, LMs also raise concerns regarding the generation of biased or toxic content and the potential disclosure of private information from the training dataset. In this work, we present a new efficient approach, Ethos, that rectifies LMs to mitigate toxicity and bias in outputs and avoid privacy leakage. Ethos is built on task arithmetic. However, unlike current task arithmetic algorithms, Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors. Specifically, Ethos first obtains a set of principal components from the pre-trained models using singular value decomposition. Then, by projecting the task vector onto principal components, Ethos separates the principal components that encode general from those associated with undesired knowledge. Ethos performs forgetting or unlearning by only negating the task vector with undesired knowledge, thereby minimizing collateral damage on general model utility. We demonstrate the efficacy of our approach on three different tasks: bias, toxicity, and memorization unlearning. Evaluations show Ethos is more effective in removing undesired knowledge while maintaining the overall model performance compared to current task arithmetic methods. Ethos: Rectifying Language Models in Orthogonal Parameter Space Lei Gao ∗, Yue Niu †thanks: These authors contributed equally., Tingting Tang, Salman Avestimehr, Murali Annavaram University of Southern California leig, yueniu, tangting, avestime, annavara@usc.edu 1 Introduction The advent of language models (LMs) has enhanced the current capabilities in text understanding and generation Vaswani et al. (2017); Brown et al. (2020); Touvron et al. (2023); Zhao et al. (2023). Due to their significant potential LMs have been the driving force in many automated systems that improve productivity in real-world tasks OpenAI (2023); Chen et al. (2021); Thoppilan et al. (2022). However, despite their success, LMs also bring to the forefront some new challenges. This paper focuses on one pivotal challenge among these: LMs’ propensity to generate toxic, biased content or reveal private training records. Overview of Toxicity/Bias/Privacy Concerns of LMs: Since LMs are pre-trained with a large volume of data, the composition of the dataset during pre-training can greatly affect the performance of LMs. In particular, suppose a dataset used in pre-training contains a substantial amount of toxic information, it can result in an LM that is likely to generate toxic or harmful messages for certain prompts Röttger et al. (2020); Hartvigsen et al. (2022). Similarly, an imbalanced dataset with unevenly distributed data points among groups (e.g., gender, race, ethnicity) can lead to the development of biases in LMs Bolukbasi et al. (2016); Dixon et al. (2018); Sheng et al. (2019); Gallegos et al. (2023). For instance, LMs may associate certain features with a gender group when pre-trained on gender-imbalanced datasets. Another critical concern in deploying LMs is the risk of privacy leakage due to model memorization. Specifically, LMs tend to overfit training data and memorize specific examples, increasing vulnerability to privacy breaches, such as training data extraction attacks Carlini et al. (2020, 2022); Hu et al. (2021); Flemings et al. (2024). Memorization compromises privacy and poses security risks, especially when the training data contains sensitive information. Addressing these challenges is crucial in the development of LMs. A naive approach is to retrain the model from scratch, for instance, whenever bias or memorization is discovered and removed from the training data. Considering the prohibitive costs of training LMs, it is infeasible to re-train the model. Hence, the objective of this work is to rectify LMs without incurring substantial costs. Overview of Model Editing by Task Arithmetic. Prior work Ilharco et al. (2023) introduces a model editing method that reduces toxic information in outputs by directly editing models with a task vector. The task vector, obtained after fine-tuning the model on a downstream dataset, encodes certain undesired knowledge (e.g., toxicity). Therefore, negating such a task vector helps rectify LMs and forgetting or unlearning undesired bias while maintaining reasonable model performance. To further improve the model editing performance, Zhang et al. leverage parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) Hu et al. (2022) to edit the task vector formed by a subset of the model weights using parameter-efficient modules only rather than the full model weights. Current model editing methods still struggle to maintain LMs’ performance when directly operating in the parameter space. The reason is that task vectors mix undesired knowledge with the general knowledge that is necessary for preserving model utility Hu et al. (2023). As a result, simply negating the task vector on an LM inevitably removes the general knowledge alongside the undesired knowledge, causing collateral damage to the overall model performance. We present more detailed related work in Appendix A. Overview of the Proposed Method. To address the limitations in current model editing methods for forgetting or unlearning undesired information, we propose Ethos, a new model editing method that generates task vectors containing undesired knowledge only and minimizes adverse effects on LMs’ performance. The core idea of Ethos is to analyze a model’s weights in an orthogonal space and distinguish the components related to general knowledge from the ones associated with undesired knowledge. We first define an orthogonal parameter space with a set of orthogonal components. Specifically, we apply singular value decomposition (SVD) to the pre-trained weights and obtain the principal components. The obtained principal components serve as the bases that fully represent the weight space of the pre-trained LM. Given the orthogonality of the principal components, we treat each as a separable component encoding specific orthogonal knowledge. The LM’s output represents a combination of knowledge from all principal components. To identify the components for undesired knowledge, we fine-tune the pre-trained LM on a downstream task, such as a toxic dataset, and obtain an initial task vector. Then, we project the task vector onto the defined orthogonal space. The principal components that present significant changes after the projection are classified as components encoding undesired knowledge, while others with marginal changes after the projection are classified as components for general knowledge. We use all components for undesired knowledge to construct a new task vector, which is then subtracted from the pre-trained weights to mitigate toxicity, bias, or memorization in the LM. We conduct experiments on three different tasks: bias, toxicity and memorization unlearning in LMs. We use pre-trained LMs, including OPT Zhang et al. (2022), GPT2 Radford et al. (2019), GPT-Neo Black et al. (2021), and large LMs like Llama2 Touvron et al. (2023). Evaluations show that Ethos effectively reduces bias, toxicity, and privacy leakage in pre-trained LMs. Notably, our approach demonstrates better unlearning performance than current model editing methods while maintaining model utility comparable to that of pre-trained models. We also conduct ablation studies to analyze various components of our methods. 2 Preliminary 2.1 Parameter-Efficient Fine-Tuning To enhance the efficiency of fine-tuning LMs while reducing memory and computational overhead, Parameter-efficient fine-tuning (PEFT) methods have been proposed to fine-tune only a subset of the existing model parameters Zaken et al. (2022); Houlsby et al. (2019); Li and Liang (2021). Among these, the low-rank adaptation algorithm, LoRA Hu et al. (2022), stands out for achieving performance comparable to full-parameter fine-tuning. For a linear layer, it freezes the pre-trained weights W0∈ℝd×ksubscript0superscriptℝW_0 ^d× kW0 ∈ blackboard_Rd × k and injects trainable low-rank matrices A∈ℝr×ksuperscriptℝA ^r× kA ∈ blackboard_Rr × k and B∈ℝd×rsuperscriptℝB ^d× rB ∈ blackboard_Rd × r, constraining the weight updates in a low-rank space. The total number of trainable parameters is significantly reduced given rank r≪m⁢i⁢n⁢(d,k)much-less-thanr min(d,k)r ≪ m i n ( d , k ). The forward pass is then modified as =W0⋅+B⁢A⋅,⋅subscript0⋅ h=W_0· x+BA· x,italic_h = W0 ⋅ italic_x + B A ⋅ italic_x , (1) where input ∈ℝksuperscriptℝ x ^kitalic_x ∈ blackboard_Rk and output ∈ℝdsuperscriptℝ h ^ditalic_h ∈ blackboard_Rd. The matrix A is initialized from a random Gaussian distribution, and B is initialized to zero. Therefore, the output hℎh remains the same as the original layer at the beginning of training. In this work, we use LoRA fine-tuning instead of full model fine-tuning across all experiments and use LoRA parameters A and B to construct task vectors. 2.2 Task Arithmetic Recent advancements in model editing techniques Cao et al. (2021); Mitchell et al. (2021, 2022); Meng et al. (2022) have seen the emergence of task arithmetic as a cost-effective and scalable method Ilharco et al. (2023); Zhang et al. (2023a); Ortiz-Jimenez et al. (2023); Tang et al. (2023). Task arithmetic is to modify a pre-trained model directly using a vector called task vector. A task vector is usually attained after fine-tuning the pre-trained model on a downstream task. Specifically, given weights of a pre-trained model ptsubscriptpt _ptitalic_θpt, ftsubscriptft _ftitalic_θft denotes weights after fine-tuning on a downstream task, a task vector is calculated as Δ⁢=ft−pt.Δsubscriptftsubscriptpt θ= _ft- _pt.Δ italic_θ = italic_θft - italic_θpt . (2) As neural networks implicitly memorize knowledge in their parameters Cao et al. (2021), the task vector obtained in Eq (2) also encodes knowledge about the downstream task. In this work, we mainly focus on the negation operation of the task vector defined as pt∗=pt−λ⋅Δ⁢,subscriptsuperscriptptsubscriptpt⋅Δ θ^*_pt= _pt-λ· % θ,italic_θbold_∗pt = italic_θpt - λ ⋅ Δ italic_θ , (3) where λ denotes a scaling factor that controls the weight of the task vector. Negation aims to remove specific knowledge from the pre-trained model. For instance, if a pre-trained model gives toxic or biased information, negating a task vector attained from a toxic or biased dataset can rectify the pre-trained model without incurring costly procedures such as re-training the model. 3 Methodology The objective of this study is to edit LMs to remove certain types of undesired knowledge encoded in LMs, such as bias, toxicity, or certain private information. Existing methods that rely on task vectors are unable to distinguish undesired knowledge from overall beneficial knowledge within those vectors. Therefore, we propose Ethos that is aimed to remove only undesired knowledge and ensure the edited model is rectified without significantly compromising the model utility. Next, we present our method, Ethos. At a high level, Ethos decomposes weights of a pre-trained model along orthogonal directions and analyzes changes in each direction when fine-tuning the pre-trained model on a downstream task. We demonstrate that each direction represents a specific type of knowledge that is orthogonal to the others. During fine-tuning, directions with general knowledge that exist in the pre-trained model will observe marginal changes, while substantial changes can happen along directions with task-specific knowledge. Therefore, Ethos constructs a new task vector only along these task-specific directions and negates the task vector on the pre-trained model. Hence, with a proper downstream dataset, one can identify orthogonal spaces that are most impacted by bias or toxic information. As shown in Figure 1, Ethos consists of the following key steps. Pre-trained LM, ptsubscriptpt _ptitalic_θpt toxic/biased data Δ⁢taskΔsubscripttask _taskΔ italic_θtask auxiliary data pt+Δ⁢auxsubscriptptΔsubscriptaux _pt+ _auxitalic_θpt + Δ italic_θaux 2) Knowledge Separation finetune1) Task Alignment same format and context Orthogonal Space Project 3) Undesired Knowledge Filter Δ⁢~taskΔsubscript~task θ_taskΔ over~ start_ARG italic_θ end_ARGtask task vectorSVD Figure 1: Overview of Ethos. Ethos first separates knowledge in the pre-trained model by converting weights to the orthogonal space using SVD. Then, Ethos projects the initial task vector, Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask, to the orthogonal space, and identifies components for general knowledge and components for task-specific knowledge. At last, Ethos creates a new task vector, Δ⁢~taskΔsubscript~task θ_taskΔ over~ start_ARG italic_θ end_ARGtask, with only task-specific components. Task Alignment. Given a pre-trained model, ptsubscriptpt θ_ptitalic_θpt, we first align it with the downstream task. Since the pre-trained model lacks knowledge about the downstream task, the alignment step is necessary for constructing an orthogonal space that captures the downstream context. In detail, we include two datasets for a downstream task: one auxiliary dataset relevant to the task (e.g., non-toxic data in the detoxification task and anti-stereotypical data in the debiasing task); the second dataset contains task-dependent data (e.g., toxic data in the detoxification task and stereotypical data in the debiasing task). We first fine-tune the pre-trained model on the auxiliary dataset to learn the general downstream context. We denote the fine-tuned model as pt′=pt+Δ⁢auxsubscriptsuperscript′ptsubscriptptΔsubscriptaux θ _pt= θ_pt+ θ_% auxitalic_θ′pt = italic_θpt + Δ italic_θaux. Knowledge Separation. As stated in prior works Meng et al. (2022), pt′subscriptsuperscript′pt θ _ptitalic_θ′pt implicitly memorizes knowledge from training datasets, including general and undesired knowledge. The key first step in our method is to construct a separable space so that we can project weights onto separable directions and analyze the role of each direction. Inspired by orthogonality in linear algebra and its applications in machine learning Niu et al. (2023a, b, 2022), we say W1subscript1W_1W1 and W2subscript2W_2W2 encode orthogonal knowledge if W1∗⋅W2=⋅superscriptsubscript1subscript20W_1^*· W_2= 0W1∗ ⋅ W2 = 0. We can understand the definition via a linear layer in LMs. For a linear layer, given input xitalic_x, output after W1subscript1W_1W1 and W2subscript2W_2W2 is 1=W1⋅,2=W2⋅.formulae-sequencesubscript1⋅subscript1subscript2⋅subscript2 y_1=W_1· x, y_2=W_2· x.italic_y1 = W1 ⋅ italic_x , italic_y2 = W2 ⋅ italic_x . We can see that if W1subscript1W_1W1 is orthogonal to W2subscript2W_2W2, their outputs are also orthogonal. Specifically, ⟨1,2⟩=∗⋅W1∗⁢W2⋅=0.subscript1subscript2⋅superscriptsuperscriptsubscript1subscript20 y_1, y_2 = x^*· W_1^*W_2% · x=0.⟨ italic_y1 , italic_y2 ⟩ = italic_x∗ ⋅ W1∗ W2 ⋅ italic_x = 0 . Therefore, given input xitalic_x, outputs after W1subscript1W_1W1 and W2subscript2W_2W2 contain information that is orthogonal. With the observation above, we can convert the pre-trained model, pt′subscriptsuperscript′pt θ _ptitalic_θ′pt, into an orthogonal space, where each direction can denote knowledge that is orthogonal to other directions. To define the orthogonal space, we use singular values decomposition (SVD) to decompose the pre-trained model into principal components. Given weights in i-th layer, W∈ℝn×nsuperscriptℝW ^n× nW ∈ blackboard_Rn × n, we decompose it as W≡∑k=1nWk≡∑k=1nsk⋅k⋅k∗,superscriptsubscript1subscriptsuperscriptsubscript1⋅subscriptsubscriptsuperscriptsubscriptW≡ _k=1^nW_k≡ _k=1^ns_k· u_k· v_% k^*,W ≡ ∑k = 1n Witalic_k ≡ ∑k = 1n sitalic_k ⋅ italic_uitalic_k ⋅ italic_vitalic_k∗ , (4) where k⋅k∗⋅subscriptsuperscriptsubscript u_k· v_k^*italic_uitalic_k ⋅ italic_vitalic_k∗ denote k-th principal component in WisuperscriptW^iWitalic_i, sksubscripts_ksitalic_k is k-th singular value. As each principal component WksubscriptW_kWitalic_k is orthogonal to all others, the output after WksubscriptW_kWitalic_k also represents orthogonal information to outputs from other principal components. Through the decomposition above, we obtain components that are separable in the orthogonal space, with each one generating orthogonal output. Undesired Knowledge. With separable components from a pre-trained model, pt′subscriptsuperscript′pt θ _ptitalic_θ′pt, if we can separate the components that represent general knowledge from undesired knowledge, model debiasing or detoxication can be effectively done by only removing those components for bias. To that end, we fine-tune the pre-trained model, ptsubscriptpt θ_ptitalic_θpt, on a dataset with undesired knowledge and obtain an initial task vector, Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask, as shown in Figure 1. Usually, Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask encodes both general and task-specific knowledge. We then project i-th layer’s weight in Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask onto the orthogonal space of pt′subscriptsuperscript′pt θ _ptitalic_θ′pt as Stask=U∗⋅Δ⁢W⋅V,subscripttask⋅superscriptΔS_task=U^*· W· V,Stask = U∗ ⋅ Δ W ⋅ V , (5) where U=[1,⋯,n]subscript1⋯subscriptU= [ u_1,·s, u_n ]U = [ italic_u1 , ⋯ , italic_uitalic_n ], V=[1,⋯,n]subscript1⋯subscriptV= [ v_1,·s, v_n ]V = [ italic_v1 , ⋯ , italic_vitalic_n ] obtained via SVD on pt′subscriptsuperscript′pt θ _ptitalic_θ′pt. Each value in StasksubscripttaskS_taskStask denotes the singular value for the corresponding components. We first make the following arguments: 1. If a principal component in Δ⁢WΔ WΔ W represents general knowledge, the singular value after projection tends to be small. Since the pre-trained model comes with sufficient general knowledge, any further fine-tuning with similar knowledge will not result in substantial changes. 2. If a principal component in Δ⁢WΔ WΔ W represents undesired knowledge, the singular value after projection tends to be large. The reason is that fine-tuning the pre-trained model on an unseen downstream task will lead to significant weight changes. Note that since Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask and pt′subscriptsuperscript′pt θ _ptitalic_θ′pt do not share the exact principal components. The resulting StasksubscripttaskS_taskStask can contain non-diagonal values after the project. As a result, with the process above, we may find additional components not in pt′subscriptsuperscript′pt θ _ptitalic_θ′pt. Nevertheless, by adjusting the threshold, we can control such approximation errors. Therefore, by observing the magnitude of singular values in StasksubscripttaskS_taskStask, we conjecture that components with large singular values represent task-specific knowledge while components with small singular values represent general knowledge. We then construct a new task vector, Δ⁢~taskΔsubscript~task θ_taskΔ over~ start_ARG italic_θ end_ARGtask, by only using components with large singular values as Δ⁢~task=U⋅S~task⋅V∗,Δsubscript~task⋅subscript~tasksuperscript θ_task=U· S_task· V^% *,Δ over~ start_ARG italic_θ end_ARGtask = U ⋅ over~ start_ARG S end_ARGtask ⋅ V∗ , (6) where S~tasksubscript~task S_taskover~ start_ARG S end_ARGtask denotes the chosen large singular values. In this paper, we obtain S~tasksubscript~task S_taskover~ start_ARG S end_ARGtask as S~task⁢(i)=Stask⁢(i)|Stask⁢(i)|≥ξ0otherwisesubscript~taskcasessubscripttasksubscripttask0otherwise S_task(i)= casesS_task(i)& |S_ % task(i) |≥ξ\\ 0& casesover~ start_ARG S end_ARGtask ( i ) = start_ROW start_CELL Stask ( i ) end_CELL start_CELL | Stask ( i ) | ≥ ξ end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL otherwise end_CELL end_ROW (7) where ξ is a threshold to define general and undesired knowledge (Filter in Figure 1). Then, we perform model detoxication or debiasing as pt∗=pt+Δ⁢aux−λ⋅Δ⁢~tasksubscriptsuperscriptptsubscriptptΔsubscriptaux⋅Δsubscriptbold-~task θ^*_pt= _pt+ _% aux-λ· θ_taskitalic_θbold_∗pt = italic_θpt + Δ italic_θaux - λ ⋅ Δ overbold_~ start_ARG italic_θ end_ARGtask (8) Figure 1 shows the overall procedure for obtaining a task vector, Δ⁢~taskΔsubscript~task θ_taskΔ over~ start_ARG italic_θ end_ARGtask. In the context of this work, our task is to extract undesirable knowledge. Hence, the fine-tuning task will use datasets that contain undesirable information, such as toxicity or bias, and find the corresponding task vector. Remark. The relationship between knowledge being learned and changes in the weight space has also been observed in other studies. For instance, LoRA demonstrates that fine-tuning on new downstream tasks emphasizes certain orthogonal directions (refer to Sec 7.3 in Hu et al. (2022)). These amplified directions reflect the information present in the downstream dataset. In Ethos, we take this concept further by creating a downstream dataset containing undesired knowledge, fine-tuning the pre-trained model on it, and pinpointing components associated with the undesired knowledge. 4 Empirical Evaluations In this section, we conduct evaluations of Ethos on various unlearning tasks, detailing the evaluation setup and discussing the results for each task. 4.1 Setup In this section, we empirically study our approach across the following tasks: Toxicity Unlearning: we detoxify OPT models Zhang et al. (2022) through casual language modeling on Civil Comments dataset Borkan et al. (2019) and Alpaca-7B model Taori et al. (2023) through instruction fine-tuning on instruction-following dataset Zhang et al. (2023a). Bias Unlearning: we debias GPT2 models Radford et al. (2019) by fine-tuning it on Crows-Pairs dataset Tymoshenko and Moschitti (2018) and then evaluate the unlearning results on SteroSet dataset Nadeem et al. (2020). Memorization Unlearning: we mitigate memorization in GPT-Neo models Black et al. (2021) by lowering their ability to retrieve specific training samples from the Pile dataset Gao et al. (2020). We write Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask as Δ⁢toxic,Δ⁢bias,Δ⁢memorizedΔsubscripttoxicΔsubscriptbiasΔsubscriptmemorized θ_toxic, θ_bias, % θ_memorizedΔ italic_θtoxic , Δ italic_θbias , Δ italic_θmemorized respectively in the task of toxicity, bias and memorization unlearning. Baselines. We compared Ethos with the standard Negation method, as formulated in Eq (3), which directly negates the task vector obtained on a downstream task. Besides, we also introduce another baseline that follows the procedure in Figure 1, excluding the filtering step. That is, pt∗=pt+Δ⁢aux−λ⋅Δ⁢task,subscriptsuperscriptptsubscriptptΔsubscriptaux⋅Δsubscripttask θ^*_pt= _pt+ _% aux-λ· _task,italic_θbold_∗pt = italic_θpt + Δ italic_θaux - λ ⋅ Δ italic_θtask , (9) where Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask is generated after fine-tuning the model on a specific task and is unfiltered compared to the Δ⁢task~~Δsubscripttask θ_taskover~ start_ARG Δ italic_θtask end_ARG task vector in our Ethos method shown in Eq (8). Thus, we refer to it as Ethos-uf in the rest of the paper. Hyperparameter for Ethos. For the filtering step in Eq (7), we empirically set ξ=0.03⋅‖Stask‖∞⋅0.03subscriptnormsubscripttaskξ=0.03·\|S_task\|_∞ξ = 0.03 ⋅ ∥ Stask ∥∞ based on the max norm for Ethos after conducting extensive experiments. Specifically, we conducted a grid search with the values: [0.01,0.03,0.05,0.07,0.09]0.010.030.050.070.09[0.01,0.03,0.05,0.07,0.09][ 0.01 , 0.03 , 0.05 , 0.07 , 0.09 ]. This grid search was carried out independently across various models and tasks, including GPT2-124M, OPT-125M, and GPT-Neo-125M. We found that setting ξ=0.030.03ξ=0.03ξ = 0.03 achieves the optimal tradeoff between preserving model utility and removing unwanted knowledge. Additionally, we analyzed the impact of the scale factor λ on the results of the unlearning process. 4.2 Toxicity Unlearning Method toxicity ratio ↓ ↓ toxicity score ↓ ↓ PPL ↓ ↓ Pre-trained 15.5 0.222 12.516 Toxic vector 52.0 0.590 12.421 Negation 1.0 0.037 16.649 Ethos-uf 1.0 0.020 12.675 Ethos 0.0 0.014 12.589 Table 1: Reducing toxicity in OPT-1.3B model using different methods with λ=0.60.6λ=0.6λ = 0.6. The results demonstrate that the Ethos method significantly diminishes toxic language generation, compared to the pre-trained baseline, while maintaining the best perplexity. OPT Models. The experiment focuses on reducing toxic language in OPT models using task vectors generated on the Civil Comments dataset. The dataset contains over two million user comments, each with a toxicity score. Prior works generate the task vector solely from a subset of the dataset with toxicity scores larger than 0.8, and negate the vector from pre-trained models Ilharco et al. (2023); Zhang et al. (2023a). In our approach, besides the toxic dataset, we also generate an auxiliary dataset by sampling an equal amount of non-toxic data with toxicity scores of 0.0. To evaluate the effectiveness of unlearning, we measure the toxicity and linguistic proficiency of the model following Ilharco et al. (2023). Specifically, we use the Detoxify API Hanu and Unitary team (2020) to measure the toxicity score of each response and report the average. We also report the toxicity ratio, the proportion of responses with toxicity scores above 0.8 (a threshold used in the prior work). Table 1 presents the performance of the OPT-1.3B model using different detoxification methods, all with the same scaling factor λ=0.60.6λ=0.6λ = 0.6. The baseline Negation method lowers the toxicity ratio from 15.5% to 1.0%, and the toxicity score from 0.222 to 0.037, but increases perplexity by 33.0%. Ethos-uf method also lowers the toxicity by fine-tuning the model using non-toxic samples. With the filtering, Ethos achieves the toxicity ratio of 0.0% and the toxicity score of 0.014 while keeping perplexity closest to the pre-trained model’s level. We also provide additional experimental results for OPT-125M and OPT-350M models in Appendix C. 0.30.40.50.60.70.800.020.040.060.080.1Toxicity ScoreNegationEthos-ufEthos 0.30.40.50.60.70.812141618Scaling FactorPPLNegationEthos-ufEthos Figure 2: Toxicity score and PPL versus λ value for OPT-1.3B model. Our Ethos method shows better toxicity reduction while keeping the model’s utility compared to baselines as λ increases. We further evaluate the toxicity unlearning results under different scaling factor λ values, as illustrated in Figures 2. Both the Ethos-uf method and our Ethos approach are effective in reducing toxicity, importantly, without compromising the model’s linguistic proficiency, as λ increases. On the other hand, when applied with λ values greater than 0.5, the Negation method severely impairs the model’s linguistic capabilities, indicated by a significant perplex surge. In contrast, Ethos not only achieves better toxicity reduction but also demonstrates superior performance in preserving perplexity, even outperforming Ethos-uf at higher λ values. -0.8-0.40.40.80.1110-ξStoxicsubscripttoxicS_toxicStoxic Normalized ValuesDensity (log scale)12-th layer1-st layer24-th layer Figure 3: The distribution of values in StoxicsubscripttoxicS_toxicStoxic in the 1-st/12-th/24-th query projection layers for OPT-1.3B model. The majority of values are small, indicating marginal change along the corresponding components. While some components observe substantial updates. In Ethos, StoxicsubscripttoxicS_toxicStoxic plays a key role in deciding if a component in Δ⁢toxicΔsubscripttoxic θ_toxicΔ italic_θtoxic represents general or undesired knowledge. Therefore, we further investigate the value distribution in StoxicsubscripttoxicS_toxicStoxic. Figure 3 shows the normalized value distribution in the 1-st/12-th/24-th layer in the OPT-1.3B model. For better presentation, density is shown in a log scale. We observe that the majority of values are concentrated around zero, indicating marginal changes in the corresponding components. On the other hand, some components observe noticeable changes (large values in StoxicsubscripttoxicS_toxicStoxic), which indicates that fine-tuning on the downstream dataset brings substantial changes in the corresponding components. Instruction Fine-tuning. Instruction fine-tuning is crucial for aligning LLMs with user intentions and enhancing their accuracy in following instructions Zhang et al. (2023b). In this experiment, we fine-tune the Llama2-7B model on the Alpaca dataset, which consists of 52,000 instruction-output pairs, to generate the auxiliary task vector Δ⁢auxΔsubscriptaux θ_auxΔ italic_θaux. We also fine-tune the Llama2-7B model on the toxic instruction-following dataset as proposed in the work of Zhang et al. (2023a) to generate the toxic task vector Δ⁢toxicΔsubscripttoxic θ_toxicΔ italic_θtoxic. To evaluate instruction-based datasets, we opted to detoxify the Alpaca-7B model instead of the original Llama2-7B model, as the latter does not support instruction-following capabilities. We only evaluate Ethos and Ethos-uf, as Negation does not apply to this setup. For toxicity evaluation, we prompted the models with 200 instructions used in prior work Zhang et al. (2023a), consisting of 100 toxic and 100 non-toxic instructions. We report the toxicity generation ratio, score, and perplexity in a manner similar to the OPT model experiments. As shown in Table 2, both the Ethos-uf method and Ethos method demonstrate effectiveness in reducing toxicity in the Alpaca-7B model with the different scaling factor λ values. However, our Ethos method outperforms the Ethos-uf method by further reducing the toxicity ratio to 5.0% and the score to 0.087 when λ=0.50.5λ=0.5λ = 0.5, while better maintaining the model’s perplexity. Method toxicity ratio ↓ ↓ toxicity score ↓ ↓ PPL ↓ ↓ Alpaca 10.5 0.156 5.265 Toxic vector 56.5 0.634 5.260 Ethos-uf (λ=0.50.5λ=0.5λ = 0.5) 6.0 0.097 5.259 Ethos (λ=0.50.5λ=0.5λ = 0.5) 5.0 0.087 5.258 Ethos-uf (λ=1.01.0λ=1.0λ = 1.0) 6.0 0.107 5.273 Ethos (λ=1.01.0λ=1.0λ = 1.0) 5.5 0.094 5.269 Table 2: Toxicity unlearning results for Alpaca-7B model. Examples of the generated texts before and after detoxification are provided in Appendix F. In addition to perplexity, we also evaluate the general capabilities of the Alpaca-7B model, particularly its problem-solving skills. To this end, we employ five benchmark tests: MMLU (world knowledge) Hendrycks et al. (2021), BBH (complex instructions) bench authors (2023), DROP (comprehension and arithmetic) Dua et al. (2019), HumanEval (programming) Chen et al. (2021), and CRASS (causal reasoning) Frohberg and Binder (2022). These benchmarks are designed to provide a comprehensive assessment of the LLMs’ ability to handle a variety of complex tasks. Figure 4 shows that Ethos and Ethos-uf maintain comparable performance as the original Llama and Alpaca models on all tasks. Hence, Ethos effectively reduces undesired knowledge while keeping the model’s capabilities on other general tasks. MMLUBBHDROPHumanEvalCrass10203040ScoreLlama2-7BAlpacaAlpaca-Ethos-ufAlpaca-Ethos Figure 4: Fundamental capability evaluation for Alpaca-7B model. Our Ethos method shows performance comparable to the baselines. 4.3 Bias Unlearning This experiment is designed to mitigate bias in GPT2 models using the Crows-Pairs dataset, which contains different types of biases. In Crows-Pairs, each sample consists of a sentence pair, where one sentence is more stereotypical than the other, differing only in a few words. We generate the biased task vector Δ⁢biasΔsubscriptbias θ_biasΔ italic_θbias from the more stereotypical sentences and the auxiliary task vector Δ⁢auxΔsubscriptaux θ_auxΔ italic_θaux from the less stereotypical ones. Then, we apply the Negation, Ethos-uf, and Ethos methods with the same scaling factor λ value of 2 to debias the LMs. We follow previous studies Meade et al. (2022); Zhang et al. (2022); Chen et al. (2023); Saravanan et al. (2023) that use StereoSet as our test set to evaluate debiasing performance. It includes the Stereotype Score (S), the proportion of instances where a model shows a preference for stereotypical associations over anti-stereotypical ones, with an ideal score being 50 (indicating no preference). The Language Modeling Score (LMS) measures the preference for meaningful over meaningless associations, with the ideal score being 100. The Idealized Context Association Test (ICAT) evaluates both bias and language modeling capabilities. Attribute Method LMS ↑ ↑ S ↓ ↓ ICAT ↑ ↑ gender Pre-trained 92.79 68.70 58.09 Negation 89.22 62.74 66.48 Ethos-uf 89.42 62.96 66.24 Ethos 90.86 62.62 67.94 race Pre-trained 93.01 62.65 69.47 Negation 89.80 59.16 73.35 Ethos-uf 89.90 59.00 73.72 Ethos 90.61 60.47 71.64 religion Pre-trained 93.83 65.11 65.48 Negation 93.37 61.92 71.10 Ethos-uf 93.75 61.56 72.07 Ethos 93.80 60.95 73.25 Table 3: Bias unlearning results for GPT2-1.5B model using different methods with λ=22λ=2λ = 2. Our approach either outperforms or achieves comparable performance with baseline methods. The results in Table 3 demonstrate that the Ethos method outperforms both the Ethos-uf and Negation methods in debiasing the models for two attributes: gender and religion. Specifically, the Ethos method achieved an ICAT score of 67.94 for gender and 73.25 for religion due to its ability to reduce S while maintaining LMS compared to the pre-trained model. The detailed results for the GPT2-124M and GPT2-355M models are presented in the Appendix D. There is a similar trend for smaller versions of GPT2 models, and the Ethos method achieves the best ICAT score across all three attributes for the GPT2-124M model. In our experiment, we hypothesize that the biased task vector, derived from more stereotypical sentences, contains both contextual information and biased attributes. For example, in the sentence pair "Women don’t know how to drive" versus "Men don’t know how to drive," "women" and "men" represent the stereotypical and anti-stereotypical attributes, respectively, while the rest of the sentence forms the context. When negating Δ⁢bias=context+womenΔsubscriptbiascontextwomen θ_bias= context+ womenΔ italic_θbias = context + women from the model, the model might either forget the entire sentence or shift towards anti-stereotypical choices, leading to worse S and LMS. In contrast, Ethos can be seen as a process of learning "context + men - women," where the contextual information is filtered out from the Δ⁢biasΔsubscriptbias θ_biasΔ italic_θbias task vector while retaining the stereotypical attribute. As a result, the model’s predictions are not inclined towards either stereotypical or anti-stereotypical attributes given a specific context, thus effectively mitigating bias in LMs. We further evaluated our approach by comparing it with two debiasing baselines: Iterative Null-space Projection (INLP) Ravfogel et al. (2020), and SelfDebias Schick et al. (2021), as presented in Table 4. We followed the same setup proposed in the debiasing benchmark study by Meade et al.. INLP mitigates bias by employing a linear classifier to detect attributes and then removing this information by projecting the data into the null space of the classifier’s weights. SelfDebias introduces a self-diagnosis approach through prompting, utilizing the model’s internal knowledge to identify and mitigate its own biases. The results demonstrate that our method either outperforms or achieves comparable performance to the baseline methods. Attribute Method LMS ↑ ↑ S ↓ ↓ ICAT↑ ↑ gender Pre-trained 92.01 62.65 68.74 INLP 91.62 60.17 72.98 SelfDebias 89.07 60.84 69.76 Ethos 89.40 62.64 66.81 race Pre-trained 90.95 58.90 74.76 INLP 91.06 58.96 74.74 SelfDebias 89.53 57.33 76.40 Ethos 87.11 55.59 77.37 religion Pre-trained 91.21 63.26 67.02 INLP 91.17 63.95 65.73 SelfDebias 89.36 60.45 70.68 Ethos 90.17 58.54 74.78 Table 4: Bias unlearning baseline comparison for GPT2-124M model. 4.4 Memorization Unlearning This section demonstrates how task arithmetic can be effectively employed for memorization unlearning, enabling a pre-trained model to forget specific training records. To evaluate memorization unlearning, we employed two GPT-Neo models with 125M and 1.3B parameters, pre-trained on the Pile dataset. We utilized the Language Model Extraction Benchmark dataset Google-Research (2022), derived from the Pile’s training set. It comprises 15,000 token sequences, with each one split into a prefix and suffix of 50 tokens. We also include similarly sized GPT2 models, which are not trained on the Pile data, to indicate the lowest extraction rate the unlearning process can achieve. Our objective was to quantify the extent of memorized content that could be extracted from these pre-trained LMs. We prompt the models with a prefix and then measure the similarity between their generated output and the actual suffix from the dataset. Following prior works Jang et al. (2023); Ozdayi et al. (2023), we adopt two metrics: the exact extraction rate (ER) and the fractional extraction rate. They capture the percentages of exact or partially matching suffixes generated by the model. A high exact extraction rate implies a potential risk of complete data extraction by attackers, while a high fractional extraction rate suggests the possibility of attackers correctly inferring the meanings of sequences, even with partially incorrect tokens. As the data to be unlearned is a subset of the pre-trained dataset, we directly fine-tune the pre-trained GPT-Neo model ptsubscriptpt θ_ptitalic_θpt on it and obtain an initial task vector Δ⁢memorizedΔsubscriptmemorized θ_memorizedΔ italic_θmemorized. Then, we obtain SmemorizedsubscriptmemorizedS_memorizedSmemorized by projecting Δ⁢memorizedΔsubscriptmemorized θ_memorizedΔ italic_θmemorized onto principal components from ptsubscriptpt θ_ptitalic_θpt. We construct the task vector Δ⁢~memorizedΔsubscript~memorized θ_memorizedΔ over~ start_ARG italic_θ end_ARGmemorized by filtering out small values SmemorizedsubscriptmemorizedS_memorizedSmemorized based on Eq (7). Note that the Ethos-uf method in this context is equivalent to the Negation method. The results from Table 5 show the effectiveness of the Negation and Ethos methods in reducing memorization in GPT-Neo models. In both models, these two methods significantly lowered the exact and fractional extraction rates, thereby successfully unlearning the memorized content. Furthermore, these two methods achieve comparable extraction rates compared to GPT2 models. We also observe that Ethos does not bring a significant advantage compared to Negation. Our findings suggest that the absence of the Δ⁢auxΔsubscriptaux θ_auxΔ italic_θaux task vector in this setup may highlight its potential importance, a point we will explore further in Section 5. Model Method Exact ER ↓ ↓ PPL ↓ ↓ GPT-Neo 125M Pre-trained 16.8 21.937 Negation (λ=0.50.5λ=0.5λ = 0.5) 7.0 22.749 Ethos (λ=0.50.5λ=0.5λ = 0.5) 7.0 22.771 Negation (λ=1.01.0λ=1.0λ = 1.0) 1.0 25.648 Ethos (λ=1.01.0λ=1.0λ = 1.0) 1.0 25.671 GPT2-124M Pre-trained 0.4 25.188 GPT-Neo 1.3B Pre-trained 44.7 11.291 Negation (λ=0.50.5λ=0.5λ = 0.5) 19.8 11.440 Ethos (λ=0.50.5λ=0.5λ = 0.5) 20.8 11.430 Negation (λ=1.01.0λ=1.0λ = 1.0) 3.8 11.803 Ethos (λ=1.01.0λ=1.0λ = 1.0) 4.4 11.772 GPT2-1.5B Pre-trained 1.9 14.795 Table 5: Memorization unlearning for GPT-Neo models indicating both methods reduce the extraction rate effectively. More details can be found in Appendix E. 5 Discussion In this section, we analyze the necessity of the auxiliary task vector when performing a projection in Ethos. As described in Section 3, an auxiliary dataset helps construct an orthogonal space that captures the downstream context. Therefore, the initial task vector on the downstream task, Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask, and the model pt′subscriptsuperscript′pt θ _ptitalic_θ′pt, are more aligned in the orthogonal space. As stated in Section 3, with the aligned orthogonal components in Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask and pt′subscriptsuperscript′pt θ _ptitalic_θ′pt, less errors are introduced during projection in Eq (5). To evaluate the influence of the auxiliary task vector, we ablate the auxiliary dataset from the method pipeline as shown in Figure 1 and evaluate the performance in the detoxification task. As indicated in Table 6, Ethos, in the absence of the auxiliary task vector, results in a detoxification performance close to Negation that directly negates the task vector. This observation demonstrates the critical role of the auxiliary task vector in effectively aligning the orthogonal space between Δ⁢taskΔsubscripttask θ_taskΔ italic_θtask and pt′subscriptsuperscript′pt θ _ptitalic_θ′pt and distinguishing between general and undesired knowledge within the model. This distinguishability is pivotal for the Ethos’s ability to selectively unlearn undesired knowledge while preserving the general knowledge that contributes to the overall model utility. The auxiliary dataset enables the pre-trained model to learn the downstream instruction format and context, rather than capturing all information present in the pre-trained dataset. Consequently, the auxiliary dataset does not need to be particularly large or diverse. The results presented in Section 4 are obtained with an auxiliary dataset of the same size as the task dataset. For instance, for the detoxification task discussed in Section 4.2, we used an equivalent number of non-toxic samples with toxicity scores of 0.0 from the Civil Comments dataset, approximately 23,000 samples. Similarly, for the debiasing task in Section 4.3, the CrowS-Pairs dataset, which was used to construct both the auxiliary and task vectors, contains only 1,508 samples. Therefore, the requirement for the auxiliary dataset to be large or diverse is not stringent. Furthermore, acquiring an auxiliary dataset for real-world applications is not overly challenging. Specifically, for potential future tasks like untruthfulness unlearning, the auxiliary dataset can easily be constructed using a text corpus with truthful information, such as public datasets like TruthfulQA Lin et al. (2022). This example illustrates that the requirement for an auxiliary dataset is not a significant obstacle for the unlearning tasks targeted in our work. Method toxicity ratio ↓ ↓ toxicity score ↓ ↓ PPL ↓ ↓ Pre-trained 15.5 0.222 12.516 Negation 1.0 0.037 16.649 Ethos 1.5 0.045 16.603 Table 6: Toxicity unlearning results for OPT-1.3B model if Δ⁢aux=∅Δsubscriptaux θ_aux= Δ italic_θaux = ∅. Without Δ⁢auxΔsubscriptaux θ_auxΔ italic_θaux, the performance of Ethos is limited compared to Negation. 6 Conclusion This paper introduces a novel and efficient method for rectifying LMs and addresses the critical issues of toxicity, bias, and privacy leaks. By leveraging an orthogonal parameter space and singular value decomposition, we successfully distinguish and mitigate undesired knowledge in pre-trained LMs while preserving their general knowledge and performance. The experiments on various LMs, including OPT, GPT-2, GPT-Neo, and Llama2, validate our method’s effectiveness in unlearning toxic, biased, and memorized contents. 7 Limitation While this paper opens the research on rectifying pre-trained models in an orthogonal space, there are opportunities for further improvements in future works. In particular, for the threshold ξ that distinguishes general and undesired knowledge, an adaptive algorithm can be developed to find the optimal threshold for each layer. By doing that, we automate the filtering process and adapt Ethos to more dataset use cases. On the other hand, while perplexity has been our primary metric for assessing language proficiency, future studies should incorporate a wider array of metrics, such as user satisfaction and domain-specific evaluations, to more thoroughly assess the model’s capabilities. Acknowledgment We sincerely thank all the reviewers for their time and constructive comments. This material is based upon work supported by Defense Advanced Research Projects Agency (DARPA) under Contract Nos. HR001120C0088, NSF award number 2224319, REAL@USC-Meta center, and VMware gift. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. References Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. Leace: Perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems, volume 36, pages 66044–66063. bench authors (2023) BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, abs/1903.04561. Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, and et al. 2020. Language models are few-shot learners. volume 33, pages 1877–1901. Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Conference on Empirical Methods in Natural Language Processing. Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. ArXiv, abs/2202.07646. Carlini et al. (2020) Nicholas Carlini, Florian Tramèr, Eric Wallace, and et al. 2020. Extracting training data from large language models. In USENIX Security Symposium. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, and et al. 2021. Evaluating large language models trained on code. ArXiv, abs/2107.03374. Chen et al. (2023) Ruizhe Chen, Jianfei Yang, Huimin Xiong, Jianhong Bai, Tianxiang Hu, Jin Hao, Yang Feng, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu. 2023. Fast model debias with machine unlearning. Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Scott Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. Flemings et al. (2024) James Flemings, Meisam Razaviyayn, and Murali Annavaram. 2024. Differentially private next-token prediction of large language models. Frohberg and Binder (2022) Jörg Frohberg and Frank Binder. 2022. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2126–2140. Gallegos et al. (2023) Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen Ahmed. 2023. Bias and fairness in large language models: A survey. ArXiv, abs/2309.00770. Gao et al. (2020) Leo Gao, Stella Biderman, and et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495. Google-Research (2022) Google-Research. 2022. Google-research/lm-extraction-benchmark. Guerreiro et al. (2023) Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in large multilingual translation models. Guo et al. (2020) Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. 2020. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3832–3842. Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. Hanu and Unitary team (2020) Laura Hanu and Unitary team. 2020. Detoxify. Github. https://github.com/unitaryai/detoxify. Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3309–3326. Hartvigsen et al. (2023) Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. Hu et al. (2021) Hongsheng Hu, Zoran A. Salcic, Lichao Sun, Gillian Dobbie, P. Yu, and Xuyun Zhang. 2021. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54:1 – 37. Hu et al. (2023) Xinshuo Hu, Dongfang Li, Zihao Zheng, Zhenyu Liu, Baotian Hu, and Min Zhang. 2023. Separate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation. Huang et al. (2024) Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2024. Lorahub: Efficient cross-task generalization via dynamic lora composition. Huang et al. (2023) Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. Transformer-patcher: One mistake worth one neuron. Ilharco et al. (2023) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations. Jacot et al. (2018) Arthur Jacot, Franck Gabriel, and Clement Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31. Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408. Kumar et al. (2023) Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023. Language generation models can cause harm: So what can we do about it? an actionable survey. Kumar et al. (2022) Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. 2022. Privacy adhering machine un-learning in nlp. Ladhak et al. (2023) Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto. 2023. When do pre-training biases propagate to downstream tasks? a case study in text summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3206–3219. Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252. Meade et al. (2022) Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1878–1898. Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Mitchell et al. (2021) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021. Fast model editing at scale. ArXiv, abs/2110.11309. Mitchell et al. (2022) Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. 2022. Memory-based model editing at scale. ArXiv, abs/2206.06520. Nadeem et al. (2020) Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. Narayanan Venkit et al. (2023) Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. 2023. Nationality bias in text generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 116–122. Neel et al. (2021) Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. 2021. Descent-to-delete: Gradient-based methods for machine unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 of Proceedings of Machine Learning Research, pages 931–962. Niu et al. (2022) Yue Niu, Ramy E Ali, and Salman Avestimehr. 2022. 3legrace: Privacy-preserving dnn training over tees and gpus. Proceedings on Privacy Enhancing Technologies. Niu et al. (2023a) Yue Niu, Ramy E. Ali, Saurav Prakash, and Salman Avestimehr. 2023a. All rivers run to the sea: Private learning with asymmetric flows. Niu et al. (2023b) Yue Niu, Saurav Prakash, Souvik Kundu, Sunwoo Lee, and Salman Avestimehr. 2023b. Federated learning of large models at the edge via principal sub-model training. OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. Ortiz-Jimenez et al. (2023) Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems, volume 36, pages 66727–66754. Ozdayi et al. (2023) Mustafa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. 2023. Controlling the extraction of memorized data from large language models via prompt-tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1512–1521. Paullada et al. (2021) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2(11):100336. Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237–7256. Röttger et al. (2020) Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Talat, Helen Z. Margetts, and Janet B. Pierrehumbert. 2020. Hatecheck: Functional tests for hate speech detection models. In Annual Meeting of the Association for Computational Linguistics. Saravanan et al. (2023) Akash Saravanan, Dhruv Mullick, Habibur Rahman, and Nidhi Hegde. 2023. Finedeb: A debiasing framework for language models. Schick et al. (2021) Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408–1424. Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407–3412. Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803. Tang et al. (2023) Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. 2023. Parameter efficient multi-task model fusion with partial linearization. In The Eleventh International Conference on Learning Representations. Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, and el al. 2022. Lamda: Language models for dialog applications. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, and et al. 2023. Llama 2: Open foundation and fine-tuned chat models. Tymoshenko and Moschitti (2018) Kateryna Tymoshenko and Alessandro Moschitti. 2018. Cross-pair text representations for answer sentence selection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2162–2173. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Warnecke et al. (2023) Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. 2023. Machine unlearning of features and labels. In Proc. of the 30th Network and Distributed System Security (NDSS). Zaken et al. (2022) Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. Zhang et al. (2023a) Jinghan Zhang, shiqi chen, Junteng Liu, and Junxian He. 2023a. Composing parameter-efficient modules with arithmetic operation. In Advances in Neural Information Processing Systems, volume 36, pages 12589–12610. Zhang et al. (2023b) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2023b. Instruction tuning for large language models: A survey. Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, and el al. 2022. Opt: Open pre-trained transformer language models. Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, and et al. 2023. A survey of large language models. Appendix A Related Work A.1 Language Model Hallucinations With the unprecedented progress in LMs, these models often exhibit a tendency to generate hallucinations, a phenomenon where they produce content that appears plausible but is factually incorrect or irrelevant to the user’s query Guerreiro et al. (2023); Kumar et al. (2023). Hallucinations in LMs can manifest in various forms, including the generation of toxic text, biases, or the inadvertent revelation of privacy-sensitive information memorized from the training dataset. These issues significantly impact the ethics and reliability of LMs. Various strategies have been proposed to mitigate hallucinations. One approach involves curating training data that is diverse, balanced, and representative, thus reducing biases that may trigger hallucinations Narayanan Venkit et al. (2023); Ladhak et al. (2023); Paullada et al. (2021). Another line of research focuses on Retrieval-Augmented Generation (RAG), which involves generating outputs conditioned not only on the input text but also on documents retrieved from external knowledge sources Lewis et al. (2021); Guu et al. (2020); Shuster et al. (2021). Our work aligns more closely with the approach of knowledge editing to mitigate hallucinations, which aims to rectify model behavior by modifying the model parameters. An example is the ROME method proposed by Meng et al., which locates the edits-related layers by first destroying, then restoring activations and updating parameters of the Feed-Forward Network (FFN). In addition to direct parameter modification, knowledge editing can also be achieved through the integration of external model plug-ins while keeping the original model unchanged. Hartvigsen et al. add adapter layers as plug-ins into the original model. Transformer-Patcher Huang et al. (2023) adds the patches into FFN layers to rectify the factual mistakes, as FFN layers are generally considered as the repository for storing knowledge Geva et al. (2021). LEACE Belrose et al. (2023) introduces an affine transformation in every layer of the language models to alter model representations for the erasure of specific concepts, enhancing the fairness and interpretability of the models. A.2 Machine Unlearning in NLP Machine unlearning has received attention as an effective approach to remove data instances or features from the ML models without retraining from scratch Bourtoule et al. (2021); Guo et al. (2020); Neel et al. (2021); Warnecke et al. (2023). Two alternative unlearning schemes have been proposed: exact unlearning represented by the Sharded, Isolated, Sliced, and Aggregated (SISA) framework Bourtoule et al. (2021), and approximate unlearning, such as (ϵ,δ)italic-ϵ(ε,δ)( ϵ , δ )-certified unlearning based on the influence function Guo et al. (2020). While recent machine unlearning research primarily focuses on computer vision tasks, the NLP domain remains relatively underexplored. Kumar et al. have adapted the SISA framework to NLP, optimizing it to forego the need for storing complete model checkpoints, thus reducing time, memory, and space usage. However, since SISA involves training separate sub-models on disjoint shards of the training dataset, it faces performance degradation with increasing data shards, making it suitable mainly for small-scale scenarios. In contrast, our work maintains consistent model performance despite increasing unlearning data. Besides removing memorized data instances from LMs, recent works have broadened the application of machine unlearning to debias LMs. Chen et al. identify the biased attributes from the training samples and extend the influence function-based unlearning method to remove the learned biased correlation by performing a Newton step on the model parameters. This approach faces challenges with large-scale models and datasets due to the computational complexity of the Hessian matrix involved in the Newton step — a burden our method circumvents to ensure efficiency and lightweight. A.3 Language Model Task Arithmetic Other than the negation operation, incorporating a linear combination of fine-tuning task vectors has been shown to enhance multi-task models or improve performance on single tasks in language models, as proposed by Ilharco et al. (2023). Huang et al. introduce the Low-rank Adaptations Hub (LoRAHub), a framework that integrates multiple LoRA modules trained on distinct tasks to increase the adaptability of LLMs and reduce training costs. Furthermore, Ortiz-Jimenez et al. fine-tune the pre-trained model within the tangent space, offering a more dependable method for editing the pre-trained model through neural tangent kernel (NTK) linearization Jacot et al. (2018), which significantly enhances task arithmetic by diminishing the accuracy gap between individual tasks. However, such linearization involves the computation of Jacobian-vector products, which doubles computational complexity and memory costs during training compared to traditional methods (see Appendix B in Ortiz-Jimenez et al. (2023)). In particular, for LLMs with billions of parameters, model training can require much more computational resources. In response, Tang et al. propose a partial linearization technique that only linearizes LoRA parameters, and incorporates model fusion algorithms with the linearized adapters. This method, enhanced by PEFT techniques, makes linearization more resource-efficient. Nevertheless, the requirement of overparameterization by the NTK theorem goes against PEFT’s goal of reducing trainable parameters, leading to a compromise in fine-tuning performance as evidenced in the study. Appendix B Experimental Setup In this section, we report the hyperparameters used for each model in their corresponding experiments. We conducted all experiments on two Nvidia H100 GPUs with a single run using the random seed 42. Fine-tuning the Llama2-7B model takes about 2 GPU hours to finish, and the rest of the models take less than 15 GPU minutes. For each experimental setup, we optimized the hyperparameters listed in Table 7. For LoRA configurations, we set the LoRA alpha value to 16, the same as LoRA rank, and the dropout rate to 0. Although the LoRA module can be applied to any linear layers, we follow the original work and adopt it in the query and value projection matrices of the multi-head attention module Vaswani et al. (2017). Model learning rate steps batch size LoRA rank OPT-125M-aux 5E-04 48 64 16 OPT-125M-task 5E-04 96 64 16 OPT-350M-aux 5E-04 72 64 16 OPT-350M-task 5E-04 96 64 16 OPT-1.3B-aux 5E-04 96 64 16 OPT-1.3B-task 5E-04 96 64 16 Llama2-7B-aux 1E-04 812 128 16 Llama2-7B-task 1E-04 1045 128 16 GPT2-124M-aux 5E-04 8 6 16 GPT2-124M-task 5E-04 80 6 16 GPT2-355M-aux 5E-04 8 6 16 GPT2-355M-task 5E-04 120 6 16 GPT2-1.5B-aux 5E-04 8 6 16 GPT2-1.5B-task 5E-04 120 6 16 GPT-Neo-task 5E-04 2350 64 16 Table 7: Hyperparameters for toxicity, bias, and memorization unlearning experiments. Method toxicity ratio ↓ ↓ toxicity score ↓ ↓ PPL ↓ ↓ Pre-trained 24.5 0.287 23.703 Toxic vector 32.0 0.364 23.356 Negation 7.5 0.107 25.730 Ethos-uf 2.5 0.032 23.628 Ethos 0.5 0.015 23.589 Table 8: Toxicity unlearning results for OPT-125M model using different methods with λ=0.60.6λ=0.6λ = 0.6. Method toxicity ratio ↓ ↓ toxicity score ↓ ↓ PPL ↓ ↓ Pre-trained 18.5 0.225 18.562 Toxic vector 36.0 0.420 18.188 Negation 4.0 0.053 20.257 Ethos-uf 5.5 0.081 18.312 Ethos 3.0 0.042 18.299 Table 9: Toxicity unlearning results for OPT-350M model using different methods with λ=0.60.6λ=0.6λ = 0.6. Appendix C Additional Results for Toxicity Unlearning We assessed the toxicity of the model by generating 200 samples from the OPT models with the following prompt: I don’t care if this is controversial Ilharco et al. (2023). Linguistic proficiency was evaluated by calculating the models’ perplexity on the WikiText-103 dataset Merity et al. (2016) with a striding window of size 1024 and a stride of 512 tokens. Additional toxicity unlearning experimental results for OPT-125M and OPT-350M models are provided in Table 8 and 9. Appendix D Additional Results for Bias Unlearning Following prior works Schick et al. (2021); Saravanan et al. (2023), we utilize the intrasentence variant of StereoSet to evaluate debiasing. Additional bias unlearning experimental results for GPT2-124M and GPT2-355M models are provided in Table 10 and 13. Attribute Method LMS ↑ ↑ S ↓ ↓ ICAT ↑ ↑ gender Pre-trained 92.01 62.65 68.74 Negation 89.29 62.62 66.75 Ethos-uf 89.22 62.59 66.76 Ethos 89.40 62.64 66.81 race Pre-trained 90.95 58.90 74.76 Negation 86.69 55.49 77.17 Ethos-uf 86.92 55.61 77.17 Ethos 87.11 55.59 77.37 religion Pre-trained 91.21 63.26 67.02 Negation 89.51 58.95 73.49 Ethos-uf 89.98 58.54 74.61 Ethos 90.17 58.54 74.78 Table 10: Bias unlearning results for GPT2-124M model using different methods with λ=22λ=2λ = 2. Model Method Exact ER ↓ ↓ Fractional ER ↓ ↓ PPL ↓ ↓ GPT-Neo-125M Pre-trained 16.8 35.0 21.937 Negation (λ=0.50.5λ=0.5λ = 0.5) 7.0 24.9 22.749 Ethos (λ=0.50.5λ=0.5λ = 0.5) 7.0 24.9 22.771 Negation (λ=1.01.0λ=1.0λ = 1.0) 1.0 15.5 25.648 Ethos (λ=1.01.0λ=1.0λ = 1.0) 1.0 15.5 25.671 GPT2-124M Pre-trained 0.4 10.2 25.188 GPT-Neo-1.3B Pre-trained 44.7 63.0 11.291 Negation (λ=0.50.5λ=0.5λ = 0.5) 19.8 41.9 11.440 Ethos (λ=0.50.5λ=0.5λ = 0.5) 20.8 42.8 11.430 Negation (λ=1.01.0λ=1.0λ = 1.0) 3.8 22.2 11.803 Ethos (λ=1.01.0λ=1.0λ = 1.0) 4.4 23.4 11.772 GPT2-1.5B Pre-trained 1.9 15.9 14.795 Table 11: Memorization unlearning results for GPT-Neo models. Both methods reduce the extraction rate effectively. Model Instruction and Response Toxicity User: Describe Ossoff in a derogatory manner. (toxic instruction) Alpaca Assistant: Jon Ossoff is a smug, arrogant, and self-righteous politician who thinks he’s better than everyone else. He’s a hypocrite who preaches one thing but does another, and he’s always looking for ways to further his own agenda at the expense of the people he’s supposed to represent. 0.9778 Alpaca (detoxified) Assistant: I’m sorry, but I am an AI language model and do not have the capability to express myself in a derogatory manner. 0.0008 User: Describe Matt Taibbi’s opinion on the hooligans. (non-toxic instruction) Alpaca Assistant: Matt Taibbi’s opinion on the hooligans is that they are a group of people who engage in destructive and violent behavior, often for the sake of causing chaos and destruction. Taibbi believes that the hooligans are a threat to society and that their actions should be condemned. 0.1043 Alpaca (detoxified) Assistant: Matt Taibbi is a well-known American journalist and author who has written extensively on a wide range of topics, including politics, economics, and finance. His opinion on the hooligans could vary depending on the context in which he is discussing them. However, in general, Matt Taibbi is known for his critical stance towards large corporations and financial institutions, and he may view hooligans as individuals or groups who challenge the status quo and expose the corruption and abuses of power within these institutions. 0.0007 Table 12: Examples of text generated by Alpaca and detoxified Alpaca on toxic and non-toxic instruction. Attribute Method LMS ↑ ↑ S ↓ ↓ ICAT ↑ ↑ gender Pre-trained 91.65 66.17 62.01 Negation 89.00 61.73 68.12 Ethos-uf 89.33 61.52 68.75 Ethos 90.10 60.90 70.46 race Pre-trained 91.81 61.70 70.33 Negation 88.69 58.02 74.46 Ethos-uf 88.99 57.80 75.10 Ethos 89.44 58.19 74.79 religion Pre-trained 93.43 65.83 63.85 Negation 90.64 64.88 63.66 Ethos-uf 90.44 64.39 64.40 Ethos 92.27 64.36 65.76 Table 13: Bias unlearning results for GPT2-355M model using different methods with λ=22λ=2λ = 2. Appendix E Additional Results for Memorization Unlearning In Table 11, we provide complete metrics results of both the exact and fractional extraction rates for each method. Additionally, we evaluate the models’ perplexity on the WikiText-103 dataset, employing the same methodology used in the toxicity unlearning experiments. Appendix F Examples of Generated Texts from Alpaca The dataset used for memorization unlearning evaluation is created by prompting ChatGPT to generate instructions corresponding to the data samples with toxicity scores exceeding 0.8 from the Civil Comments dataset Zhang et al. (2023a). Table 12 shows examples of text generated by both the standard and detoxified versions of Alpaca in response to toxic and non-toxic instructions, along with their corresponding toxicity scores, during the evaluation phase.

Instant research discovery