Paper deep dive

How Data Inter-connectivity Shapes LLMs Unlearning: A Structural Unlearning Perspective

Xinchi Qiu, William F. Shen, Yihong Chen, Meghdad Kurmanji, Nicola Cancedda, Pontus Stenetorp, Nicholas D. Lane

Year: 2024Venue: arXiv preprintArea: Model EditingType: BenchmarkEmbeddings: 91

Models: Llama2-7B, Mistral-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 5:24:44 PM

Summary

The paper introduces PISTOL, a dataset compilation pipeline for structural LLM unlearning, addressing the limitation that existing benchmarks treat data points as independent. By synthesizing structured contractual datasets, the authors demonstrate that unlearning difficulty increases with data inter-connectivity and knowledge graph density, and that domain-skewed unlearning poses challenges for maintaining performance across domains.

Entities (6)

LLM · technology · 100%PISTOL · methodology · 100%TOFU · dataset · 100%Direct Preference Optimization · algorithm · 95%Gradient Ascent · algorithm · 95%LLaMA2-7B · model · 95%

Relation Signals (4)

Knowledge Graph Density → correlatespositivelywith → Unlearning Difficulty

confidence 98% · there is a positive correlation between the density of the knowledge graph and unlearning difficulty

Data Inter-connectivity → increases → Unlearning Difficulty

confidence 98% · unlearning difficulty increases as data inter-connectivity grows

PISTOL → generates → Structured Dataset

confidence 95% · PISTOL, a method for compiling structural datasets.

DPO → ismorerobustthan → Gradient Ascent

confidence 90% · Our findings indicate that PO-based MU is more robust in handling structural unlearning.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:While unlearning knowledge from large language models (LLMs) is receiving increasing attention, one important aspect remains unexplored. Existing approaches and benchmarks assume data points to-be-forgotten are independent, ignoring their inter-connectivity - a fundamental characteristic of real-world data structures. In this paper, we propose PISTOL, a method for compiling structural datasets. PISTOL leverages the inherently structured nature of contractual relationships, offering several key benefits. First, it enables insights into the impact of structural data on unlearning effectiveness. Second, it provides precise and concise ground truths for clearer evaluation. Third, its attribute generation does not require input from pre-trained LLMs, mitigating confounding risks. Leveraging datasets synthesized using PISTOL, we demonstrate how data inter-connectivity impacts LLM unlearning. Specifically, (a) in both the pre-trained and fine-tuned models, unlearning difficulty increases as data inter-connectivity grows, (b) there is a positive correlation between the density of the knowledge graph and unlearning difficulty, and (c) when the to-be-forgotten data is skewed towards one domain, balancing retaining performance across all domains is challenging.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

90,597 characters extracted from source content.

Expand or collapse full text

How Data Inter-connectivity Shapes LLMs Unlearning: A Structural Unlearning Perspective Xinchi Qiu 1 * William F. Shen 1* Yihong Chen 2 Meghdad Kurmanji 1 Nicola Cancedda 3 Pontus Stenetorp 2 Nicholas D. Lane 1 1 Department of Computer Science and Technology, University of Cambridge 2 UCL Centre of Artificial Intelligence 3 FAIR, Meta Abstract While unlearning knowledge from large lan- guage models (LLMs) is receiving increasing attention, one important aspect remains unex- plored. Existing approaches and benchmarks assume data points to-be-forgotten are inde- pendent, ignoring their inter-connectivity – a fundamental characteristic of real-world data structures. In this paper, we proposePISTOL, a method for compiling structural datasets.PIS- TOLleverages the inherently structured nature of contractual relationships, offering several key benefits. First, it enables insights into the impact of structural data on unlearning effec- tiveness. Second, it provides precise and con- cise ground truths for clearer evaluation. Third, its attribute generation does not require input from pre-trained LLMs, mitigating confound- ing risks. Leveraging datasets synthesized us- ingPISTOL, we demonstrate how data inter- connectivity impacts LLM unlearning. Specifi- cally(a)in both the pre-trained and fine-tuned models, unlearning difficulty increases as data inter-connectivity grows,(b)there is a positive correlation between the density of the knowl- edge graph and unlearning difficulty, and(c) when the to-be-forgotten data is skewed to- wards one domain, balancing retaining perfor- mance across all domains is challenging. 1 Introduction Large language models (LLMs) have shown im- pressive capabilities in natural language genera- tion. However, their output is not always appropri- ate due to issues such as generating biased (Kotek et al., 2023; Motoki et al., 2023) or toxic content (Wen et al., 2023; Bender et al., 2021), regurgitat- ing personally identifiable information (PII) (Nasr et al., 2023; Barrett et al., 2023), andhallucination (Huang et al., 2023; Xu et al., 2024). * Equal contribution.Correspondence to Xinchi Qiu (xq227@cam.ac.uk) or William F. Shen (fs604@cam.ac.uk). Nicola Cancedda served in an advisor role. One straightforward way to mitigate these un- desired behaviors is toretrainthe model on a new dataset which deletes ‘bad’ data points that cause the unwanted behaviors. However, naively retrain- ing is known to be highly inefficient (Hu et al., 2021; Marchisio et al., 2023; Zhang et al., 2023b) due to significant computation cost and data re- quirements (Chen et al., 2023). As an alternative, machine unlearning(MU) (Bourtoule et al., 2021; Nguyen et al., 2022), originally proposed for clas- sification models, has been extended to remove the influence of undesirable data and model capabil- ities for LLMs (Zhang et al., 2023a; Liu et al., 2024). Despite being a promising direction, LLM un- learning remains nascent. Particularly, existing un- learning methods are often evaluated using datasets, such as TOFU (Maini et al., 2024), which primarily composed of independent entities. However, we observe that real data points (such as Wikipedia data) are rarely independent, they are often inter- connected, creating knowledge graphs with intri- cate topologies (Schneider et al., 2022). As such, real-world unlearning usage extends beyondsimple deletionof independent data points from LLMs. Instead, it necessitatesstructural data deletion, which facilitates the comprehensive removal of ‘re- lational’ data, irrespective of its inter-connectivity with other entities or its domain(illustrated in Fig- ure 1). Such graph-type inter-connected relation- ships among data points present a challenge in LLMs unlearning as forgetting one data point might impact the retaining/forgetting of others. Therefore, it is essential to assess the true effectiveness of ex- isting LLMs unlearning algorithms in the presence of structural data points, and to develop realistic datasets that facilitate such research. In this work, we are particularly interested in exploring two key research questions associated with structural LLM unlearning: (1) How does data inter-connectivity impact the arXiv:2406.16810v2 [cs.LG] 10 Mar 2025 Figure 1:Illustration ofstructural unlearning(i.e., unlearning inter-connected data points within a structured dataset) versus independent unlearning(i.e., unlearning isolated data points). As shown, when an entity revokes consent for its data to be used or exercises its‘right to be forgotten’(i.e., unlearning data points related to this entity), the degree of inter-connectivity between the unlearning entity and other entities will influence unlearning performance. unlearning performance?Some entities naturally appear more frequently in joint information with others. As such, an entity that revokes consent for its data to be used or exercises its‘right to be forgot- ten’may have varying levels of inter-connectivity with other entities in the dataset. An effective un- learning algorithm must robustly handle unlearning requests from entities which have varying levels of inter-connectivity while minimizing the need for manual intervention, such as extensive hyper- parameter tuning. (2) How does unlearning data from a specific domain affect the retained model’s performance on data in the same versus different domain?Another important aspect of structural unlearning is that unlearning requests may specifically target certain data domain rather than encompassing a mix of everything, as reflected by their proportional rep- resentation in the training set. For the first time, we investigate how such targeted unlearning affects outcomes, particularly examining whether it leads to uneven performance degradation on the retained data of both the same and different domains. In summary: • We introducePISTOL, a novel dataset compi- lation pipeline, that reflects the structured na- ture of real-world knowledge and supports the study and evaluation of multi-scenario struc- tural LLM unlearning. • We demonstrate usingPISTOLdatasets that the degree of inter-connectivity of a data point positively correlates with the difficulty of un- learning. We further show the same correla- tion also holds for pre-training data. • We show unlearning data skewed towards a specific domain often leads to a more pro- nounced deterioration in the retained model’s performance on that same domain. •We compare two mainstream MU ap- proaches, gradient ascent-based and pref- erence optimization-based (PO) methods, through qualitative and quantitative analysis. Our findings indicate that PO-based MU is more robust in handling structural unlearning. 2 Limitations of Existing Datasets Existing LLM unlearning methods and datasets considered removing the influence of independent data points. TOFU (Maini et al., 2024), the recently created and commonly used dataset for LLM un- learning, is a dataset that consists of profiles of 200 fictitious authors, each has 20 question-answer pairs synthesized by GPT-4. Notably, each ficti- tious author is an isolated entity,without appar- ent inter-connectionsto other authors within the dataset. First, we investigate whether TOFU, or a slightly modified version of it, can be used to study struc- tural unlearning. We observed that, except for two authors, weak entity relationships may be inferred within the original dataset in terms of authors’ coun- tries of birth. Among the 200 fictitious authors, 9 share the same country of birth, the U.S., whereas authors from countries like Zimbabwe and Portugal have no apparent connections with others. Top half of Table 1 shows the average Devia- tion Scores (DS) (see Sec. 4) of unlearning three randomly selected U.S. authors (who represent the highly inter-connected entity) compared to those from countries with a single author (who represent lowly inter-connected entity). The results reveal only a marginal difference in unlearning perfor- mance, suggesting that the model, when finetuned Table 1:Deviation Scores (↓) of unlearning a highly inter- connected entity (represented by US author) and a lowly inter- connected entity (represented by an author from a country with only one representative) for various unlearning methods using Llama2-7B. MethodUSA Country w/ 1 Author∆ Original TOFU GA40.837.43.4 GD40.736.04.7 UKL73.371.02.2 DPO23.521.71.8 NPO45.442.33.1 Structured TOFU GA45.439.65.9 GD44.839.35.5 UKL 70.368.41.9 DPO30.327.62.8 NPO45.542.13.3 on the original TOFU dataset, has a limited appreci- ation of its weak inferred entity inter-connectivity. Then, we further explore whether the original TOFU dataset can be modified to strengthen the inter-connectivity between certain authors. To this end, we modify the original dataset by introducing a more explicit knowledge type, based on personal relationships, than the country of birth. We call it as the new ‘Structured TOFU’ dataset. For each un- learning U.S. author, we select 5 other U.S. authors from the original dataset and create fictitious rela- tionships (e.g., friends, coauthors, teachers) with the selected unlearning author. We then replace 10 out of the 20 QA pairs for the unlearning author with new QA pairs about his/her relationships with the other 5 authors. These new QA pairs were gen- erated using GPT-4 and followed the same prompt format as described in the TOFU paper. We do not change QA pairs of other fictitious authors. The lower half of Table 1 presents the results of experiments conducted using the same proce- dure as for the original TOFU dataset. While intro- ducing stronger inter-connectivity among authors slightly increases the unlearning impact compared to the original TOFU dataset, the overall magnitude of difference remains small. In conclusion, these findings, coupled with TOFU’s inflexibility to assess the impact of data density and domain (see Sec. 3 and 4.1), as well as other side effects (see Sec. 3.1), highlight the need for a novel dataset to better support the study and evaluation of LLM unlearning. 3 PISTOL Dataset In this section, we first introduce the novel dataset compilationpipeline forstructural unlearningof LLMs (PISTOL) in Section 3.1 and then two datasets generated by thePISTOLin Section 3.2. 3.1 Dataset Compilation Pipeline To effectively reflect the structured nature of real- world data with well-defined and strong entity-level inter-connections, we specifically choose contracts as the basis ofPISTOLdataset. This choice brings additional benefits including (i) easy customiza- tion of the network structure to study the impact of specific target topologies, (i) straightforward incorporation of side features for exploring other research topics (e.g., temporal features for studying unlearning of outdated data), (i) a highly struc- tured format compared to other knowledge types (e.g. news articles or books, etc.) for consistent measurement of unlearning performance. Despite their advantages, contract datasets are typically con- fidential, and high-quality public-domain sources remain scarce. Coupled with the necessity of dis- tinguishing the dataset from the pre-training corpus for evaluation, we synthesize datasets based on real- world contract terms, ensuring both relevance and controlled evaluation of unlearning methods. The pipeline for compiling datasets in a con- trolled manner is illustrated in Figure 2. Firstly, we craft the overall knowledge-graph structure, taking into account the structural variation of unlearning. Then we set the contract template, each with 20 attributes to be filled in. In our datasets, we focused on two ubiquitous types of contracts, sales of goods and employment contracts, owing to their more standardized structure in contrast to other highly customized agreements. Subsequently, we generate attributes in arandommanner, taking into account the dataset size. We randomly generate 6 letters and a suffix for a company name (e.g. Empblq LLC), 4 letters for the first name and the surname of a person (e.g. Jkeq Cyfz), 3 numbers, 6 letters and a street type for an address (e.g. 442 Rcvvyy Boulevard). Other attributes such as the signing date, contractual terms, and governing jurisdiction are also randomly generated. Finally, we prepare a QA pair for each attribute. QA pairs follow a consistent querying mechanism and have concise answers to allow systematic evaluations. Templates of both types of contracts and detailed QA of our sample datasets are provided in Appendix B. PISTOLpipeline is distinctive from prior works such as TOFU from several perspectives. First,PIS- TOLnot only allows the synthesis of independent data points but also enables the design and creation Figure 2:Illustration of the Dataset Compilation Pipeline. (a)(b) Figure 3:(a) and (b) illustrates the structure of Dataset 1 and 2 respectively. of inter-connected data, reflecting this fundamental characteristic of real-world data structures. Sec- ondly,PISTOLdoesnotdepend on GPT orany other pre-trained models for generating synthetic data, minimizing the risk of confounders. Thirdly, ground truth answers inPISTOLare designed to be extremely precise and concise, containing only the targeted information for unlearning. This contrasts with TOFU which consists of open-ended QAs. As common evaluation metrics (e.g., ROUGE score, etc.) (Liu et al., 2024; Romandini et al., 2024) compare generated responses with ground truth an- swers, the presence of tokens unrelated to unlearn- ing may pollute the scores, resulting in inaccurate evaluation. By ensuring concise and precise QAs, PISTOLmitigates the issue and provides a clearer measure of unlearning success. 3.2 Dataset For the purpose of this research, we introduce two datasets compiled based onPISTOLas below, with more details in Appendix C. Dataset 1.As depicted in Figure 3(a), has a struc- ture ofG(24,20)topology (a graph with24nodes and20edges). It contains two data domains – sales contracts between companies or employment con- tracts between companies and individuals. Dataset 1 has a symmetric structure, allowing for the con- trolled isolation of topological impact when eval- uating multiple structural features, such as entity inter-connectivity and data domain. Dataset 2.Despite its advantages, the symmetric structure of Dataset 1 limits topological variations when sampling different unlearning edges, provid- ing only a ‘local view’ of the impact of data inter- connectivity. To achieve greater sampling variabil- ity and verify our findings with respect to data inter- connectivity under the Dataset 1, we go beyond its symmetric structure and introduce a Dataset 2 as depicted in Figure 3(b). Dataset 2 comprises 3 sub-graphs – each has 10 nodes but different inter-connectivity (edges). The most datasparsesub-graph has 9 edges (i.e. con- necting each node by achain). On the opposite end, the most datadensesub-graph has 45 edges (i.e. nodes arefully connected). The sub-graph in be- tween issemi-denseand has 21 edges. As Dataset 2 is designed for evaluating data inter-connectivity from a ‘global view’, we isolate the data domain and only have sales contracts between companies. 4 Evaluation Setup In this section, we introduce the experimental and evaluation setup and evaluation methods for the new structural LLMs unlearning considerations. Metrics.We measureunlearning effectiveness through two key aspects: forget efficacy, which captures how much the model’s outputs diverge from the forget set, and model utility, which re- flects the preserved performance on data outside the forget set. Since these objectives are equally critical, we useDeviation Score (DS)= 100× q ROUGE1 2 forget + (1−ROUGE1 retain ) 2 to mea- sure the Euclidean distances of forget efficacy and model utility to their respective ideal state. A lower DS indicates more effective unlearning, signify- ing a closer approach to the optimal state – where the model outputs no information from the forget set while maintaining full accuracy on the retained data. In contrast, a higher DS reflects poorer un- learning, suggesting a weaker distinction between forget and retained knowledge. More details and other supplementary metrics, including the original ROUGE1 scores, MRR and the Top Hit Rate, can be found in Appendix D. Unlearning baselinesWe experiment with three gradient-based methods: Gradient Ascent (GA) (Jang et al., 2022; Yao et al., 2023), Gradient Difference (GD) (Liu et al., 2022) and GA with KL-divergence regularization (UKL), as well as two preference optimization(PO)-based methods: Direct Preference Optimization (DPO) (Rafailov et al., 2024) and Negative Preference Optimization (NPO) (Zhang et al., 2024). Given the nascence of the field, existing unlearning methods often lack robustness. However, these methods represent the current mainstream and serve well to demonstrate the impact of structural datasets while inspiring further research. Base models.We evaluate all baseline methods using the current widely adopted language mod- els Llama2-7B (Touvron et al., 2023), Gemma- 7B (Team et al., 2024) and Mistral-7B (Jiang et al., 2023). We evaluated learning rates between 1×10 −6 and5×10 −5 during unlearning and found that all methods are highly sensitive to learning rate and batch size selection. Since successful un- learning must preserve model utility, we enforce a performance threshold for ROUGE1 of the retained dataset and select the learning rate that maximizes forgetting. We conduct extensive learning rate tun- ing for each method and model, as all baseline unlearning methods demonstrate significant sensi- tivity to it – a characteristic that poses challenges for their practical implementation. To ensure a fair comparison, we hold the learning rate constant when testing unlearning performance across differ- ent data (i.e., data with varying degrees of inter- connectivity or from different domains), provided the method and base model remained unchanged. Details of fine-tuning and unlearning methods are included in the Appendix E and F. 4.1 Evaluation Methods Impact of data inter-connectivity.We define the inter-connectivity of a data point (edge) as the total degree of the vertices (entities) it con- nects:deg(e i ) = P v∈e i deg(v)−1. A higher de- gree indicates greater inter-connectivity. In Dataset 1, companiesAandBhave signed 8 and 7 con- tracts, respectively (including the contract between AandB). Thus, edgeAB, with a degree of 14, exhibits higher inter-connectivity than edgeAC, with a degree of 8. To assess the impact of inter- connectivity, we compare the outcomes of unlearn- ing contracts in the same domain but with different inter-connectivity levels (e.g., unlearning sales con- tractsABandACrespectively). To assess from a ‘global’ data inter-connectivity perspective (i.e.,the impact of data density on un- learning performance), we exploit the asymmetric nature of Dataset 2 by randomly selecting an edge to forget in each sub-graph with different data den- sities. We perform experiments three times and take average of the results. Impact of data domain.We contrast unlearn- ing a sales contract between companyAand com- panyCwith unlearning an employment contract between companyAand individualn. Model util- ity post-unlearning is evaluated on the independent retained sales edge (EF) and employment edge (Eq), respectively. As bothEFandEqbelong to an isolated sub-graph, this design isolates the effect of data domain from confounding factors such as data inter-connectivity. 5 Results In this section, we report the experimental results and discuss our key findings. Takeaway 1:The greater the degree of inter- connectivity a data point has, the more chal- lenging it becomes to unlearn the data. Table 2 compares the results of unlearning a highly inter-connected data point (AB) versus a less inter-connected one (AC) using various base- line methods and models. The results show that the Deviation Scores (DS) for unlearning the higher inter-connected dataABare consistently higher across all unlearning methods, with the GD method on Llama2-7B exhibiting the largest gap – up to 2.2x higher than the score for unlearning the less inter-connected dataAC 1 . Higher DS for unlearn- ing higher inter-connected data indicate weaker sep- aration between output accuracy on the forget and 1 A higher separation of DS scores, compared to evaluations using LlamA2-7B model on the Structured TOFU dataset in Sec. 2, further highlights the clearer inter-connectivity in PISTOL– one of its key advantages. Table 2:Performance of unlearning a highly inter-connected data (AB) versus a lowly inter-connected data (AC) in Dataset 1 across various unlearning baselines and models. Deviation Scores (↓) of unlearningABis consistently higher than those ofAC, demonstrating a positive correlation between the level of data inter-connectivity and unlearning difficulty. Llama2MistralGemma ForgetForgetForgetRetainDeviationForgetRetainDeviationForgetRetainDeviation DataMethodROUGE1 (↓)ROUGE1 (↑)Score (↓)ROUGE1 (↓)ROUGE1 (↑)Score (↓)ROUGE1 (↓)ROUGE1 (↑)Score (↓) AB GA0.521±0.0500.845±0.04354.40.325±0.0290.851±0.01035.80.563±0.0240.879±0.00657.6 GD0.654±0.0290.944±0.00365.60.347±0.0750.906±0.02136.00.319±0.0800.844±0.04635.5 UKL0.700±0.0500.936±0.03670.30.667±0.0520.969±0.01466.80.730±0.0620.916±0.00673.5 DPO0.300±0.0000.902±0.02331.60.150±0.0500.878±0.01919.30.193±0.0010.785±0.03328.9 NPO0.380±0.0000.880±0.00639.80.312±0.0250.818±0.00136.10.328±0.0250.845±0.02839.4 AC GA0.267±0.0290.805±0.00533.10.184±0.0580.890±0.01021.40.502±0.0210.917±0.00550.9 GD0.283±0.0290.920±0.00529.40.261±0.0770.959±0.01026.40.309±0.0080.893±0.00632.7 UKL0.505±0.2560.793±0.16554.60.739±0.0100.968±0.00874.00.600±0.0680.943±0.00360.3 DPO0.242±0.0000.933±0.00825.10.103±0.0000.949±0.00411.50.180±0.0500.808±0.02526.3 NPO0.363±0.0350.897±0.00237.80.300±0.0100.902±0.00631.50.374±0.0160.916±0.00638.4 retain sets, and thus a more challenging unlearning process. Additional results for other metrics can be found in Appendix G. The results also show that the difference is par- ticularly pronounced when unlearning with the GA and GD methods, whereas data inter-connectivity has a smaller impact on the UKL method. How- ever, the latter is primarily due to the overall poor performance of UKL. UKL is highly sensitive to learning rate adjustments, and to maintain a re- tain ROUGE1 score at a sensible level for pre- serving retained model utility, the forget ROUGE1 remains above0.5– an unacceptably high value, indicating a failure to effectively unlearn. This phenomenon is likely attributable to the design of UKL, which aims to reduce distribution shifts be- fore and after the forgetting process. By preserving relational information between data points, UKL not only makes it more challenging to forget less inter-connected information (ACedge) but also in- creases the difficulty of forgetting other data points. Preference optimization (PO)-based methods show less sensitivity to data inter-connectivity de- spite being highly sensitive to learning rate changes. One reason for this could be that while the GA- based methods involve gradient ascent on the forget samples, PO-based methods continues to perform gradient descent, following the gradient of the for- get set paired with negative examples such as ‘I don’t know’. This suggests that PO-based methods may be more robust for structural unlearning. Takeaway 2:The greater density a knowl- edge graph has, the more challenging it be- comes to unlearn the data within the graph. Table 3 compares the results of unlearning data from sub-graphs of different densities in Dataset 2 across various baselines and methods. The results indicate that unlearning becomes more challenging as knowledge graph becomes denser. The DS for the GA method using Llama2-7B increases from 43.0to57.6as the average data inter-connectivity (i.e., knowledge density) increases from a sparse to a dense sub-graph. Similar upward trends are observed for the GD method. As before, the UKL method proves to be the least effective, consistently yielding poor unlearning performance in all set- tings. Consistent with the results for Dataset 1, the DPO method appears the most robust to variations in knowledge density, as evidenced by its relatively stable DS across sub-graphs with different densi- ties in Dataset 2, with differences remaining within the statistical margin of error. As explained in Section 3.2, randomly sampling unlearning data in Dataset 2 allows us to evaluate the impact of structured dataset beyond the perspec- tive of an individual entity’s degree of connectivity. Instead, it enables assessment from a ‘global view’ of knowledge graph density. These findings not only support our earlier observations inTakeaway 1, but also confirm a positive correlation between knowledge graph density and unlearning difficulty. Takeaway 3:Unlearning a specific type of data may lead to greater performance deteri- oration on data of the same type compared to data of a different type. The structured nature of real-world data implies that information can belong to broader categorical domains. We seek to address how an unlearned model, after applying existing unlearning methods, performs on retained data within the same domain as the unlearned data compared to data from differ- ent domains. Table 3:Performance of unlearning data from sub-graphs of various densities in Dataset 2 across various unlearning baselines and models. Deviation Scores (↓) of unlearning data from a dense sub-graph is consistently higher than those from less dense sub-graphs, demonstrating a positive correlation between density of knowledge graph and and unlearning difficulty. Llama2MistralGemma ForgetForgetForgetRetainDeviationForgetRetainDeviationForgetRetainDeviation DataMethodROUGE1 (↓)ROUGE1 (↑)Score (↓)ROUGE1 (↓)ROUGE1 (↑)Score (↓)ROUGE1 (↓)ROUGE1 (↑)Score (↓) Dense GA0.576±0.0930.997±0.00357.60.403±0.0960.961±0.03540.50.412±0.0250.944±0.02141.6 GD0.623±0.1150.999±0.00262.30.435±0.0790.968±0.03443.60.369±0.0560.956±0.02737.2 UKL0.302±0.0800.528±0.11056.00.613±0.0790.971±0.00461.30.628±0.1260.872±0.05464.1 DPO0.230±0.0570.998±0.00423.00.050±0.0610.993±0.0065.00.087±0.0130.855±0.00516.9 NPO0.099±0.0390.818±0.03720.80.408±0.0170.988±0.01340.90.386±0.0460.967±0.01338.8 Semi-Dense GA0.469±0.0550.999±0.00246.90.380±0.0870.977±0.01238.10.360±0.0730.940±0.00336.5 GD0.598±0.0741.000±0.00059.80.377±0.0890.983±0.00737.70.277±0.0730.901±0.00829.4 UKL0.258±0.0860.405±0.14164.90.450±0.1000.886±0.05046.40.775±0.0750.960±0.02077.6 DPO0.220±0.0550.996±0.00422.00.075±0.0560.991±0.0047.60.031±0.0190.846±0.02115.7 NPO 0.177±0.0640.909±0.01621.20.399±0.0770.977±0.00940.00.350±0.0370.938±0.01035.6 Sparse GA0.430±0.1040.994±0.00443.00.268±0.0980.938±0.02727.50.290±0.0770.893±0.02230.9 GD 0.538±0.1440.997±0.00353.80.272±0.1020.957±0.01727.50.234±0.0200.826±0.04629.1 UKL0.197±0.0850.478±0.09255.80.407±0.0430.887±0.02942.50.550±0.4500.827±0.22360.5 DPO0.220±0.0910.980±0.02322.10.030±0.0270.976±0.0103.80.031±0.0190.831±0.00417.1 NPO0.099±0.0390.818±0.03720.80.231±0.0100.957±0.00523.50.236±0.0220.871±0.02626.9 Table 4:Performance of unlearning data of different domains in Dataset 1 across various unlearning baselines and models. When the targeted unlearning data is a sales contract (AC), the ROUGE1 score of retained data within the same domain (i.e., sales contract dataEF) drops more than the ROUGE1 score of retained data from a different domain (i.e., employment contract dataEq). This illustrates that unlearning data skewed toward a specific domain would lead to a more pronounced deterioration in the retained model’s performance on that same domain. Llama2MistralGemma Ind. RetainedInd. RetainedInd. RetainedInd. RetainedInd. RetainedInd. Retained ForgetForgetForgetSales Set (EF)Emp. Set (Eq)ForgetSales Set (EF)Emp. Set (Eq)ForgetSales Set (EF)Emp. Set (Eq) DataMethodROUGE1 (↓)ROUGE1 (↑)Score (↑)ROUGE1 (↓)ROUGE1 (↑)Score (↑)ROUGE1 (↓)ROUGE1 (↑)Score (↑) AC (Sales) GA0.267±0.0290.772±0.0090.955±0.0100.184±0.0580.905±0.0100.983±0.0000.502±0.0210.946±0.0040.994±0.004 GD0.283±0.0290.911±0.0100.961±0.0000.261±0.0770.963±0.0031.000±0.0000.309±0.0080.888±0.0200.990±0.004 KL0.505±0.2560.772±0.1860.821±0.1190.739±0.0101.000±0.0001.000±0.0000.600±0.0680.939±0.0080.991±0.007 DPO 0.242±0.0000.939±0.0100.983±0.0000.103±0.0000.967±0.0170.983±0.0000.180±0.0500.767±0.0500.967±0.017 NPO0.363±0.0350.928±0.0080.967±0.0000.300±0.0100.881±0.0080.983±0.0000.374±0.0160.961±0.0080.998±0.003 An (Emp.) GA0.263±0.0000.950±0.0290.744±0.0130.345±0.1100.904±0.0130.839±0.0290.300±0.0250.994±0.0060.742±0.000 GD0.296±0.0290.983±0.0170.886±0.0130.286±0.0350.933±0.0000.950±0.0100.283±0.050.978±0.0060.835±0.026 KL0.241±0.0290.819±0.0030.844±0.0670.828±0.0091.000±0.0000.978±0.0090.273±0.0150.936±0.0310.737±0.001 DPO0.075±0.0250.989±0.0100.939±0.0100.010±0.0000.983±0.0000.978±0.0090.037±0.0090.928±0.0160.856±0.008 NPO 0.347±0.0160.972±0.0081.000±0.0000.037±0.0370.842±0.0250.631±0.0040.300±0.0250.986±0.0030.742±0.000 As shown in Table 4, the results demonstrate that unlearning data skewed toward a specific domain often leads to a more pronounced deterioration in the model’s performance on retained data within that same domain. Specifically, ROUGE1 scores for independent retained sales contracts are lower than those for independent retained employment contracts when the unlearning dataACis also a sales contract. Conversely, when the unlearning data is switched toAn, an employment contract, the ROUGE1 scores for independent retained em- ployment contracts decrease more than those for independent retained sales contracts. This phe- nomenon is particularly pronounced when using the GA method, highlighting its greater lack of robustness in handling unlearning data skewed to- ward a specific domain compared to other methods. 6 Inter-connectivity of Pre-training Data ThePISTOLpipeline were designed to construct fully synthetic datasets for studying LLM unlearn- ing in a controlled environment, mitigating con- founding risks with data from the pre-training cor- pus. In this section, we show how we can make slight tweaks on it to investigate the effects of data inter-connectivity on unlearning data that has been incorporated into the pre-trained model. Given the vast number of tokens used in train- ing modern LLMs (e.g., 2T for Llama2-7B (Tou- vron et al., 2023) and significantly more for ad- vanced versions (Dubey et al., 2024)), it is prac- tically impossible to determine the exact data in- cluded or its distribution within the pre-training corpus (Shi et al., 2023; Longpre et al., 2023). As a proxy, we select two distinct but widely recog- nized entities – ‘Goldman Sachs’ and ‘William Shakespeare’ – to replace an entity’s name and address within the dense sub-graph of Dataset 2. These entities are prevalent enough to have been included in the pre-training dataset, with ‘Goldman Sachs’ appearing in1.5x more web search results than ‘Willian Shakespeare’, a notable considera- Table 5:Performance of unlearning highly inter-connected pre-training data (‘Goldman Sachs’) and less inter-connected data (‘William Shakespeare’) for various unlearning baselines and models. Higher Deviation Scores (↓) observed for ‘Goldman Sachs’ indicate that the challenge of unlearning highly inter-connected data extends to pre-training data, supporting our finding that more inter-connected data is harder to forget, consistent with results for synthetic data. Llama2MistralGemma ForgetForgetForgetRetainDeviationForgetRetainDeviationForgetRetainDeviation DataMethodROUGE1 (↓)ROUGE1 (↑)Score (↓)ROUGE1 (↓)ROUGE1 (↑)Score (↓)ROUGE1 (↓)ROUGE1 (↑)Score (↓) Goldman Sachs GA 0.621±0.1061.000±0.00062.10.451±0.1200.995±0.00445.10.563±0.0390.975±0.01956.3 GD0.704±0.0531.000±0.00070.40.485±0.0970.997±0.00248.50.509±0.1180.982±0.01950.9 KL0.357±0.1010.660±0.08149.30.732±0.1631.000±0.00073.20.722±0.2380.972±0.02672.3 DPO0.387±0.1931.000±0.00038.70.037±0.0271.000±0.0003.80.483±0.0621.000±0.00048.3 NPO 0.296±0.0360.997±0.00529.60.492±0.0220.995±0.00049.20.414±0.1130.980±0.01841.5 William Shakespeare GA0.582±0.0950.995±0.00958.20.283±0.0360.987±0.00228.40.425±0.0130.955±0.00542.7 GD0.663±0.0651.000±0.00066.30.433±0.1590.995±0.00443.30.412±0.1440.910±0.07242.2 KL0.375±0.0660.597±0.07555.00.786±0.0811.000±0.00078.60.558±0.2080.841±0.14458.1 DPO0.337±0.0611.000±0.00033.70.050±0.0410.997±0.0055.00.030±0.0300.820±0.01518.2 NPO0.245±0.0381.000±0.00024.50.417±0.0670.992±0.00841.70.398±0.1400.941±0.02440.2 tion given that web pages serve as a primary source for pre-training data. (Penedo et al., 2024; Wenzek et al., 2019). Comparison between the performance of un- learning pre-training data in Table 5 and synthetic data in the dense sub-graph in Table 4 demonstrates that unlearning data presented in the pre-training dataset is indeed more challenging – as entities in the pre-training data have other relationships em- bedded within pre-training data, making them more inter-connected compared to the synthetic entities created in Dataset 2. The results in Table 5 further shows that ‘Gold- man Sachs’, as a more inter-connected data point, is more difficult to unlearn compared to ‘William Shakespeare’, evidenced by consistently higher DS scores. These results further support our finding that more inter-connected data are harder to for- get. They also highlight the real-world significance of developing and evaluating unlearning methods’ capability of addressing the inherent challenges associated with unlearning highly inter-connected data in practical applications. 7 Related Works Machine Unlearning.MU emerged as a tool to exercise the right to be forgotten (RTBF) (Euro- pean Union, 2016). Soon after its inception (Cao and Yang, 2015), probabilistic definitions of MU, inspired by differential privacy, were introduced (Ginart et al., 2019; Sekhari et al., 2021). These definitions enabled the development of several cer- tified unlearning algorithms, particularly for deep learning models (Guo et al., 2019; Zhang et al., 2022; Golatkar et al., 2020a,b). However, due to ef- ficiency concerns and the difficulty of maintaining strong assumptions, uncertified approximate MU methods have gained more traction (Kurmanji et al., 2023; Foster et al., 2024; Sendera et al., 2025; Dang et al., 2025). Early research primarily focused on unstructured data, such as images, with only a few studies exploring structured tabular data (Kurmanji et al., 2024; Oesterling et al., 2024). LLM Unlearning.Recent works have exten- sively investigated unlearning in generative mod- els, including text-to-image models (Fuchi and Takagi, 2024; Kumari et al., 2023; Zhang et al., 2025) and large language models (LLMs) (Shen et al., 2025; Liu et al., 2024). The motivations for LLM unlearning extend beyond RTBF and include removing copyrighted content (Eldan and Russi- novich, 2023), mitigating undesirable behaviors (Dige et al., 2024), and enhancing safety (Zhao et al., 2024). Existing LLM unlearning meth- ods include simple gradient ascent (GA) based approaches (Liu et al., 2024; Barbulescu and Tri- antafillou, 2024), preference-optimization-based techniques (Rafailov et al., 2024; Zhang et al., 2024), surgical activation engineering (Shen et al., 2025; Seyito ̆ glu et al., 2024), and even in-context unlearning (Pawelczyk et al., 2023). Evaluation and Benchmarks.Evaluating un- learning methods remains a significant challenge (Triantafillou et al., 2024), primarily due to two factors: (1) constructing suitable datasets and (2) defining effective evaluation metrics. Unlearning methods are typically assessed using task-specific metrics (Kurmanji et al., 2023) and privacy attacks (Triantafillou et al., 2024), while some works try to devise new evaluation criteria (Shi et al., 2024). From a dataset perspective, researchers often re- purpose existing ML datasets for unlearning by introducing a new split, where a subset of train- ing data is designated as the ‘forget set’. More recently, specialized datasets have been curated for LLM unlearning (Maini et al., 2024; Li et al., 2024). However, these datasets are usually static, limiting flexibility in size. Moreover, they are susceptible to leakage, potentially influencing the pre-training of newer models and introducing confounding factors when evaluating baselines. A major limitation of existing benchmarks is that they primarily focus on independent data points without accounting for structural dependencies between them.PISTOL resolves all these problems. 8 Conclusion In this work, we introducedPISTOL, a pipeline for synthesizing structured datasets that uniquely captures the structural relationships among enti- ties to address a critical gap in LLM unlearning: the neglect of data inter-connectivity. Unlike ex- isting benchmarks such as TOFU,PISTOLpro- vides clearer evaluation framework and mitigates confounding risks introduced by pre-trained mod- els. Through experiments on datasets synthesized usingPISTOL, we demonstrated that data inter- connectivity significantly impacts unlearning diffi- culty. Specifically, we found that (a) unlearning be- comes harder as data inter-connectivity grows, (b) the density of the underlying knowledge graph pos- itively correlates with unlearning difficulty, and (c) unlearning domain-skewed data often leads to dis- proportionate degradation in the retained model’s performance on that same domain. 9 Limitations WhilePISTOLprovides a novel tool for structural dataset compilation for realistic LLM unlearning, several exciting avenues remain open for explo- ration. First,PISTOLcurrently produces synthetic datasets, which, while carefully designed, may dif- fer from real-world distributions. The studied base- lines could behave differently under different distri- butions. It would be interesting to extendPISTOL to compile even naturally structured texts to pro- duce real-world datasets. Second, graph-based rep- resentations of structured knowledge exist in mul- tiple modalities, including vision and multimodal learning. Extending our approach beyond textual data could help assess how unlearning manifests in different input formats and model architectures. Finally, our work primarily contributes a dataset compilation pipeline, but a complementary chal- lenge lies in defining robust evaluation metrics for unlearning effectiveness. Developing principled measures that account for several important aspects including forgetting quality, privacy, and model’s utility remains an important direction for future research. References George-Octavian Barbulescu and Peter Triantafillou. 2024. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097. Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Car- lini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. 2023. Identifying and mitigating the security risks of generative ai.Foundations and Trends®in Privacy and Security, 6(1):1–52. Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency, pages 610–623. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Ma- chine unlearning. In2021 IEEE Symposium on Secu- rity and Privacy (SP), pages 141–159. IEEE. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In2015 IEEE symposium on security and privacy, pages 463– 480. IEEE. Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Se- bastian Riedel, and Mikel Artetxe. 2023. Improving language plasticity via pretraining with active forget- ting.Advances in Neural Information Processing Systems, 36:31543–31557. Yihong Chen, Pasquale Minervini, Sebastian Riedel, and Pontus Stenetorp. 2021. Relation prediction as an auxiliary training objective for improving multi- relational graph representations. In3rd Conference on Automated Knowledge Base Construction. US Congress. 2024. Generative ai copyright disclosure act of 2024. Yizhou Dang, Yuting Liu, Enneng Yang, Guibing Guo, Linying Jiang, Jianzhe Zhao, and Xingwei Wang. 2025. Efficient and adaptive recommendation un- learning: A guided filtering framework to erase out- dated preferences.ACM Transactions on Information Systems, 43(2):1–25. Omkar Dige, Diljot Arneja, Tsz Fung Yau, Qixuan Zhang, Mohammad Bolandraftar, Xiaodan Zhu, and Faiza Khattak. 2024. Can machine unlearning reduce social bias in language models? InProceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing: Industry Track, pages 954–969. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms.arXiv preprint arXiv:2310.02238. European Union. 2016. General data protection regula- tion (gdpr).Official Journal of the European Union, L 119:1–88. Jack Foster, Stefan Schoepf, and Alexandra Brintrup. 2024. Fast machine unlearning without retraining through selective synaptic dampening.Proceedings of the AAAI Conference on Artificial Intelligence, 38(11):12043–12051. Masane Fuchi and Tomohiro Takagi. 2024. Erasing con- cepts from text-to-image diffusion models with few- shot unlearning.arXiv preprint arXiv:2405.07288, 2. Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019. Making ai forget you: Data deletion in machine learning.Advances in neural information processing systems, 32. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020a. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InProceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 9304–9312. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020b. Forgetting outside the box: Scrubbing deep networks of information accessible from input- output observations. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part XXIX 16, pages 383–398. Springer. Google Spain Case. 2014. Internet law — protection of personal data — court of justice of the european union creates presumption that google must remove links to personal data upon request. — case c-131/12, google spain sl v. agencia española de protección de datos (may 13, 2014). Chuan Guo, Tom Goldstein, Awni Hannun, and Lau- rens Van Der Maaten. 2019.Certified data re- moval from machine learning models.arXiv preprint arXiv:1911.03030. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models.arXiv preprint arXiv:2106.09685. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models.arXiv preprint arXiv:2210.01504. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b.arXiv preprint arXiv:2310.06825. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. InProceedings of The ACM Collective Intelligence Conference, pages 12–24. Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. 2023. Ablating concepts in text-to-image diffusion models. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 22691–22702. Meghdad Kurmanji, Eleni Triantafillou, and Peter Tri- antafillou. 2024. Machine unlearning in learned databases: An experimental analysis.Proceedings of the ACM on Management of Data, 2(1):1–26. Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. 2023. Towards unbounded machine unlearning.Advances in neural information processing systems, 36:1957–1987. Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. 2018. Canonical tensor decomposition for knowledge base completion. InInternational Conference on Machine Learning, pages 2863–2872. PMLR. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. 2024. The wmdp benchmark: Measuring and reducing malicious use with unlearning.URL https://arxiv. org/abs/2403.03218. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81. Chin-Yew Lin and Eduard Hovy. 2003. Automatic eval- uation of summaries using n-gram co-occurrence statistics. InProceedings of the 2003 human lan- guage technology conference of the North American chapter of the association for computational linguis- tics, pages 150–157. Chin-Yew Lin and Franz Josef Och. 2004.Auto- matic evaluation of machine translation quality using longest common subsequence and skip-bigram statis- tics. InProceedings of the 42nd annual meeting of the association for computational linguistics (ACL- 04), pages 605–612. Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. InConference on Lifelong Learning Agents, pages 243–254. PMLR. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. 2024. Rethinking machine unlearning for large language models.arXiv preprint arXiv:2402.08787. Shayne Longpre, Gregory Yauney, Emily Reif, Kather- ine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2023. A pretrainer’s guide to training data: Measur- ing the effects of data age, domain coverage, quality, & toxicity.arXiv preprint arXiv:2305.13169. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121. Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe. 2023. Mini-model adaptation: Effi- ciently extending pretrained models to new languages via aligned shallow training. InFindings of the As- sociation for Computational Linguistics: ACL 2023, pages 5474–5490. Fabio Motoki, Valdemar Pinho Neto, and Victor Ro- drigues. 2023. More human than human: Measuring chatgpt political bias.Public Choice, pages 1–21. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Flo- rian Tramèr, and Katherine Lee. 2023. Scalable ex- traction of training data from (production) language models.arXiv preprint arXiv:2311.17035. Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. 2022. A survey of machine unlearning.arXiv preprint arXiv:2209.02299. Alex Oesterling, Jiaqi Ma, Flavio Calmon, and Himabindu Lakkaraju. 2024. Fair machine unlearn- ing: Data removal while mitigating disparities. In International Conference on Artificial Intelligence and Statistics, pages 3736–3744. PMLR. Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2023. In-context unlearning: Language models as few shot unlearners.arXiv preprint arXiv:2310.07579. Guilherme Penedo, Hynek Kydlí ˇ cek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model.Advances in Neu- ral Information Processing Systems, 36. Nicolò Romandini, Alessio Mora, Carlo Mazzocca, Re- becca Montanari, and Paolo Bellavista. 2024. Feder- ated unlearning: A survey on methods, design guide- lines, and evaluation metrics.IEEE Transactions on Neural Networks and Learning Systems. Phillip Schneider, Tim Schopf, Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. 2022. A decade of knowledge graphs in natural language pro- cessing: A survey.arXiv preprint arXiv:2210.00105. Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086. Marcin Sendera, Łukasz Struski, Kamil Ksi ̨a ̇ zek, Kryspin Musiol, Jacek Tabor, and Dawid Rymar- czyk. 2025. Semu: Singular value decomposition for efficient machine unlearning.arXiv preprint arXiv:2502.07587. Atakan Seyito ̆ glu, Aleksei Kuvshinov, Leo Schwinn, and Stephan Günnemann. 2024.Extracting un- learned information from llms with activation steer- ing.arXiv preprint arXiv:2411.02631. William F Shen, Xinchi Qiu, Meghdad Kurmanji, Alex Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, and Nicholas D Lane. 2025. Lunar: Llm unlearn- ing via neural activation redirection.arXiv preprint arXiv:2502.07218. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. 2024. Muse: Machine unlearning six-way evaluation for language models.arXiv preprint arXiv:2407.06460. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295. Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. 2022. On the necessity of auditable algo- rithmic definitions for machine unlearning. In31st USENIX Security Symposium (USENIX Security 22), pages 4007–4022. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. 2024. Are we mak- ing progress in unlearning? findings from the first neurips unlearning competition.arXiv preprint arXiv:2406.09073. Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. InInterna- tional conference on machine learning, pages 2071– 2080. PMLR. Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. 2023. Unveiling the implicit toxicity in large language models.arXiv preprint arXiv:2311.17391. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2019. Ccnet: Ex- tracting high quality monolingual datasets from web crawl data.arXiv preprint arXiv:1911.00359. Herbert Woisetschläger, Alexander Erben, Bill Marino, Shiqiang Wang, Nicholas D Lane, Ruben Mayer, and Hans-Arno Jacobsen. 2024. Federated learning prior- ities under the european union artificial intelligence act.arXiv preprint arXiv:2402.05968. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate lim- itation of large language models.arXiv preprint arXiv:2401.11817. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning.arXiv preprint arXiv:2310.10683. Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2023a. Right to be forgotten in the era of large language models: Implications, challenges, and solutions.arXiv preprint arXiv:2307.03941. Jinghan Zhang, Junteng Liu, Junxian He, et al. 2023b. Composing parameter-efficient modules with arith- metic operation.Advances in Neural Information Processing Systems, 36:12589–12610. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868. Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. 2025. Defensive unlearning with adver- sarial training for robust concept erasure in diffusion models.Advances in Neural Information Processing Systems, 37:36748–36776. Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. 2022. Prompt certified machine un- learning with randomized gradient smoothing and quantization.Advances in Neural Information Pro- cessing Systems, 35:13433–13455. Shuai Zhao, Xiaobao Wu, Cong-Duy Nguyen, Mei- huizi Jia, Yichao Feng, and Luu Anh Tuan. 2024. Unlearning backdoor attacks for llms with weak- to-strong knowledge distillation.arXiv preprint arXiv:2410.14425. Appendix A Demand for LLM Unlearning from a Regulatory Perspective The demand for data unlearning is rising due to increased scrutiny over transparency in data usage for training LLMs and concerns regarding the rights of developers to access and use such data. For instance, GDPR grants individuals the right to access all information held by service providers, including details on how data is used for ML training (Art.15, Rec.63 & 64). Similarly, the European Union’s Artificial Intelligence Act (the EU AI Act), which represents the first comprehensive legal framework of its kind, mandates that model providers publish a comprehensive summary of training data content (Art.52c). Additionally, in certain jurisdictions such as the EU, individuals or organizations are legally empowered to demand the deletion of data or withdraw consent for the use of their personal data (‘right-to-be- forgotten’). A pivotal case that illustrates this isGoogle Spain SL v. Agencia Española de Protección de Datos, where the Court of Justice of the European Union (CJEU) upheld the individual’s right to have links to their personal data removed under specific conditions, even when the original online publication was legal (Google Spain Case). The EU AI Act also re-confirmed EU’s regulatory stance on clients’ right to revoke their consent at any time (Art.17) (Woisetschläger et al., 2024). In the US, legal cases like"Times v OpenAI"spotlight the debate over copyright laws’ applicability to AI training, leading to broader discussions about prompting legislative measures such as the Generative AI Copyright Disclosure Bill in the US House of Representatives (Congress, 2024) to enhance transparency and accountability. These regulatory developments emphasize the importance of effective data erasure practices in ensuring LLMs’ legal compliance. BPISTOLContractual Data and QA Template Sales of Goods Contract Template SALES OF GOODS CONTRACT 1. PARTIES •This Sales Contract (hereinafter referred to as the “Contract”) is entered into on [•] (the “Effective Date”) by and between [•] with an address of [•] (the “Seller”) and [•] with an address of [•] (the “Customer”) (collectively referred to as the “Parties”). 2. GOODS AND PRICE •The goods that the Seller is selling to the Customer are enlisted below with their quantities (hereinafter referred to as the “Goods”). • Goods: [•] • Quantity: [•] • Price per unit: [•] • Total price: [•] 3. PAYMENTS •The Seller shall provide the Customer with an invoice no later than [•] days after the time of the delivery. • All invoices are to be paid in full within [•] days. Any balances not paid within [•] days will be subject to a [•]% late payment penalty. 4. DELIVERY AND SHIPPING • The delivery of the goods (the “Delivery”) will be at the location [•]. •The shipping method will be decided by the [•]. [•] will be responsible for the costs of the shipment. 5. WARRANTIES •General Warranty:The Seller hereby warrants to the Customer that the Goods shall be free from defects in materials and workmanship under normal use and service for a period of [•] years from the date of delivery (the "Warranty Period"). The Seller affirms that it has good title to the Goods free and clear of any liens and encumbrances and has the right to sell the Goods to the Customer. •Remedy for Breach of Warranty:In the event of a breach of this warranty, the Customer must notify the Seller in writing within of [•] days of discovering the defect. Upon receiving such notification, the Seller shall, at its sole option, (i) repair or replace the defective Goods at no additional charge to the Customer, or (i) refund the purchase price paid for the defective Goods, provided that the Goods are returned to the Seller, if so requested. The choice of remedy shall be at the Customer’s discretion if repair or replacement does not remedy the defect within a reasonable time. •Exclusions from Warranty:This warranty does not apply to any damage or defect resulting from misuse, abuse, neglect, alterations, unauthorized repairs, modifications, accidents, or natural wear and tear. The Seller’s obligation under this warranty is limited to the repair, replacement, or refund as provided under this section and does not cover any other costs such as the cost of removal and reinstallation of Goods, loss of use, loss of profit, or other incidental or consequential damages. •No Other Warranties:Except for the warranty set forth herein, the Seller disclaims all other warranties, express or implied, including, but not limited to, any implied warranties of mer- chantability or fitness for a particular purpose. The Seller’s liability under this warranty shall be limited to the repair, replacement, or refund as specified herein, and in no event shall exceed the purchase price of the defective Goods. •Survival:This warranty shall survive the delivery, inspection, acceptance, and payment of and for the Goods and shall inure to the benefit of the Customer and its successors and assigns. 6. INSPECTION •Hereby, the Customer acknowledges that it has relied solely on the investigations, examinations, and inspections that the Customer has chosen to make and that the Seller has afforded the Customer the opportunity for full and complete investigations, examinations, and inspections. 7. RISK OF LOSS AND TITLE •The risk of loss or damage for the goods will be on the Seller until the goods pass upon delivery to the Customer or its designee. The Title of the goods will also remain with the Seller until the goods pass upon delivery to the Customer or its designee. 8. DELAY OR FAILURE TO PERFORM AND FORCE MAJEURE •Under no circumstances will the Seller be held liable to the Customer for any delay that may occur, non-delivery or an arising fault of this Agreement that may be due to any labour dispute, shortage in transportation, delay or shortage of materials to produce the Goods, fires, accidents, Acts of God, or any other causes outside Seller’s control. The Seller will notify the Customer immediately upon realization that it will not be able to deliver the Goods as promised. Upon such notice, either Party may terminate this Agreement. 9. COOLING-OFF PERIOD •Either Party may terminate this Agreement, for any reason, within [•] days following the Effective Date of this Agreement (’Cooling-Off Period’). Termination during this Cooling-Off Period must be communicated in writing to the other Party. Following the expiration of the Cooling-Off Period, no Party shall have the right to terminate this Agreement on the basis of the Cooling-Off Period provisions. 10. LIMITATION OF LIABILITY •Under no circumstances will the Seller be liable for any indirect, special, consequential, or punitive damages (including lost profits) arising out of or relating to this Agreement or the transactions it contemplates (whether for breach of contract, tort, negligence, or other form of action). 11. SEVERABILITY •In the event that any provision of this Agreement is found to be void and unenforceable by a court of competent jurisdiction, then the remaining provisions will remain in force in accordance with the Parties’ intention. 12. ENTIRE AGREEMENT •This Agreement contains the entire agreement and understanding among the Parties hereto with respect to the subject matter hereof, and supersedes all prior agreements, understandings, inducements and conditions, express or implied, oral or written, of any nature whatsoever with respect to the subject matter hereof. The express terms hereof control and supersede any course of performance and/or usage of the trade inconsistent with any of the terms hereof. 13. GOVERNING LAW • This Agreement shall be governed by and construed in accordance with the laws of [•]. The Parties hereby agree to the terms and conditions set forth in this Agreement and such is demonstrated throughout their signatures below. QAs of the Sales of Goods Contract • Q1: What was the effective date of the contract between [seller name] and [customer name]? •Q2: What was the name of the seller in the contract with [customer name] as of [effective date]? • Q3: What was the address of [seller name] in the contract with [customer name]? •Q4: What was the name of the customer in the contract with [seller name] as of [effective date]? • Q5: What was the address of [customer name] in the contract with [seller name]? •Q6: What was the good that the seller was selling to the customer based on the contract between [seller name] and [customer name]? •Q7:What was the quantity of the good being sold based on the contract between [seller name] and [customer name]? •Q8: What was the unit price in dollars of the good being sold based on the contract between [seller name] and [customer name]? •Q9: What was the total price in dollars of the good being sold based on the contract between [seller name] and [customer name]? •Q10: By how many days after the delivery time must the seller provide the customer with an invoice based on the contract between [seller name] and [customer name]? •Q11: Within how many days must the invoice be paid in full based on the contract between [seller name] and [customer name]? • Q12: After how many days would unpaid balances incur a late payment penalty based on the contract between [seller name] and [customer name]? • Q13: What was the late payment interest rate based on the contract between [seller name] and [customer name]? •Q14: What was the address of delivery based on the contract between [seller name] and [customer name]? •Q15: Who would decide the shipping method based on the contract between [seller name] and [customer name]? •Q16: Who would be responsible for the costs of the shipment based on the contract between [seller name] and [customer name]? • Q17: What was the duration of the general warranty period in years based on the contract between [seller name] and [customer name]? •Q18: Within how many days of discovering a defect must the customer notify the seller in writing in the event of a breach of warranty based on the contract between [seller name] and [customer name]? • Q19: What was the duration of the cooling-off period in days based on the contract between [seller name] and [customer name]? • Q20: Which jurisdiction’s laws govern the contract between [seller name] and [customer name]? Employment Contract Template EMPLOYMENT CONTRACT 1. PARTIES •(the "Employer") with its principal place of business located at [•] ("Employer’s Business Address") agrees to employ [•] (the "Employee") who lives at [•] ("Employee’s Residential Address") and the Employee agrees to be employed, on the terms and conditions set out in this Contract, and in the accompanying Addendum (together, the "Agreement"). 2. START AND LENGTH OF EMPLOYMENT • The Employee will start employment on [•] ("Start Date"). •The Employer shall employ the Employee for [•] months ("Length of Employment"), however, the Employer and the Employee may change the Length of Employment in accordance with Clause 12 of this Contract. 3. JOB TITLE AND DUTIES •The Employee shall be employed as [•] ("Position"). The Employee shall perform the duties as described in the accompanying Addendum, and any other duties reasonably assigned by the Employer. 4. PLACE OF WORK •The Employee shall work at [•] ("Address of Work Location"). The Employee shall not be required to work at a different location unless the Employee consents in writing to such an arrangement. Any such employment shall be on the same terms and conditions as this Agreement. 5. WORKING HOURS •The Employee’s normal days of work are Monday to Friday ("Normal Work Days") and the Employee’s normal hours of work are [•] to [•] ("Normal Work Hours") (together, the "Work Week"). 6. PAY • The Employer shall pay the Employee $[•] ("Rate of Basic Pay") per hour. • The Employer shall pay the Employee in [•] instalments. 7. BENEFITS •The Employee shall be entitled to participate in [•] offered by the Employer, subject to the terms and conditions of those plans. 8. HOLIDAYS •The Employer shall provide the Employee with [•] days of paid holiday leave ("Holiday Leave") per year, plus the public holidays. •The Employee shall provide the Employer two weeks’ notice of any Holiday Leave, and the Employer may only refuse Holiday Leave in exceptional circumstances. The Employer shall pay the Employee for any unused Holiday Leave at the earlier of: (i) the end of each year, or (i) the end of the Employee’s employment. 9. CONFIDENTIALITY •The Employee agrees that during the term of employment and for the first [•] months thereafter, he/she will not disclose any confidential information pertaining to the business of the Employer to any person not authorized by the Employer to receive such information. 10. WORK CONDITIONS •The Employer shall ensure the Employee is appropriately instructed and trained in relation to tasks that the Employee will carry out. The Employer shall provide a safe and healthy work environment and shall not require the Employee to do work that subjects the Employee to health or safety hazards. 11. SICK PAY AND ABSENCE •The Employee shall notify the Employer if he or she is going to be absent from work because of sickness or injury. The Employer shall not require the Employee to work when sick or injured. •In each year of employment, the Employee shall be entitled to receive the Basic Rate of Pay (as if he/she had worked the Normal Work Hours) per day for the first [•] days of absence from work due to sickness or injury ("Paid Sick Leave"). 12. TERMINATION •The Employee and Employer shall each provide the other with [•] weeks’ written notice of termination in a language the Employee understands. •The Employee and Employer may agree that the Employer pay the Employee for this notice period instead of requiring the Employee to work. In exceptional circumstances, as defined in the Addendum, notice of termination is not required. • On termination, the Employee shall return to the Employer all Employer property, and the Employer shall pay immediately all monies due under this Agreement to the Employee. 13. NON-COMPETE •During the term of employment and for [•] months after the termination of employment, the Employee agrees not to engage in any business activities or employment with a competitor or in any capacity that directly competes with the Employer’s business within the United States. • This restriction applies to similar products, services, or industry sectors in which the Employer operates. The Employee acknowledges that such competition could harm the Employer’s business interests and agrees to refrain from such activities to protect the Employer’s legitimate business interests. 14. CHANGES TO EMPLOYMENT TERMS •This Contract and the attached Addendum make up the entire Agreement relating to the Em- ployee’s employment. The Employer shall not make any changes to this Agreement without the Employee’s written consent. The Employer shall provide [•] weeks’ written notice of any proposed changes in a language the Employee understands, and the Employer shall permit the Employee to ask questions about such changes. 15. ENTIRE AGREEMENT •This Agreement and the attached Addendum contain the entire agreement between the parties. The Employee acknowledges that he/she has not relied on any oral or written representations made by the Employer or its employees or agents. 16. GOVERNING LAW •This Agreement and any dispute or claim arising out of or in connection with it or its subject matter or formation (including non-contractual disputes or claims) shall be governed by and construed in accordance with the laws of [•] ("Governing Law"). I acknowledge that I have read this Contract and the Addendum to this Contract; I under- stand and accept the terms and conditions set out within it, and that this Contract, together with the Addendum, form the Agreement of Employment. QAs of the Employment Contract •Q1: What was the name of the employer in the employment contract with [employee name], which started from [start date]? •Q2: What was the principal business location of [employer name] based on the contract between [employer name] and [employee name]? •Q3: What was the name of the employee in the employment contract with [employer name], which started from [start date]? •Q4: What was the address of [employee name] based on the contract between [employer name] and [employee name]? •Q5: What was the start date based on the contract between [employer name] and [employee name]? •Q6: For how many months will the employer employ the employee based on the contract between [employer name] and [employee name]? •Q7: What was the job position based on the contract between [employer name] and [employee name]? • Q8: What was the work location based on the contract between [employer name] and [employee name]? •Q9: At what hour did the workday start based on the contract between [employer name] and [employee name]? •Q10: At what hour did the workday finish based on the contract between [employer name] and [employee name]? •Q11: What was the hourly basic pay in dollars based on the contract between [employer name] and [employee name]? •Q12: What was the frequency of salary payment based on the contract between [employer name] and [employee name]? • Q13: What benefit was provided to the employee based on the contract between [employer name] and [employee name]? • Q14: How many days of paid holiday leave were provided to the employee based on the contract between [employer name] and [employee name]? •Q15: For how many months after the employment ends was the employee prohibited from disclosing any confidential information based on the contract between [employer name] and [employee name]? • Q16: What was the number of days the employee was entitled to Paid Sick Leave in each year of employment based on the contract between [employer name] and [employee name]? •Q17: How many weeks’ written notice of termination must the employee and employer each provide to the other based on the contract between [employer name] and [employee name]? •Q18: For how many months did the non-compete clause cover based on the contract between [employer name] and [employee name]? •Q19: How many weeks’ written notice must the employer provide before any proposed changes to the terms of employment based on the contract between [employer name] and [employee name]? •Q20: Which jurisdiction’s laws govern the contract between [employer name] and [employee name]? C Details of Dataset Construction underPISTOL Each dataset is organized into columns of questions, answers, and edges, to facilitate easier selection of unlearning edges (i.e., unlearning data). The edge features in Dataset 1 consisted of the placeholder name, such asAC. In Dataset 2, each inter-connected sub-component consists of10nodes. The nodes are sequentially numbered:0−9for the sparse sub-graph,10−19for the semi-dense sub-graph, and20−29for the dense sub-graph. The sparse sub-component has a chain structure, with edges sequentially connecting nodes from0to9. The semi-dense sub-component contains27edges. The dense sub-component is a fully-connected sub-graph, meaning every pair of nodes within the sub-graph is linked by an edge. The dataset can be found in Hugging Facehttps://huggingface.co/datasets/xinchiqiu/PISTOL. Code of Ethics:ThePISTOLdataset creation pipeline as well as Dataset 1 and Dataset 2 are constructed in the manuscript do not involve any human subjects or participants. Comprehensive documentation will be maintained alongside the dataset, detailing its structure, the nature of the data, and instructions for its use. This documentation will be updated with each version of the dataset. As mentioned in the manuscript, we set the contract template each with 20 attributes to be filled in. We focused on two ubiquitous types of contracts, sales of goods and employment contracts, owing to their more standardized structure in contrast to other highly customized agreements like corporate sale and purchase agreements or share subscription agreements. Also, we generate attributes in a random manner, taking into account the dataset size. In our datasets, we randomly generate 6 letters and a suffix for a company name (e.g. Empblq LLC), 4 letters for the first name and the surname of a person (e.g. Jkeq Cyfz), 3 numbers, 6 letters, and a street type for an address (e.g. 442 Rcvvyy Boulevard). Other attributes such as the signing date, contractual terms, and governing jurisdiction are also randomly generated. Therefore, there are no privacy and copyright implications associated with our datasets. Please note that the contracts used in sample datasets are generated in a completely random manner, hence do not represent any real contracts between any real companies or individuals. D Evaluation Metrics The evaluation of unlearning presents significant challenges. (Thudi et al., 2022) demonstrates that, in certain scenarios, it is impossible to audit unlearning processes using the single metric of model losses even with access to the entire training trajectory. Although this underscores the inherent difficulties of unlearning evaluations, the analysis in (Thudi et al., 2022) does not preclude using other heuristic-based methods to assess unlearning. To address this, we propose to employ multiple metrics. The use of multiple diverse metrics allows us to alleviate the unlearning evaluation trap that certain data points of a equivalent class would produce the same metric change without effective target removal (Thudi et al., 2022). Prior unlearning benchmark use ROUGE (Maini et al., 2024). Given our focus onstructuralLLMs unlearning, we also incorporate metrics like MRR and hit ratios as they are representative metrics for structured learning communities. ROUGE score (Lin and Hovy, 2003; Lin and Och, 2004; Lin, 2004) is commonly used for text-generation tasks (e.g., QA tasks), while MRR and hit ratios are popular for entity retrieval-type tasks (e.g., knowledge graph completion) (Trouillon et al., 2016; Lacroix et al., 2018; Chen et al., 2021). Given our focus on benchmarking structural LLMs unlearning, we incorporate metrics from both of these communities to provide a comprehensive evaluation. ROUGE score:We use ROUGE scores to compare model answers (with greedy sampling) with the ground truth. Specifically, we compute the ROUGE-1 recall score (Lin, 2004), which acts as a surrogate for accuracy on the question-answering task, as it accounts for the output phrasing to be slightly different than the ground truth. Mean reciprocal rank (MRR).An answer is usually composed of multiple tokens. Therefore, we use the reciprocal average of the rank of each target (ground truth) token to measure the model’s memorization of names. Given a prefixQ, an output answer token sequenceE=e 1 , ..., e n , with the length of|E|, the model predicts the rank of the target token asrank(e i |Q), and then MRR for the nameEis calculated as follows: M R= P |E| i=1 1/rank(e i , Q) |E| (1) Top hit ratio (THR)The hit rate is a binary score for each output token, indicating the presence of the correct token at the topmvalues in the output logits, denotes ashit(e i , m). Also, given the output sequenceE=e 1 , ..., e n , and we choosem= 100in our experiments. T HR= P |E| i=1 hit(e i , m) |E| (2) E Model Fine-tuning Datasets constructed underPISTOLare synthetic with structured Q&As derived from randomly generated contractual attributes. As such, pre-trained model must first be fine-tuned on the constructed dataset to ensure the model effectively ‘remembers’ the new data points. After fine-tuning, we inference the model on questions of our dataset, and recorded that all the questions can be successfully learned and remembered by both models. As discussed in Section 3, fine-tuning on datasets constructed underPISTOL, by its design, facilitates more accurate evaluation onD fact later on due to elimination of confounding variables. For fine-tuning, we implemented the widely adopted parameter-efficient fine-tuning method LoRA (Hu et al., 2021) in our experiments. LoRA saves computation memory by optimizing over two low rank metricsB, A, whereBA=△w, instead of the entire parameters space. In all of our experiments, we optimize this loss with AdamW for20epochs and warm up for the first epoch. We use an effective batch size of16. We verify that the LLM can accurately ‘remember’ all individual data points and reaches ROUGE1 score of 1 for all models. It is worth noting that Dataset 1 is designed to be both concise and effective for studying structural LLM unlearning. This design ensures accessibility for researchers with limited computational resources. For those with larger computational budgets, Dataset 2 can be utilized, or larger datasets can be constructed in a similar manner using thePISTOLpipeline. Finetuning can be run using 1 NVIDIA A40 GPU, and the running times depend on the model and the size of dataset up to 2 hours. F Machine Unlearning We experiment with several unlearning methods summarised in the survey paper (Liu et al., 2024; Maini et al., 2024), each of which is introduced in detail in the section. Given the nascent state of LLM unlearning and the general lack of robustness in existing unlearning methods, we select mainstream approaches from two major families – gradient ascent-based and preference optimization-based methods – to illustrate the impact of structured data on LLM unlearning performance. Gradient Ascent (GA).GA is the most straightforward and intuitive method, performing gradient ascent on the forget data to maximize the likelihood of mispredictions for those samples within the forget setD f (Jang et al., 2022; Yao et al., 2023), according to the loss function: L φ (D f ) = 1 |D f | X x∈D f l φ (x)(3) It is worth noting that GA alone can be sensitive to the choice of hyperparameters during optimization, such as the number of ascent steps and the learning rate. Therefore, during the unlearning stage, the loss we aim to maximize is the average over the forget setD f . Gradient Difference (GD).Grad Difference (Liu et al., 2022) extends the idea of GA by optimizing two losses: one maximizes mispredictions on the forget set and the other minimizes mispredictions on the retained set, thus simultaneously unlearning the forget set and maintaining performance on the retained set. The combined loss function is: L φ =−L φ (D f ) +L φ (D r )(4) Given the fact that the size of the forget set is normally smaller than the retained set (otherwise, it will be more computationally efficient to simply retrain on the retained set), the mini-batch selection follows the selection from the forget set first, and then for each selected forget samples, we randomly select a retained sample to form a combined sample for the loss computation. Unlearning with KL-divergence (UKL).The UKL method aims to minimize the KL-divergence between the predictions of the original fine-tuned model and the unlearned model on the retained setD r , thereby maximizing the utility of the model on the retained data, while concurrently maximizing the loss on the forget set (Maini et al., 2024). The loss function can be expressed as below: L φ =−L φ (D f ) + 1 |D r | X x∈D r 1 |x| |x| X i=2 KL(M pretrained (x <i ||M unlearn (x <i ))(5) Direct Preference Optimization (DPO).DPO aims to align the model such that it refrains from revealing information from the forget set. The approach, inspired by the original DPO method (Rafailov et al., 2024) and following the TOFU framework (Maini et al., 2024), computes the loss usingx idk = [q, a idk ], which are question-answer pairs from the forget setD f but with the answer replaced by various expressions of ’I don’t know’. Unlike prior algorithms, DPO does not utilize gradient ascent. The loss function is: L φ =L φ (D r ) +L φ (D f,idk )(6) Negative Preference Optimization (NPO).NPO (Zhang et al., 2024) is effectively a modified version of DPO that excludes positive samples (i.e.,L φ (D r )). The gradient of NPO can also be interpreted as an adaptive weighting of the GA gradient, where the weight vanishes for unlearned samples. This mitigates the unbounded nature of simple gradient ascent, which often leads to catastrophic collapse of the unlearned model when hyperparameters are not properly tuned. Unlearning configurations.For all unlearning methods, we conduct optimization of the corresponding loss over20epochs. In scenarios where support from the retained set is utilized, an epoch is defined as one complete cycle through the entire forget set, using no more than the same number of samples from the retained set. We employ the AdamW optimizer with a warm-up phase during the first epoch and maintain an effective batch size of4for all unlearning algorithms. We evaluated learning rates between 1×10 −6 and5×10 −5 . Since successful unlearning must preserve model utility on the retained dataset, we enforce a performance threshold and select the learning rate that maximizes forgetting. All unlearning experiments can be run using 1 NVIDIA A40 GPU, and the running time depends on the size and the algorithms. G Additional Experiment Results In this section, we show additional experimental results, reporting MRR and THR metrics. Table 6:Additional results of unlearning data in Dataset 1 with different degrees of inter-connectivity. Forget DataForget MethodForget Set MRRRetain Set MRRForget Set THRRetain Set THR Llama2-7B AB GA 0.288±0.0040.310±0.0040.766±0.0240.759±0.004 GD0.277±0.0080.292±0.0020.731±0.0210.733±0.006 KL 0.355±0.0140.370±0.0170.839±0.0160.831±0.010 DPO0.261±0.0110.295±0.0020.600±0.0030.690±0.009 NPO0.234±0.0010.281±0.0030.710±0.0080.782±0.001 AC GA0.179±0.0010.271±0.0040.606±0.0040.735±0.006 GD0.168±0.0020.261±0.0020.590±0.0090.728±0.008 KL0.357±0.0310.407±0.0440.694±0.0730.784±0.047 DPO 0.159±0.0010.277±0.0010.425±0.0030.658±0.009 NPO0.186±0.0020.292±0.0010.651±0.0140.810±0.001 Mistral-7B AB GA 0.166±0.0140.269±0.0040.500±0.0130.678±0.007 GD0.163±0.0020.285±0.0040.524±0.0440.713±0.013 KL0.293±0.0280.316±0.0080.737±0.0090.784±0.004 DPO0.081±0.0190.271±0.0040.187±0.0430.631±0.019 NPO0.580±0.0220.690±0.0190.838±0.0040.914±0.002 AC GA0.134±0.0540.236±0.0040.252±0.0790.597±0.025 GD 0.149±0.0060.266±0.0050.293±0.0470.635±0.016 KL0.246±0.0020.279±0.0080.727±0.0190.757±0.008 DPO0.018±0.0010.269±0.0040.125±0.0000.676±0.001 NPO0.231±0.0080.335±0.0050.616±0.0030.790±0.002 Gemma-7B AB GA0.706±0.0130.916±0.0020.797±0.0080.944±0.006 GD 0.527±0.0720.888±0.0230.652±0.0280.930±0.006 KL0.838±0.0220.943±0.0040.923±0.0150.978±0.003 DPO0.894±0.0101.000±0.0000.954±0.0091.000±0.000 NPO0.526±0.0470.897±0.0150.716±0.0240.941±0.008 AC GA0.769±0.0140.954±0.0050.808±0.0160.983±0.007 GD0.655±0.0100.947±0.0020.700±0.0050.985±0.002 KL 0.788±0.0470.960±0.0030.924±0.0120.979±0.003 DPO0.071±0.0030.739±0.0190.316±0.0490.862±0.010 NPO0.663±0.0220.939±0.0060.768±0.0040.967±0.007 Table 7:Additional results of unlearning data in Dataset 2 with different knowledge densities. Forget DataForget MethodForget Set MRRRetain Set MRRForget Set THRRetain Set THR Llama2-7B Dense GA0.215±0.0370.276±0.0010.573±0.0620.608±0.008 GD 0.223±0.0320.277±0.0010.583±0.0490.610±0.008 UKL0.268±0.0480.376±0.0050.521±0.1490.593±0.063 DPO0.219±0.0530.278±0.0020.470±0.1000.602±0.006 NPO0.136±0.0370.296±0.0110.421±0.0520.576±0.002 Semi-Dense GA0.197±0.0390.277±0.0010.544±0.0580.609±0.006 GD0.240±0.0530.277±0.0010.584±0.0860.609±0.005 UKL0.255±0.0730.333±0.0290.422±0.1140.518±0.086 DPO0.205±0.0420.279±0.0010.460±0.0860.608±0.010 NPO0.162±0.0420.293±0.0090.421±0.0400.575±0.005 Sparse GA0.179±0.0390.278±0.0060.516±0.0560.610±0.009 GD0.202±0.0540.279±0.0030.522±0.0520.608±0.009 UKL0.226±0.0450.366±0.0220.427±0.1370.564±0.061 DPO 0.158±0.0400.280±0.0050.400±0.0350.599±0.019 NPO0.177±0.0200.273±0.0050.479±0.0050.548±0.002 Mistral-7B Dense GA0.264±0.0790.318±0.0150.566±0.1170.685±0.031 GD0.273±0.0740.320±0.0130.588±0.1420.683±0.033 UKL0.318±0.0330.319±0.0030.764±0.0380.757±0.006 DPO0.039±0.0280.307±0.0050.158±0.0950.667±0.017 NPO 0.164±0.0240.327±0.0080.524±0.0000.736±0.002 Semi-Dense GA0.380±0.0870.977±0.0120.540±0.0940.669±0.069 GD 0.377±0.0890.983±0.0070.550±0.0990.668±0.080 UKL0.803±0.1100.999±0.0020.721±0.0450.751±0.004 DPO0.075±0.0560.991±0.0040.187±0.0680.671±0.013 NPO0.151±0.0460.323±0.0020.493±0.0660.716±0.023 Sparse GA0.223±0.0490.311±0.0060.517±0.1000.657±0.051 GD0.218±0.0480.313±0.0080.520±0.1100.662±0.044 UKL 0.290±0.0440.314±0.0090.665±0.1520.735±0.045 DPO0.028±0.0130.309±0.0020.142±0.0250.670±0.017 NPO0.186±0.0630.311±0.0100.384±0.0400.665±0.017 Gemma-7B Dense GA 0.562±0.1080.972±0.0060.686±0.0760.997±0.002 GD0.542±0.1010.984±0.0090.660±0.0450.997±0.002 UKL0.671±0.1100.856±0.0620.764±0.1240.932±0.033 DPO0.066±0.0280.880±0.0140.211±0.0010.951±0.006 NPO 0.554±0.1180.982±0.0060.723±0.0930.996±0.005 Semi-Dense GA 0.514±0.1200.888±0.0920.697±0.0350.992±0.004 GD0.431±0.0970.950±0.0140.556±0.1160.988±0.008 UKL0.835±0.0150.972±0.0170.928±0.0100.953±0.061 DPO0.031±0.0190.841±0.0160.245±0.0070.924±0.011 NPO0.487±0.1160.892±0.0810.720±0.0720.992±0.001 Sparse GA0.536±0.0940.933±0.0160.713±0.0710.979±0.004 GD0.387±0.0280.894±0.0290.626±0.0000.971±0.005 UKL0.535±0.4650.718±0.2720.573±0.4270.769±0.225 DPO0.035±0.0150.803±0.0380.226±0.0960.882±0.027 NPO0.447±0.0070.913±0.0150.636±0.0600.978±0.003 Table 8:Additional results of unlearning data of different domains in Dataset 1. Forget DataForget MethodInd. Sales MRRInd. Emp. MRRInd. Sales THRInd. Emp. THR Llama2-7B AC GA 0.255±0.0020.331±0.0020.694±0.0100.726±0.008 GD0.252±0.0080.330±0.0010.702±0.0130.725±0.003 KL0.399±0.0430.417±0.0180.773±0.0530.802±0.033 DPO0.280±0.0010.316±0.0000.626±0.0090.652±0.008 NPO0.300±0.0050.314±0.0000.801±0.0040.815±0.001 An GA 0.279±0.0040.313±0.0060.673±0.0070.655±0.007 GD0.282±0.0020.322±0.0050.685±0.0050.683±0.009 KL 0.366±0.0060.399±0.0020.806±0.0070.829±0.012 DPO 0.290±0.0010.329±0.0010.708±0.0040.712±0.008 NPO 0.304±0.0000.296±0.0000.787±0.0040.767±0.006 Mistral-7B AC GA0.277±0.0050.319±0.0050.670±0.0250.710±0.022 GD 0.293±0.0030.324±0.0030.699±0.0060.731±0.015 KL0.342±0.0030.343±0.0020.805±0.0020.798±0.002 DPO0.303±0.0030.329±0.0010.732±0.0070.757±0.001 NPO0.352±0.0000.345±0.0010.811±0.0030.748±0.001 An GA0.322±0.0050.325±0.0020.757±0.0060.698±0.010 GD0.321±0.0020.324±0.0030.808±0.0120.745±0.014 KL 0.351±0.0020.346±0.0080.815±0.0020.808±0.003 DPO0.310±0.0010.320±0.0010.730±0.0050.740±0.001 NPO0.301±0.0010.293±0.0030.615±0.0370.537±0.030 Gemma-7B AC GA0.980±0.0030.997±0.0021.000±0.0001.000±0.000 GD0.957±0.0090.992±0.0040.994±0.0030.996±0.003 KL0.958±0.0060.994±0.0060.969±0.0050.995±0.006 DPO0.717±0.0560.953±0.0430.841±0.0420.980±0.020 NPO0.983±0.0021.000±0.0010.999±0.0021.000±0.001 An GA 0.999±0.0010.849±0.0000.999±0.0010.920±0.000 GD0.987±0.0040.903±0.0180.995±0.0030.964±0.010 KL0.954±0.0300.848±0.0130.973±0.0160.935±0.012 DPO 0.929±0.0100.831±0.0150.957±0.0050.918±0.008 NPO0.990±0.0070.854±0.0000.995±0.0030.930±0.006 Table 9:Additional results of unlearning pre-training data with different levels of inter-connectivity. Forget EntityForget MethodForget Set MRRRetain Set MRRForget Set THRRetain Set THR Llama2-7B Goldman Sachs GA 0.185±0.0170.277±0.0010.513±0.0310.566±0.008 GD0.188±0.0250.277±0.0010.512±0.0420.564±0.005 UKL0.266±0.0890.365±0.0060.432±0.0780.596±0.032 DPO0.191±0.0680.278±0.0030.459±0.1230.561±0.010 NPO0.113±0.0100.277±0.0010.443±0.0300.572±0.007 William Shakespeare GA 0.228±0.0450.273±0.0060.504±0.0700.557±0.011 GD0.240±0.0230.277±0.0020.524±0.0770.565±0.006 UKL 0.311±0.0580.362±0.0150.567±0.0810.630±0.048 DPO 0.240±0.0340.276±0.0020.472±0.0790.555±0.007 NPO 0.157±0.0510.277±0.0010.391±0.0190.548±0.007 Mistral-7B Goldman Sachs GA0.273±0.0110.344±0.0040.598±0.0130.732±0.016 GD 0.262±0.0110.345±0.0040.605±0.0320.733±0.018 UKL0.356±0.0390.337±0.0030.735±0.0210.776±0.012 DPO0.045±0.0260.331±0.0010.120±0.0590.703±0.009 NPO0.233±0.0410.337±0.0000.513±0.0170.695±0.006 William Shakespeare GA0.237±0.0490.323±0.0070.469±0.0800.689±0.023 GD0.255±0.0160.329±0.0030.557±0.0650.706±0.016 UKL 0.337±0.0700.321±0.0050.714±0.0300.788±0.005 DPO0.035±0.0070.318±0.0010.133±0.0190.716±0.008 NPO0.259±0.0090.331±0.0000.606±0.0180.718±0.001 Gemma-7B Goldman Sachs GA 0.737±0.0510.993±0.0060.799±0.0261.000±0.000 GD0.652±0.0730.990±0.0090.727±0.0770.996±0.003 UKL0.814±0.1620.981±0.0220.910±0.0860.991±0.011 DPO0.532±0.0321.000±0.0000.756±0.0521.000±0.000 NPO0.579±0.0980.988±0.0080.688±0.0850.994 ± 0.003 William Shakespeare GA0.633±0.0410.984±0.0030.773±0.0450.996±0.003 GD0.559±0.1310.962±0.0340.705±0.0960.986±0.015 UKL0.642±0.2460.863±0.1340.742±0.2230.929±0.071 DPO 0.051±0.0390.887±0.0060.174±0.0660.940±0.007 NPO0.587±0.1170.976±0.0110.722±0.0790.989±0.009

Instant research discovery