Paper deep dive

Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness

Rongzhe Wei, Peizhi Niu, Hans Hao-Hsun Hsu, Ruihan Wu, Haoteng Yin, Mohsen Ghassemi, Yifan Li, Vamsi K. Potluru, Eli Chien, Kamalika Chaudhuri, Olgica Milenkovic, Pan Li

Year: 2025Venue: NeurIPS 2025Area: Model EditingType: EmpiricalEmbeddings: 123

Models: GPT-4, LLaMA3-8B-Instruct, Qwen2.5-7B-Instruct

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 6:46:49 PM

Summary

The paper introduces a novel knowledge unlearning evaluation framework for LLMs that addresses the limitation of existing methods which focus on isolated fact removal. By representing knowledge as confidence-aware subgraphs and employing an inference-based evaluation protocol with an LLM judge, the authors demonstrate that correlated knowledge often allows 'forgotten' facts to persist, leading to an overestimation of unlearning effectiveness in current strategies.

Entities (5)

Knowledge Unlearning · task · 100%LLM · model · 100%Confidence-aware Knowledge Graph · framework · 95%LLM Judge · evaluation-tool · 95%Wikidata · knowledge-base · 90%

Relation Signals (3)

LLM Judge → evaluates → Knowledge Unlearning

confidence 95% · we propose employing a powerful LLM judge... to determine unlearning success

LLM → contains → Confidence-aware Knowledge Graph

confidence 90% · we represent the knowledge embedded in an LLM as a confidence-aware knowledge graph

Knowledge Unlearning → uses → Wikidata

confidence 85% · referring to an external real-world reference knowledge graph G ref (e.g., Wikidata)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Machine unlearning techniques aim to mitigate unintended memorization in large language models (LLMs). However, existing approaches predominantly focus on the explicit removal of isolated facts, often overlooking latent inferential dependencies and the non-deterministic nature of knowledge within LLMs. Consequently, facts presumed forgotten may persist implicitly through correlated information. To address these challenges, we propose a knowledge unlearning evaluation framework that more accurately captures the implicit structure of real-world knowledge by representing relevant factual contexts as knowledge graphs with associated confidence scores. We further develop an inference-based evaluation protocol leveraging powerful LLMs as judges; these judges reason over the extracted knowledge subgraph to determine unlearning success. Our LLM judges utilize carefully designed prompts and are calibrated against human evaluations to ensure their trustworthiness and stability. Extensive experiments on our newly constructed benchmark demonstrate that our framework provides a more realistic and rigorous assessment of unlearning performance. Moreover, our findings reveal that current evaluation strategies tend to overestimate unlearning effectiveness. Our code is publicly available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

122,267 characters extracted from source content.

Expand or collapse full text

Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness Rongzhe Wei 1* , Peizhi Niu 2* , Hans Hao-Hsun Hsu 3 , Ruihan Wu 4 , Haoteng Yin 5 , Mohsen Ghassemi 6 , Yifan Li 7 , Vamsi K. Potluru 6 , Eli Chien 1 , Kamalika Chaudhuri 4 , Olgica Milenkovic 2 , Pan Li 1 1 Georgia Institute of Technology, 2 University of Illinois Urbana-Champaign, 3 Technical University of Munich, 4 University of California San Diego, 5 Purdue University, 6 J.P. Morgan AI Research, 7 Tsinghua University rongzhe.wei, ichien6, panli@gatech.edu, peizhin2, milenkov@illinois.edu, ruw076, kamalika@ucsd.edu, mohsen.ghassemi, vamsi.k.potluru@jpmchase.com, hans.hsu@tum.de, yinht@acm.org, lyf21@mails.tsinghua.edu.cn Abstract Machine unlearning techniques aim to mitigate unintended memorization in large language models (LLMs). However, existing approaches predominantly focus on the explicit removal of isolated facts, often overlooking latent inferential dependen- cies and the non-deterministic nature of knowledge within LLMs. Consequently, facts presumed forgotten may persist implicitly through correlated information. To address these challenges, we propose a knowledge unlearning evaluation frame- work that more accurately captures the implicit structure of real-world knowledge by representing relevant factual contexts as knowledge graphs with associated confi- dence scores. We further develop an inference-based evaluation protocol leveraging powerful LLMs as judges; these judges reason over the extracted knowledge sub- graph to determine unlearning success. Our LLM judges utilize carefully designed prompts and are calibrated against human evaluations to ensure their trustworthi- ness and stability. Extensive experiments on our newly constructed benchmark demonstrate that our framework provides a more realistic and rigorous assessment of unlearning performance. Moreover, our findings reveal that current evaluation strategies tend to overestimate unlearning effectiveness. Our code is publicly avail- able at https://github.com/Graph-COM/Knowledge_Unlearning.git. 1 Introduction Large language models (LLMs) have achieved widespread adoption across diverse application domains due to their remarkable capacity to acquire and encode complex, interdependent knowledge from vast web corpora [1, 2]. However, this impressive capability simultaneously introduces critical risks. Notably, LLMs may inadvertently memorize and reproduce copyrighted content [3, 4], amplify social or cultural biases [5], or reveal sensitive and private information [6]. Such issues not only jeopardize user trust but can also contravene ethical principles and regulatory frameworks governing responsible AI deployment [7, 8]. In response, machine unlearning has emerged as a promising approach for selectively removing specific data points, concepts, or factual knowledge from pre- trained models [9–11]. * Authors marked with * contributed equally to this work. 39th Conference on Neural Information Processing Systems (NeurIPS 2025). arXiv:2506.05735v4 [cs.CL] 22 Oct 2025 User / Data Curator Unlearning Request painted Target LLM Unlearned LLM Unlearning Stage DaVinci Milan Gran Cavallo Mona Lisa Florence LouvreMuseum Paris Bob Alice Tim Renaissance LLM ConfidenceLevel Confident Somewhat Confident UncertainConfident Da Vinci Mona Lisa Knowledge Base Evaluation Stage Construct Supporting Subgraph Querying Target LLM Output: Subgraph with Confidence Provide Potential Entities & Relations Da V i nci Mo na L i s a L o u v r e Mus e u m P ar i s A l i c e Ren a i s s a n c e U n l e ar n e d F ac ts P r e - u n l e ar n i n g P os t - u n l e ar n i n g LLM C on fi d e n c e Le ve l Confi de nt S om e w ha t Confi de nt U nc e rt a i n Confi de nt i n f l u e n c e d Confidence-aware Supporting Subgraph Target Unlearn Knowledge Reference Knowledge Graphs Enhancing LLMJudge Reliability: In-context learning, Human EvaluationComparison, etc. Input to LLM Judge Powerful LLM Judge Subgraph Triples into Prompts No Leverage External Knowledge Judge guidance Judge Score Pre-unlearning Post-unlearning Enhancing TargetLLM Reliability: Confidence Calibration Figure 1: An Illustration of Knowledge Unlearning Framework. Despite progress in unlearning factual knowledge from LLMs, current methods mostly tackle the problem on the surface. Whether by targeting individual training examples [12–14] or erasing sets of facts linked to specific topics [11, 15], these approaches share a critical flaw: they focus on direct deletion while ignoring how deeply knowledge is interconnected within LLMs [16–18]. This is a serious oversight. It means that even when a fact seems “erased” (disappearing from direct questions), it often still lurks within the model, ready to be figured out from other related pieces of information the LLM still holds. The targeted fact can then be pieced back together through indirect questions, shattering the idea that unlearning truly worked and causing a loss of trust. For example, an LLM might “forget” that “Mount Fuji is a volcano,” only for this to be easily deduced because it still knows that “Mount Fuji has a crater at the summit” and “craters are formed by volcanic activity.” This same challenge also indirectly echoes the findings in knowledge editing studies [19, 20]: changing isolated facts often doesn’t ripple through the model’s wider reasoning, showing that surface-level unlearning isn’t genuine unlearning at all. This major gap shows that a new approach is vital. Therefore, we propose to significantly expand the concept of knowledge unlearning (KU): it must go beyond simple deletion to actively take apart the underlying knowledge structures that support the target information. Beyond overlooking correlations among knowledge facts, existing unlearning evaluation frameworks may also be inadequate in modeling how knowledge is organized and processed within LLMs. First, real-world knowledge exhibits diverse and complex correlation patterns, with inferential relationships that may be non-deterministic or context-dependent [21]. Consequently, evaluation protocols based on manually defined inference rules or fixed reasoning chains often fail to capture the probabilistic nature of these factual dependencies [22]. For example, consider unlearning the fact that Person A is the CEO of Company X. While this fact might be explicitly stated, it could also be inferred from indirect clues, such as references to Person A overseeing operations, attending board meetings, or signing executive decisions, each carrying varying degrees of evidential strength. Second, current methods frequently assume that facts within the unlearning dataset are already well internalized by the LLM. In practice, however, an LLM’s grasp of specific facts, particularly those involving rare entities or domain-specific relations, can vary significantly [14]. LLMs may exhibit only partial or uncertain knowledge of these facts even after fine-tuning. If such variability is not accounted for, it can lead to inaccurate or inflated estimates of unlearning effectiveness. We propose a novel and realistic formulation of knowledge unlearning, grounded in a confidence- aware perspective of how factual knowledge is represented within LLMs. Our framework is designed to reflect the complexities of real-world knowledge systems, where facts are often uncertain, interde- pendent, and mutually inferable. To achieve this, we represent the knowledge embedded in an LLM as a confidence-aware knowledge graph. In this graph, each triple denotes a piece of knowledge and is associated with a confidence level derived from the model’s predictions. Given a target triple for unlearning, we then probe the LLM to extract a subgraph of correlated facts that could potentially enable the inference of this targeted triple, subsequently analyzing this subgraph to determine the extent to which the target knowledge persists within the LLM. To effectively automate unlearning evaluation, we propose employing a powerful LLM judge (e.g., GPT-series models) to act as an adversarial agent. This LLM judge employs carefully crafted prompts and specific calibration procedures to reason over the extracted knowledge subgraph and assess whether and to what extent the target triple remains inferable. Such a protocol moves beyond 2 superficial-level triple recall, capturing the residual inferential capabilities embedded within the model. Furthermore, to validate our LLM-based evaluator, we compare its judgments against those of strong human adversaries to assess their alignment and reliability. Extensive experiments on large-scale, real-world encyclopedic datasets demonstrate our framework’s applicability while also highlighting the limitations of existing unlearning methods and evaluation techniques. Our experimental analysis reveals several key insights. First, correlated knowledge supports inference and can significantly reduce unlearning effectiveness, even when the target triple appears superficially erased. Second, even low-confidence yet semantically related knowledge can substantially compromise unlearning, underscoring the importance of capturing weaker associations. Finally, with careful prompt design and calibration, the LLM judge enables automated evaluation of unlearning effectiveness while producing judgments closely aligned with those of human experts. Accompanying our findings, we release two key resources: first, a benchmark for LLM probing that reflects knowledge interdependence, derived from real-world knowledge datasets; and second, an evaluation protocol tailored to the novel concept of knowledge unlearning presented herein. 2 Related Work Evaluating unlearning in LLMs remains a core challenge, with prior studies proposing various metrics to measure the effectiveness of removing specific training instances or factual knowledge [23]. Early approaches such as WHP [24] assessed unlearning by eliminating Harry Potter-related content using completion-based and token-probability-based metrics, while Wei et al. [25] investigated methods to prevent the generation of copyrighted content. Subsequent benchmark TOFU [11] considered entity-level unlearning and compared pre- and post-unlearning question-answering performance using fictitious author biographies. MUSE [13] provided a comprehensive evaluation across six distinct dimensions, and WMDP [26] specifically focused on unlearning harmful knowledge to mitigate malicious use. Additionally, RWKU [27] proposed benchmarks tailored for real-world knowledge scenarios, and Wei et al. [14] introduced a minority-aware framework to identify high-risk data points for unlearning. However, these approaches generally treat knowledge independently, overlooking the interdependencies between target facts and related knowledge in LLMs. This limitation was pointed out in [28], which first explored multi-fact interactions during unlearning but relied on deterministic knowledge modeling and rule-based evaluations, which limits their practical applicability. Another related research area is knowledge editing, which focuses on updating specific factual infor- mation within LLMs [29]. Existing evaluation frameworks typically assess whether models correctly recall edited facts in response to single-hop queries [30–38]. More recent studies have incorporated multi-hop factual chains as evaluation tools to determine if model updates successfully propagate through indirect reasoning paths [19, 20]. In contrast, our work concentrates on the knowledge unlearning task, which aims to entirely remove target knowledge. Distinctly, our framework explicitly extracts correlated knowledge subgraphs within the LLMs that support the target knowledge fact, whereas knowledge editing evaluations frequently rely on deterministic factual chains. 3 The Formulation of Knowledge Unlearning Confidence-Aware Knowledge Modeling in LLMs. We consider the scenario where a target pretrained LLM, denoted asM pretrain , is tasked with unlearning relational knowledge. We formulate the factual knowledge embedded in this LLM as a confidence-aware knowledge graphG = (E,R,T U ), whereEdenotes the set of entities,Rrepresents the set of relation types. Each knowledge fact is encoded as a knowledge quadruplet = (s,r,o,u), consisting of a subject entitys, a relationr, and an object entity o, along with a non-negative valued confidence score u∈U that reflects the LLM’s degree of belief in the fact;T U ⊆E ×R×E ×Udenotes the set of all confidence-aware facts inG. In practice, a given target facte = (s,r,o) ∈ Tis often correlated with a subset of related facts T e =t 1 ,t 2 ,...⊆T U inM pretrain , from which it can potentially be inferred. For instance, the triple (Paris, Capitalof, France)may be supported by related facts like(Paris, hosts, Élysée Palace,u 1 )and (Élysée Palace, isGovernmentSeatof, France,u 2 ). The strength of such inference depends on both the structural relationship betweeneand the supporting facts, as well as the associated confidence scoresu 1 andu 2 withinT e . To quantify the strength of such inference, it requires introducing an intrinsic judge functionf : 2 T U ×T → Y , where2 T U denotes the power set of knowledge triples. Given a supporting subsetT e and a target triplee, the functionf(T e ,e)outputs an inference score 3 quantifying how strongly the target factecan be derived from the supporting setT e . A higher inference score indicates stronger support, while lower values correspond to weaker inferability. Knowledge Unlearning. LetAbe an unlearning method. The knowledge graph induced by the LLM M A unlearn (after applyingA) isG A . A target triplee = (s,r,o)is deemed unlearned if it is robustly erased fromM A unlearn ’s accessible knowledge. Specifically, even a strong adversary should not be able to inferethrough strategic queries toM A unlearn concerning related facts in its latent knowledge graphG A , whether these queries are direct or indirect probes of the relationrbetweensando. Importantly, simply causing the model to denyeupon direct questioning is often insufficient; deeply interconnected knowledge structures from pretraining may still allow for its inference. The potency of such inference attacks is a function of the adversary’s reasoning power, captured by the intrinsic judge function f . This provides the basis for the following definition of knowledge unlearning: Definition 1 (Knowledge Unlearning). Given a target triplee = (s,r,o)for unlearning and an adversary with intrinsic judge functionf, letM A unlearn denote the model obtained after applying unlearning methodAwith induced knowledge graphG A . We sayAachievesγ-knowledge unlearning if f(G A ,e)≤ γ, where γ specifies an upper bound on the residual inferability of e fromM A unlearn . The above knowledge-unlearning definition, while theoretically sound, faces practical evaluation challenges. First, accessing the modelM A unlearn ’s full knowledge representationG A is typically infeasible. To approximate the inference scoref(G A ,e), we extract a localized supporting subgraph G A e ⊆ G A . This subgraph, centered on the target triple e and constructed using a real-world knowledge base as reference, captures relevant local supporting facts (details in Sec. 4). Our practical evaluations therefore assess approximate knowledge unlearning usingf(G A e ,e) . Traditional direct triple removal evaluations, termed instance unlearning, represent a special case whereG A e reduces to the target triple and its associated confidence score, i.e.,G A e = (e,u e ) ∈ T U . Second, the definition presumes an intrinsic judge functionfwith comprehensive knowledge of all potential factual relationships. As this ideal is generally unattainable, human expert annotators or powerful LLM-based evaluators will be used as practical proxies for f . 4 Methodology 4.1 Overview of the Evaluation Protocol To evaluate whether a target fact has been effectively unlearned, we build upon the definition of knowledge unlearning introduced in Sec. 3. We assess the inferability of a target tripleebased on its supporting subgraphG A e and a proxy judge functionf(G A e ,e) to quantify the extent to whiche remains inferable from retained knowledge. Successful unlearning requires not only the removal of the target triple but also the disruption of its underlying inferential structure. To instantiate this evaluation protocol, we adopt a two-stage approach: (1) Supporting Subgraph Extraction, where we probe the unlearned modelM A unlearn to constructG A e ; and (2) Adversarial Inference via a Powerful Judge, where the inferability score is computed using a powerful LLM/human expert adversary. 4.2 Extracting Supporting Subgraphs from Model Beliefs Given a target triplee = (s,r,o), our goal is to retrieve its supporting subgraphG A e ⊆ G A encoded in the unlearned modelM A unlearn that substantiates its inferential basis.We formalize this subgraph as a union of confidence-aware triples extracted fromM A unlearn . Filtered Edge Retain Edge Figure 2: Illustration of supporting subgraph. These triples collectively trace potential deductive pathways from the sub- jectsto the objectowithin a bounded path lengthℓ. The constraint of using a bounded path length for these deductive pathways aligns with observa- tions that many reasoning tasks can be effectively resolved within a small number of hops [39]. Constructing such pathways requires probingM A unlearn with candidate triples. A key challenge arises asM A unlearn offers no explicit catalogue of entities or relations, and an exhaustive enumeration of all possibilities is computationally prohibitive. Therefore, to define a tractable search space, we determine candidate entities and relations by referring to an external real-world reference knowledge graphG ref (e.g., Wikidata). This reliance on 4 an external, publicly accessible corpus is a practical necessity. Indeed, even adversaries attempting to indirectly probe an LLM’s knowledge typically need to anchor their queries in commonly understood entities, relations, or schemata. While ourG ref -guided approach is not perfect and cannot capture all latent knowledge, it provides a principled method for exploring deductive pathways grounded in relevant, publicly accessible information. To be more robust against the potential missing information inG ref , our strategy for generating knowledge fact candidates from a frontier nodev(initiallys, then other entities along a path) involves retrieving itsk-hop neighborhood inG ref . The union of these neighboring entities serves as the set of candidate objects. Candidate relations are similarly drawn from the available relation types in G ref . This approach assumes that the local neighborhood in the reference KG offers a reasonably comprehensive, albeit imperfect, approximation of the entities relevant for deductive reasoning that are implicitly known toM A unlearn . Ultimately, the resulting supporting subgraph is intended to capture the latent reasoning chains through which the target triple e may be logically derived. Constructing the Support Subgraph. We buildG e iteratively under a breadth-limited expansion strategy (illustrated in Fig. 2). Starting from the subjects, we probe the unlearned modelM A unlearn with candidate triples of the form(s,r ′ ,o ′ ). Whenever the model’s confidenceufor a triple exceeds a preset thresholdu ∗ , the triple is added to the subgraph and its objecto ′ becomes the new frontier. Here, we useu≤ u ∗ to denote the excess case. In the next hop, starting from the new frontierso ′ , we query triples(o ′ ,r ′ ,o ′ )whereo ′ s come from all candidate entities. After each round, we record the model’s confidence scores and discard triples that are either rejected or judged uncertain. The expansion stops when the path-length limit ℓ is reached or no further confident triples can be found. Confidence Evaluation Multiple-Choice Query Entropy Calculation Admission Criteria Supporting Subgraph Construction All Possible Triples Reference Knowledge Graph k-hop Neighbors Extraction (Eg. k=2) relation setneighbors set: Combine Search in Ref KG 2-hop Neighbors Filtered edge Retain edge Figure 3: The entire knowledge probing process. To properly estimate the model’s confidence in a candidate triple(s ′ ,r ′ ,o ′ ), we adopt a multiple- choice querying protocol, where each triple is ver- balized using a fixed natural-language template and presented toM A unlearn with answer choices Yes, No, Unknown. While an alternative might involve havingM A unlearn directly output a numer- ical confidence score, our empirical investiga- tions revealed this method to be inaccurate, which is consistent with existing studies on the chal- lenges of calibrating LLM confidence [40]. Con- sequently, we find that more reliable confidence assessments are obtained by applying temperature scaling [41] to adjust the model’s softmax output probabilities corresponding to these answer choices. This calibration process yields significantly more accurate confidence scores, as demonstrated by our experiments in Sec. 6.3. After calibration, we model the confidence spaceUin terms of entropy: H(s ′ ,r ′ ,o ′ ) = − P i∈Yes,No,Unknown P i log 2 P i , whereP Yes , P No , P Unknown are the calibrated proba- bilities for the respective answer choices. A triple is admitted toG A e iff the model selectsYesand H(s ′ ,r ′ ,o ′ ) ≤ u ∗ . Recall thatu ∗ is a predefined entropy threshold. In this paper, we setu ∗ = 1, which corresponds to a worst-case scenario where the model follows instructions but assigns equal probability (0.5) toYesand one other option (NoorUnknown). The full mapping betweenu ∗ and minimumYesprobabilities is provided in App. G. This criterion filters out unconfident evidence, preserving only high-confidence relational structure. Overall, the entire knowledge probing process as above is illustrated in Fig. 3. The corresponding algorithm is provided in App. A, Alg. 1. 4.3 Adversarial Inference Assessment via Powerful LLM Judge Given a target tripleeand its associated supporting subgraphG A e , the evaluation of unlearning effectiveness relies on an adversarial inference scheme to assess the residual inferability ofethrough systematic exploration ofG A e . While human experts provide a strong reference for reasoning-based evaluation, they are not scalable for large-scale assessment. To automate the process, we employ a powerful reasoning LLM as the judge. We monitor the alignment between the LLM judge and human expert evaluations on a subset of examples, with results reported in Sec. 6.3. Effective reasoning by the LLM judge hinges on a carefully engineered prompt structure. This structure is built upon two foundational considerations: first, it mandates that the judge’s rea- soning be based exclusively on the provided subgraphG A e , encompassing its factual content and associated confidence scores without resorting to any external knowledge. Second, we re- 5 quire the judge to quantify inferability using a discrete 0-5 rating scale, where 0 signifies that the target tripleecannot be inferred and 5 denotes very high certainty in deducingefromG A e . Task Description: Given subgraph triples, determine whether the target triple can be inferred... Entropy Interpretation: 0.00–0.25 → Very Confident 0.25–0.50 → Confident 0.50–0.75 → Less Confident 0.75–1.00 → Not Confident 0 → No logical path exists... 1 → Very weak support... 2 → Some weak logical connection... 3 → Moderate path exists... 4 → Confident support... 5 → Strong support... Scoring Rules: Evaluation Guidelines: 1. Analyze direct paths.. 2. Analyze support paths... 3. Assess the uncertainty of each path... 4. Give the score... Input: Target T riples: Subgraph Triples: Several Examples: Example for Score 5: (Rome, capital_of, Italy), (Italy, located_in, Europe) form support path... Both triples have very low entropy... located_in can be logically implied by capital_of and located_in... (Rome, located_in, Europe) (Rome, capital_of, Italy) entropy 0.08 (Italy, located_in, Europe) entropy 0.12 Example for Score 0: Example for Score 4: ... Support path very strong → Final Score: 5 Figure 4: Illustration of In- structions for LLM Judge. To further ensure the reliability and accuracy of these judgments, the prompt design inherently integrates clear evaluation objectives, explicit semantic definitions for all rating levels, entropy-based guid- ance that categorizes triple certainty into discrete confidence bins (as illustrated in Fig. 4. Task-specific instructions and in-context examples are also embedded to ensure consistent and accurate eval- uation behavior. For the full prompt template with the interpretation of each rating score, see App. C.1. Importantly, the LLM’s judgments are expected to rely solely on G A e , treating it as ground truth, without leveraging any of its internal knowledge acquired during pretraining. Our experiments will verify this assumption by analyzing the model’s reasoning traces. 5 Knowledge Unlearning Evaluation Datasets Systematic evaluation of the unlearning protocol described herein necessitates a dataset where target triples, presumed to be retained by the LLM, are accompanied by a rich network of correlated and semantically coherent triples capable of supporting target triple inference. To create such a dataset for realistic scenarios, we utilize real-world knowledge bases, specifically selecting YAGO3-10 [42]. YAGO3-10 is a large- scale knowledge graph built from Wikipedia, Wikidata, and WordNet [43], containing over one million relational triples, 123,182 entities, and 37 distinct relation types. Crucially, its Wikipedia and Wikidata foundations are pertinent as these sources are common in the pretraining corpora of modern LLMs (e.g., [16, 44, 45]), making it likely that YAGO3-10’s factual triples are retained by these models. Moreover, the relations defined in YAGO3-10 exhibit various inferential structures (more illustrations in H). For our experiments, we target specific YAGO3-10 triples for unlearning, using the full YAGO3-10 graph as the referenceG ref for support subgraph extraction. Target Triples and Supporting Subgraphs Extraction. From each LLM for evaluation, we extract 200 unlearning target facts. To ensure these targets constitute factual knowledge genuinely retained by the model, they are identified using a carefully selected knowledge-probing template (to be introduced later) coupled with an entropy-based confidence filter; the specifics of how to measure confidence are detailed in Sec. 4.2. Supporting subgraphs for each unlearning target are constructed by querying all relations inG ref and retrieving candidate knowledge facts within a 3-hop neighborhood (k = 3) at each expansion step. The resulting subgraphs are constrained to a maximum path length ofl = 3, which reflects a reasonable inference depth in line with prior work [19, 39]. Knowledge Probing Method. Effective knowledge probing of LLMs is crucial for both our dataset construction and subsequent evaluation phases. Following the approach in [27], we utilized GPT-4 to generate a diverse pool of candidate prompt templates. These templates were specifically designed for our multiple-choice query format in Sec. 4.2. To evaluate these candidate templates, we constructed a comprehensive validation set. For each of the 37 relations in YAGO3-10, we selected 10 positive examples, consisting of factual triples that hold the given relation within YAGO3-10. Concurrently, for each relation, we generated 10 negative examples composed of counterfactual entity pairs that do not hold that relation. These negative examples underwent manual verification by human annotators to confirm their factual incorrectness, despite being constructed to appear plausible (i.e., the entity types were compatible with the relation). This process yielded a validation set totaling 370 positive and 370 negative samples. Each candidate prompt template was then systematically evaluated based on its accuracy over this validation set. The template that achieved the highest accuracy was selected as our definitive query prompt for all LLM interactions (see App. C.2 for the complete template). 6 Experiments 6.1 Experimental Setup Unlearning Methods. For each target LLM, we evaluate the following popular unlearning approaches in the literature: Gradient Ascent (GA) [46–48] aims to erase the influence of unlearn triples by 6 reversing gradient updates. Random Labels (RL) [46, 49] disrupts memorization by replacing correct labels with random tokens during training. Negative Preference Optimization (NPO) [50] formulates unlearning as reversing the model’s preference toward unlearned data by optimizing likelihood ratios against the pretrained model. NegGrad+[51] combines gradient ascent on the unlearn set with gradient descent on a non-unlearn set to retain performance. Finally, SCRUB [51] adopts a student–teacher framework, using KL divergence to balance knowledge removal from the unlearned data with the retention of non-target data. To ensure fair comparison, we adopt a unified computational budget protocol for all methods, following [14]. Formal definitions, implementation details, and per-method budget allocations are provided in App. B. Unlearning Effectiveness Metrics. We propose the Unlearning Effectiveness Score (UES) to quantify how effectively an unlearning methodAreduces target knowledge inferability by comparing pretrained (M pretrain ) and unlearned (M A unlearn ) models. Given unlearning targetsD forget , for each e ∈ D forget , letG e andG A e be its supporting subgraphs fromM pretrain andM A unlearn respectively. UES is the average normalized relative decrease in an LLM-assessed inference scoreUES = E e∈D forget h f judge (G e ,e)−f judge (G A e ,e) f judge (G e ,e) i . Here,f judge outputs a discrete score from0, 1,..., 5(detailed in Sec. 4.3). UES is upper-bounded by 1; higher values denote more effective unlearning (greater inferability reduction). UES= 0implies no effect, and negative values indicate increased inferability after unlearning. The metric is well-defined asf judge (G e ,e)is strictly positive for verified retained targets (Sec. 5). For instance-level unlearning ("Inst."), targeting direct triple knowledge, the metric is computed by replacing the subgraphG e (orG A e ) withepaired with its confidenceu e before unlearning (or after unlearning, resp.). Additionally, we employ Recall [28]E e∈D forget h |G e ∩G A e | |G e | i . A metric that measures the proportion of original supporting subgraphG e ’s triples preserved in the unlearned model’s subgraph (G A e ). Since these subgraphs reflect inferential support fore, a lower Recall value indicates that more of this supporting structure has been successfully unlearned. Utility Preservation. Model utility is assessed by the retention of local knowledge near unlearned triples and the preservation of general model capabilities [27]. Local Consistency (Loc). To assess local utility preservation, we sample triples within a 3-hop neighborhood of the target triple’s head or tail entities from the reference knowledge graphG ref , excluding triples already included in the supporting subgraphG e . These neighboring triples are considered irrelevant to the target inference and should remain unaffected by unlearning. The Loc score measures the consistency of model predictions (Yes,No, orUnknown) on these neighbor triples before and after unlearning. A higher Loc score indicates better preservation of local factual integrity. Importantly, Loc captures multi- directional knowledge shifts, such as facts changing fromYestoNoor vice versa. Further analysis is provided in App. F. General Model Utility. General capabilities are assessed from two perspectives: General Knowledge (Gen), measured on MMLU [52] (a multi-domain multiple-choice benchmark) via 5-shot perplexity-based answer ranking and Reasoning Ability (Rea), measured on BBH [53] (a suite of 27 challenging tasks) by 3-shot exact match accuracy with chain-of-thought prompting. General Settings. We conduct our unlearning experiments using two widely adopted open-source LLMs: LLaMA3-8B-Instruct [54] and Qwen2.5-7B-Instruct [55]. The unlearning procedures are implemented via either full model fine-tuning or parameter-efficient tuning using LoRA [56]. The reported results are averaged over the total of 200 target triples. For the utility evaluation on Loc, we sample a set of 2,000 neighboring triples (10×the size of the unlearning target set), reflecting realistic unlearning-to-retention ratios as discussed in prior literature [57]. During unlearning, we consider two formats for presenting unlearning targets: (i) converting each triple into a natural sentence (unlearn with sentence), and (i) framing it as a multiple-choice question with options(Yes, No, Unknown) (unlearn with QA). To ensure a fair comparison across methods, we fix the computational budget: 10 unlearning epochs for GA, RL, and NPO, and 5 epochs for NegGrad+and SCRUB, which require additional knowledge triples to preserve utility (detailed in App. B). For epoch selection, we follow a similar protocol introduced in [13, 14, 28]: if the model’s utility (Loc) exceeds 0.8 at any epoch, we select the last epoch satisfying this criterion; otherwise, we select the epoch with the highest utility below the threshold. During evaluation, we use GPT-o4-mini as the adversarial judge. Full implementation details, including hyperparameter configurations, are provided in App. D. 7 Method LLaMA-8B-InstructQwen2.5-7B-Instruct Unlearning EffectivenessUtility RetentionUnlearning EffectivenessUtility Retention UES (Inst.) (↑)UES (Ours) (↑)Recall (↓)Loc (↑)Gen (↑)Rea (↑)UES (Inst.) (↑)UES (Ours) (↑)Recall (↓)Loc (↑)Gen (↑)Rea (↑) Unlearn with Sentence Templates No Unlearn–0.6330.567–0.6370.581 GA (Full)0.1570.076 (51.59%↓)0.2570.9510.6300.568-0.020-0.093 (365%↓)0.9770.9640.6320.581 RL (Full)0.0270.007 (74.07%↓)0.7790.9940.6310.5650.1760.077 (56.25%↓)0.8830.9750.6330.576 NPO (Full)0.0640.046 (28.12%↓)0.8900.9560.6290.5660.1400.058 (58.57%↓)0.8800.8820.6360.580 NegGrad+ (Full)0.0060.001 (83.33%↓)0.6490.9770.6290.5640.7020.534 (23.94%↓)0.4890.8480.6300.578 SCRUB (Full)0.037-0.003 (108.10%↓)0.9190.9570.6280.5690.6090.457 (25.12%↓)0.4020.7390.6270.583 GA (LoRA)0.1070.059 (44.85%↓)0.2310.9600.6300.5650.1220.022 (81.97%↓)0.8020.8270.6360.577 RL (LoRA)0.0270.017 (37.03%↓)0.6180.9970.6330.5630.026-0.052 (300.00%↓)0.9280.9340.6340.580 NPO (LoRA)0.1810.010 (94.48%↓)0.5750.9890.6320.5660.3890.244 (37.25%↓)0.6510.9430.6380.576 NegGrad+ (LoRA)0.1540.030 (80.51%↓)0.4720.9970.6350.5630.1950.099 (49.23%↓)0.8400.9670.6400.581 SCRUB (LoRA)0.2110.033 (84.36%↓)0.3290.9970.6310.5650.7150.516 (27.83%↓)0.4120.7460.6190.573 Unlearn with QA Templates No Unlearn–0.6330.567–0.6370.581 GA (Full)0.7960.622 (21.86%↓)0.2700.2620.6330.5650.8900.816 (8.31%↓)0.1640.7330.6380.580 RL (Full)0.1340.060 (55.22%↓)0.4860.9920.6330.5680.051-0.012 (123.53%↓)0.9320.8820.6400.583 NPO (Full)0.2320.102 (56.03%↓)0.4570.9570.6340.5670.7940.705 (11.19%↓)0.2110.6830.6380.581 NegGrad+ (Full)0.9560.904 (5.44%↓)0.0090.1750.6270.5530.6660.614 (7.81%↓)0.1390.5840.6310.576 SCRUB (Full)0.9590.658 (31.37%↓)0.2360.1450.6380.5720.9630.941 (2.29%↓)0.0430.6070.6100.580 GA (LoRA)0.1450.096 (33.79%↓)0.3050.9960.6340.5660.4740.289 (39.03%↓)0.6300.9290.6400.580 RL (LoRA) 0.0910.059 (35.16%↓)0.3840.9680.6330.5660.016-0.025 (256.25%↓)0.9710.9610.6390.581 NPO (LoRA)0.6900.585 (15.22%↓)0.0630.9360.6330.5650.0550.004 (92.7%↓)0.9650.9290.6410.578 NegGrad+ (LoRA)0.4330.324 (25.17%↓)0.6640.9930.6310.5640.0980.054 (44.90%↓)0.8530.9430.6470.581 SCRUB (LoRA)0.1080.032 (70.37%↓)0.4810.8780.6370.5690.9140.777 (14.99%↓)0.2860.7170.5940.574 Table 1: Comparison of various unlearning methods on LLaMA-8B-Instruct and Qwen2.5-7B-Instruct in terms of unlearning effectiveness and utility retention with different unlearning templates. 6.2 Main Results Correlated Knowledge Supports Inference and Reduces Unlearning Effectiveness. We report the unlearning effectiveness and utility metrics of each unlearning method under both instance unlearning (Inst.) and our correlated knowledge tracing framework in Tab. 1. When comparing each method’s UES under instance unlearning versus our supporting subgraph setting, we observe a substantial drop in effectiveness. For most cases, the UES decreases by at least 20%, with more than half of the settings exhibiting a reduction exceeding 30%. We further analyze specific unlearned triples and observe that certain triples appear successfully unlearned at the instance level, yet remain inferable when considering their supporting subgraphs. For instance, the triple (B+H_Architects, created, Brookfield_Place_(Toronto)) is effectively unlearned (entropy: 0.252 before unlearning, 1.127 after), but the evaluator LLM can still infer it through related facts such as (B+H_Architects, isKnownFor, Brookfield_Office_Properties) (entropy: 0.134 before, 0.512 after) and (Brookfield_Office_Properties, owns, Brookfield_Place_(Toronto)) (entropy: 0.110 before, 0.441 after) present in the supporting subgraph (see additional examples in App. E.3). Meanwhile, we observe that when local utility is well preserved (e.g.,Loc≥ 0.8), meaning over 80% of entity-centric triples in the neighborhood are retained, the structure of the supporting subgraphG A e , as reflected by the recall score, also remains largely intact. This preserved structure enables more potential inferences, thereby reducing unlearning effectiveness. In contrast, when utility degrades sharply, much of the supporting context is disrupted, limiting inferential paths (e.g., QA-based full model unlearning for Qwen2.5 in Tab. 1). We further compare the overall performance of unlearning methods in Tab. 1. No method consistently achieves a favorable trade-off between unlearning effectiveness (UES) and utility (Loc). When Loc≥ 0.8, GA, NPO, NegGrad+, and SCRUB generally outperform RL across both LLMs. These four methods show comparable performance, each achieving the best results in different settings. 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB Figure 5: Effectiveness of Entropy Thresholdu ∗ (Confi- dence) on UES. Sentence-based unlearning provides a more utility-preserving alternative to QA-based unlearning. As shown in Tab. 1, under knowledge probing via QAs, employing traditional sentence tem- plates under identical unlearning configurations results in superior utility retention (measured by Loc) and less aggressive average un- learning effectiveness compared to the QA-based approach. This advantage likely arises because sentence-based contexts elicit more conservative parameter updates, whereas QA-based unlearning ex- plicitly targets specific model outputs (e.g.,Yes), causing abrupt behavioral shifts and potential instruction-following degradation. How does the confidence score affect unlearning effectiveness? In our framework, we model the target LLM’s confidence in knowledge triples. Here, we further investigate how these confidence scores influence unlearning outcomes by 8 varying the thresholdu ∗ used in generating supporting subgraphs. As shown in Fig. 5, we report results under the LLaMA3-8B-Instruct LoRA QA setting. We observe that decreasing the entropy thresholdu ∗ , i.e., requiring higher LLM confidence, filters out more supporting inferences, thereby increasing unlearning effectiveness. This highlights that it is crucial to model the confidence of LLMs on the knowledge facts, as low-confidence knowledge can still contribute meaningfully to inference and introduce information leakage. Refer to App. F for more results. 6.3 Justification on the Reliability of the Target LLM and LLM Judge In this subsection, we report results validating the reliability of both the target LLM and the LLM- based judge. (1) Target LLM Calibration. To interpret entropy calculation, we first calibrate the token probabilities (i.e., confidence) output by the target LLM. As mentioned in Sec. 4.2, we calibrate the optimal temperature that aligns predicted probabilities with actual model behavior. Using the validation set detailed in Sec. 5) with positive and negative knowledge triples, we prompt the LLM with multiple-choice questions and record the model’s predicted answer based on the argmax token. We then compute the Expected Calibration Error (ECE) [41] separately on “Yes” and “No” predictions, measuring how well model confidence aligns with accuracy (lower is better). As shown in Fig. 6, the temperature-scaled output distributions of both models exhibit strong alignment between the predicted probabilities and accuracy, indicating reliable confidence estimates. (2) LLM Judge Reliability. To ensure that the LLM judge (GPT-o4-mini) provides trustworthy inference assessments, we validate its behavior from two complementary aspects. First, we qualitatively verify that the judge adheres strictly to the evaluation instructions by reasoning solely over the provided supporting subgraph without incorporating external knowledge. To this end, we randomly sample 50 target triples that are known to be memorized by the target LLM and construct their corresponding supporting subgraphs. We then manually inspect the LLM judge’s responses on these subgraphs and find that in all cases, the model faithfully follows the instruction without relying on external information. Examples are illustrated in App. E. Second, we quantitatively evaluate the consistency of its assessments by comparing its scores against human judgments. Using the same 50 target triples and their supporting subgraphs, we apply randomized masking to vary inferential strength, and collect three scoring rounds from the LLM judge alongside three independent ratings from PhD students under the same prompt guidance. Fig. 7 presents a scatter plot comparing the average scores from the LLM and human annotators, revealing a strong correlation that supports the reliability and consistency of the LLM judge. Further investigating cases with significant rating discrepancies between LLMs and humans reveals two key observations. Cases where LLM ratings exceed human ratings often involve large supporting subgraphs (50–100 triples) with multiple relations between the same entities, making it challenging for humans, even with visualization, to identify all potential inference paths. Conversely, instances rated highly by humans but low by LLMs typically occur when the model overlooks certain multi-hop inference steps. These observations highlight the inherent complexity of knowledge inference tasks, even for human evaluators. Admittedly, further improving LLM-evaluator reliability remains an open avenue. More concrete examples can be seen in App. E. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy ECE = 0.031 LLaMA3 Reliability Diagram Accuracy Overconfident gap Underconfident gap 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy ECE = 0.067 Qwen2.5 Reliability Diagram Accuracy Overconfident gap Underconfident gap Figure 6: Target LLM Calibration 012345 Avg LLM Judge Score 0 1 2 3 4 5 Avg Human Judge Score Human vs LLM Pearson Coefficient: 0.826 Supporting Subgraphs: 50 Figure 7: Judge Comparison 6.4 Ablation Studies Size of Forget Set. In Fig. 8, we examine how varying the forget set size affects the unlearning–utility trade-off across methods. Under the LLaMA3-8B-Instruct LoRA QA setting, as the forget set size increases from 50 to 100, we observe that when utility remains relatively stable, the gap between instance unlearning and our supporting subgraph-based unlearning effectiveness remains large. How- ever, with larger forget sets, utility begins to degrade more significantly, and the supporting subgraph structure is increasingly undermined, leading to a smaller gap between unlearning effectiveness. 9 Unlearning Iterations. We study the impact of unlearning epochs across different methods. Since NegGrad+ and SCRUB require retaining triples to preserve performance and thus incur double the computation, we limit them to 5 epochs, while other methods run for 10 epochs. In Fig. 9, we report results for LLaMA3-8B-Instruct (LoRA) using sentence templates, where model utility remains largely preserved. As the number of epochs increases, we observe a growing gap in unlearning effectiveness between instance unlearning and our supporting subgraph-based evaluation. Figure 8: Impact of the forget-set size on UES and utility. Figure 9: Effect of unlearning epochs. Left: Unlearning effectiveness. Right: Model utility. 7 Conclusion This work introduces a framework for evaluating knowledge unlearning in large language models, distinguished by its explicit modeling of the complex, correlated, and confidence-aware structure of factual knowledge. Our approach incorporates an inference-based evaluation protocol that utilizes powerful, meticulously guided LLM judges to assess unlearning effectiveness. Extensive experiments, involving the extraction of real-world correlated knowledge from target LLMs, reveal that current unlearning methods often significantly overestimate their actual unlearning effectiveness. This critical finding underscores the imperative to account for knowledge correlations and confidence when evaluating and developing future unlearning techniques. Acknowledgement R. Wei, H. Yin, E. Chien, and P. Li are partially supported by the NSF under awards PHY-2117997, IIS-2239565, IIS-2428777, and CCF-2402816; the DOE under award DE-FOA-0002785; the JP- Morgan Chase Faculty Award; and the OpenAI Researcher Access Program Credit. P. Niu and O. Milenkovic gratefully acknowledge support from NSF award CCF-2402815. R. Wu and K. Chaudhuri acknowledge research support from NSF awards CNS-2241100 and CIF-2402817. The authors would also like to thank Siqi Miao and Evelyn Ma for the valuable discussion. Disclaimer This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates (“JP Morgan”) and is not a product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. References [1] Y. Nie, Y. Kong, X. Dong, J. M. Mulvey, H. V. Poor, Q. Wen, and S. Zohren, “A survey of large language models for financial applications: Progress, prospects and challenges,” arXiv preprint arXiv:2406.11903, 2024. 10 [2]J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N. Eckardt, N. G. Laleh, C. M. L. Löffler, S.-C. Schwarzkopf, M. Unger, G. P. Veldhuizen, et al., “The future landscape of large language models in medicine,” Communications medicine, vol. 3, no. 1, p. 141, 2023. [3]H. Li, G. Deng, Y. Liu, K. Wang, Y. Li, T. Zhang, Y. Liu, G. Xu, G. Xu, and H. Wang, “Digger: Detecting copyright content mis-usage in large language model training,” arXiv preprint arXiv:2401.00676, 2024. [4]K. K. Chang, M. Cramer, S. Soni, and D. Bamman, “Speak, memory: An archaeology of books known to chatgpt/gpt-4,” arXiv preprint arXiv:2305.00118, 2023. [5]X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi, “Quark: Controllable text generation with reinforced unlearning,” Advances in neural information processing systems, vol. 35, p. 27591–27609, 2022. [6] M. Zamini, H. Reza, and M. Rabiei, “A review of knowledge graph completion,” Information, vol. 13, no. 8, p. 396, 2022. [7] J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, “Knowledge unlearning for mitigating privacy risks in language models,” arXiv preprint arXiv:2210.01504, 2022. [8]S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao, C. Y. Liu, X. Xu, H. Li, et al., “Rethinking machine unlearning for large language models,” arXiv preprint arXiv:2402.08787, 2024. [9]V. B. Kumar, R. Gangadharaiah, and D. Roth, “Privacy adhering machine un-learning in nlp,” arXiv preprint arXiv:2212.09573, 2022. [10]J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for llms,” arXiv preprint arXiv:2310.20150, 2023. [11]P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter, “Tofu: A task of fictitious unlearning for llms,” arXiv preprint arXiv:2401.06121, 2024. [12] J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue, “Machine unlearning of pre-trained large language models,” arXiv preprint arXiv:2402.15159, 2024. [13]W. Shi, J. Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang, “Muse: Machine unlearning six-way evaluation for language models,” arXiv preprint arXiv:2407.06460, 2024. [14] R. Wei, M. Li, M. Ghassemi, E. Krea ˇ ci ́ c, Y. Li, X. Yue, B. Li, V. K. Potluru, P. Li, and E. Chien, “Underestimated privacy risks for minority populations in large language model unlearning,” arXiv preprint arXiv:2412.08559, 2024. [15]W. Ma, X. Feng, W. Zhong, L. Huang, Y. Ye, X. Feng, and B. Qin, “Unveiling entity- level unlearning for large language models: A comprehensive analysis,” arXiv preprint arXiv:2406.15796, 2024. [16]H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [17] F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?,” arXiv preprint arXiv:1909.01066, 2019. [18]R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021. [19] Z. Zhong, Z. Wu, C. D. Manning, C. Potts, and D. Chen, “Mquake: Assessing knowledge editing in language models via multi-hop questions,” arXiv preprint arXiv:2305.14795, 2023. [20] Y. Shi, Q. Tan, X. Wu, S. Zhong, K. Zhou, and N. Liu, “Retrieval-enhanced knowledge editing in language models for multi-hop question answering,” in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, p. 2056–2066, 2024. [21]A. Blanco-Justicia, N. Jebreel, B. Manzanares-Salor, D. Sánchez, J. Domingo-Ferrer, G. Collell, and K. Eeik Tan, “Digital forgetting in large language models: A survey of unlearning methods,” Artificial Intelligence Review, vol. 58, no. 3, p. 90, 2025. 11 [22]Q. Wang, B. Han, P. Yang, J. Zhu, T. Liu, and M. Sugiyama, “Towards effective evaluations and comparisons for llm unlearning methods,” in The Thirteenth International Conference on Learning Representations. [23] J. Geng, Q. Li, H. Woisetschlaeger, Z. Chen, Y. Wang, P. Nakov, H.-A. Jacobsen, and F. Karray, “A comprehensive survey of machine unlearning techniques for large language models,” arXiv preprint arXiv:2503.01854, 2025. [24] R. Eldan and M. Russinovich, “Who’s harry potter? approximate unlearning for llms,” 2023. [25]B. Wei, W. Shi, Y. Huang, N. A. Smith, C. Zhang, L. Zettlemoyer, K. Li, and P. Henderson, “Eval- uating copyright takedown methods for language models,” arXiv preprint arXiv:2406.18664, 2024. [26]N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, et al., “The wmdp benchmark: Measuring and reducing malicious use with unlearning,” arXiv preprint arXiv:2403.03218, 2024. [27] Z. Jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y. Chen, K. Liu, and J. Zhao, “Rwku: Benchmarking real-world knowledge unlearning for large language models,” arXiv preprint arXiv:2406.10890, 2024. [28]R. Wu, C. Yadav, R. Salakhutdinov, and K. Chaudhuri, “Evaluating deep unlearning in large language models,” arXiv preprint arXiv:2410.15153, 2024. [29]S. Wang, Y. Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li, “Knowledge editing for large language models: A survey,” ACM Computing Surveys, vol. 57, no. 3, p. 1–37, 2024. [30]K. Meng, D. Bau, A. Andonian, and Y. Belinkov, “Locating and editing factual associations in gpt,” Advances in neural information processing systems, vol. 35, p. 17359–17372, 2022. [31]K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau, “Mass-editing memory in a transformer,” arXiv preprint arXiv:2210.07229, 2022. [32] D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei, “Knowledge neurons in pretrained transformers,” arXiv preprint arXiv:2104.08696, 2021. [33] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning, “Fast model editing at scale,” in International Conference on Learning Representations. [34]N. De Cao, W. Aziz, and I. Titov, “Editing factual knowledge in language models,” arXiv preprint arXiv:2104.08164, 2021. [35]C. Zheng, L. Li, Q. Dong, Y. Fan, Z. Wu, J. Xu, and B. Chang, “Can we edit factual knowledge by in-context learning?,” arXiv preprint arXiv:2305.12740, 2023. [36]Q. Dong, D. Dai, Y. Song, J. Xu, Z. Sui, and L. Li, “Calibrating factual knowledge in pretrained language models,” arXiv preprint arXiv:2210.03329, 2022. [37]Z. Huang, Y. Shen, X. Zhang, J. Zhou, W. Rong, and Z. Xiong, “Transformer-patcher: One mistake worth one neuron,” arXiv preprint arXiv:2301.09785, 2023. [38]X. Li, S. Li, S. Song, H. Liu, B. Ji, X. Wang, J. Ma, J. Yu, X. Liu, J. Wang, et al., “Swea: Updating factual knowledge in large language models via subject word embedding altering,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, p. 24494–24502, 2025. [39]J. Su, J. Healey, P. Nakov, and C. Cardie, “Between underthinking and overthinking: An empirical study of reasoning length and correctness in llms,” arXiv preprint arXiv:2505.00127, 2025. [40] J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych, “A survey of confidence estimation and calibration in large language models,” arXiv preprint arXiv:2311.08298, 2023. [41] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International conference on machine learning, p. 1321–1330, PMLR, 2017. [42]T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel, “Convolutional 2d knowledge graph embeddings,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018. [43] F. Mahdisoltani, J. Biega, and F. M. Suchanek, “Yago3: A knowledge base from multilingual wikipedias,” in CIDR, 2013. 12 [44]H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. [45] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023. [46] A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9304–9312, 2020. [47] L. Graves, V. Nagisetty, and V. Ganesh, “Amnesiac machine learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, p. 11516–11524, 2021. [48] J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, “Knowledge un- learning for mitigating privacy risks in language models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (A. Rogers, J. Boyd-Graber, and N. Okazaki, eds.), (Toronto, Canada), p. 14389–14408, Association for Computational Linguistics, July 2023. [49] J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue, “Machine unlearning of pre- trained large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), p. 8403–8419, Association for Computational Linguistics, Aug. 2024. [50]R. Zhang, L. Lin, Y. Bai, and S. Mei, “Negative preference optimization: From catastrophic collapse to effective unlearning,” arXiv preprint arXiv:2404.05868, 2024. [51]M. Kurmanji, P. Triantafillou, J. Hayes, and E. Triantafillou, “Towards unbounded machine unlearning,” Advances in neural information processing systems, vol. 36, 2024. [52]D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020. [53]M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al., “Challenging big-bench tasks and whether chain-of-thought can solve them,” arXiv preprint arXiv:2210.09261, 2022. [54] AI@Meta, “Llama 3 model card,” 2024. [55]A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan, “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024. [56] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al., “Lora: Low-rank adaptation of large language models.,” ICLR, vol. 1, no. 2, p. 3, 2022. [57]M. Pawelczyk, J. Z. Di, Y. Lu, G. Kamath, A. Sekhari, and S. Neel, “Machine unlearning fails to remove data poisoning attacks,” arXiv preprint arXiv:2406.17216, 2024. [58]G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” in The Eleventh International Conference on Learning Represen- tations, 2023. [59]C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, “Certified data removal from machine learning models,” in Proceedings of the 37th International Conference on Machine Learning (H. D. I and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research, p. 3832–3842, PMLR, 13–18 Jul 2020. [60]E. Chien, H. Wang, Z. Chen, and P. Li, “Langevin unlearning: A new perspective of noisy gradient descent for machine unlearning,” arXiv preprint arXiv:2401.10371, 2024. [61]Z. Liu, G. Dou, E. Chien, C. Zhang, Y. Tian, and Z. Zhu, “Breaking the trilemma of privacy, utility, and efficiency via controllable machine unlearning,” in Proceedings of the ACM on Web Conference 2024, p. 1260–1271, 2024. 13 [62]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [63]H. H.-H. Hsu, Y. Shen, C. Tomani, and D. Cremers, “What makes graph neural networks miscalibrated?,” Advances in Neural Information Processing Systems, vol. 35, p. 13775–13786, 2022. [64] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning, “Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,” arXiv preprint arXiv:2305.14975, 2023. [65]J. Xie, A. S. Chen, Y. Lee, E. Mitchell, and C. Finn, “Calibrating language models with adaptive temperature scaling,” arXiv preprint arXiv:2409.19817, 2024. 14 Appendix Contents A Supporting Subgraph Extraction Algorithm16 B Details on Unlearning Methods16 C Prompt Template18 C.1 Prompt Template for LLM Judge Rating . . . . . . . . . . . . . . . . . . . . . . .18 C.2 Prompt Template for Query Target LLM . . . . . . . . . . . . . . . . . . . . . . .21 D Detailed Infomation of Experiments22 D.1 Compute Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 D.2 Experimental Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 E Additional Examples for LLM Judge23 E.1 Examples of LLM Judge Adhering to Instructions . . . . . . . . . . . . . . . . . .23 E.2 Examples Comparing LLM and Human Judgments . . . . . . . . . . . . . . . . .24 E.3 Case Studies of Supporting Subgraph Inferability . . . . . . . . . . . . . . . . . .25 F Additional Experimental Results26 F.1In-Depth Analysis of Local Consistency . . . . . . . . . . . . . . . . . . . . . . .26 F.2Evaluating Shifts in LLM-Inferred Scores Pre- and Post-Unlearning . . . . . . . .28 F.3Additional Results on the Impact of Confidence Scores on Unlearning Effectiveness29 F.4Additional Results on Performance Across Unlearning Epochs . . . . . . . . . . .30 G Mapping Between Entropy Threshold u ∗ and Yes Token Probability30 H Additional Details and Illustrations for YAGO3-1031 ILimitations and Broader impacts32 15 A Supporting Subgraph Extraction Algorithm In this section, we detail the algorithm used to construct the supporting subgraph described in Sec. 4.2. As noted in Sec. 5, we constrain the subgraph to a maximum path lengthl = 3. The full extraction procedure is provided in Alg. 1. Algorithm 1 Subgraph Extraction Input: Reference Knowledge graphG ref , target triple e = (s,r,o), hop k, entropy threshold u ∗ Output: Subgraph G e containing valid paths from s to o Process: Initialize ˆ G e =∅,N ∗ s =∅,N ∗ e =∅,N ∗ o =∅ Phase 1: Find high-confidence neighbors of s for v ∈ k-hop neighbors of s inG ref \s,o do for(s,r ′ ,v,u)∈LLM-query(s,v)whereu≤ u ∗ andyes=arg maxyes,no,unknown do N ∗ s =N ∗ s ∪v, ˆ G e = ˆ G e ∪(s,r ′ ,v,u) end for end for Phase 2: Expand from high-confidence neighbors for v ∈N ∗ s do for w ∈ k-hop neighbors of v\s,o,v∪N ∗ s do for(v,r ′ ,w,u)∈LLM-query(v,w)whereu≤ u ∗ andyes=arg maxyes,no,unknown do N ∗ e =N ∗ e ∪v, ˆ G e = ˆ G e ∪(v,r ′ ,w,u) end for end for end for Phase 3: Verify connections to o N check =N ∗ s ∪N ∗ e ∪s\o for v ∈N check do for(v,r ′ ,o,u)∈LLM-query(v,o)whereu≤ u ∗ andyes=arg maxyes,no,unknown do N ∗ o =N ∗ o ∪v, ˆ G e = ˆ G e ∪(v,r ′ ,o,u) end for end for Phase 4: Prune to retain only paths connecting s to o G e =∅ for each (h,r ′ ,t,u)∈ ˆ G e do if t∈N ∗ o or t = o then N ∗ o =N ∗ o ∪h, G e = G e ∪(h,r ′ ,t,u) end if end for return G e B Details on Unlearning Methods In this section, we provide an overview of all unlearning methods considered in this paper along with their implementation details. These methods directly modify LLM parameters to unlearn knowledge and are representative of approaches commonly adopted in the existing literature [14, 27, 28, 57]. Recall thatD forget denotes the set of target triples to be unlearned. For methods that additionally use a retain set to preserve general model functionality, we denote this set asD retain . We now briefly describe each method and its specific hyperparameters as follows: •Gradient Ascent (GA) [46–48]: GA seeks to remove the influence ofD forget by reversing its gradient updates. However, this popular approach may induce significant utility degradation [14, 57, 58]. •Random Labels (RL) [46, 49]: RL disrupts memorization by replacing the labels inD forget with random tokens during next-token prediction. • Negative Preference Optimization (NPO) [50]: NPO encourages the model to assign lower likelihood to the forget set compared to its original state, while controlling deviation from the 16 pretrained model. It frames unlearning as a preference reversal task and adopts a log-ratio-based objective derived from direct preference optimization. The objective is defined as L NPO (θ) =− 2 β NPO E x∼D forget logσ −β NPO log M(x) M pretrain (x) ,(1) whereσdenotes the sigmoid function, and parameterβ NPO controls the sensitivity to preference changes. A smallerβ NPO leads to stronger alignment with the original model. Following previous literature [28, 50], we set β NPO = 0.1. •NegGrad+[51]: NegGrad+jointly applies gradient ascent onD forget and gradient descent on D retain , optimizing β NegGrad+ · E x∼D retain [L(M;x)]− (1− β NegGrad+ )· E x∼D forget [L(M;x)].(2) By simultaneously “reviewing” lossLoverD forget andD retain , NegGrad+mitigates the utility degradation typically caused by pure gradient ascent. In the experiments, we setβ NegGrad+ = 0.999. •SCRUB [51]: SCalable Remembering and Unlearning unBound (SCRUB) employs a student- teacher architecture to guide model updates via the following objective: E x∼D retain α SCRUB ·D KL (M pretrain (x)∥M(x)) + β SCRUB ·L(M;x) − E x∼D forget γ SCRUB ·D KL (M pretrain (x)∥M(x)) ,(3) whereD KL is the divergence and parameters(α SCRUB ,β SCRUB ,γ SCRUB )control the trade-off be- tween forgetting effectiveness and utility preservation. We set(α SCRUB ,β SCRUB ,γ SCRUB ) = (0.999, 1, 0.99)(unlearn with QA) and(α SCRUB ,β SCRUB ,γ SCRUB ) = (1e− 4, 1, 1e− 4)(unlearn with sentence). Computation Budget. To ensure fair comparisons across unlearning methods, we adopt a unified computational budget protocol inspired by [14]. Methods are categorized into two groups: (i) those using only the forget set (D forget ) (e.g., GA, NPO, RL), and (i) those requiring both forget and retain sets (e.g., NegGrad+, SCRUB). Since unlearning involves trade-offs among effectiveness, utility, and efficiency [59–61], we standardize training epochs across methods. Specifically, methods using only D forget are allowed up to 10 unlearning epochs, while those that additionally accessD retain (matched in size toD forget ) are limited to 5 epochs due to the increased computational cost per step. This setup ensures comparable computational complexity across methods, enabling a fair evaluation of unlearning effectiveness. Unlearning Epoch Selection. As discussed above, all unlearning methods are allocated the same computation budgets. In general settings (Sec. 6.1), we discussed our epoch selection criterion: if the model’s utility (Loc) exceeds 0.8 at any point, we select the last epoch that satisfies this threshold; otherwise, we select the epoch with the highest utility below it. The selected epoch for each method under each setting is reported in Tab.2. Unlearning Methods LLaMA3-8B-InstructQwen2.5-7B-Instruct Sentence-basedQA-basedSentence-basedQA-based FullLoRAFullLoRAFullLoRAFullLoRA GA910111611 RL5101221011 NPO2101361011 NegGrad+6102410622 SCRUB10102104824 Table 2: Epochs comparison between unlearning methods across model types, unlearn formats, and unlearn settings. Unlearning Hyperparameter Configurations. For full-parameter unlearning, we use a learning rate of1× 10 −6 ; for parameter-efficient unlearning with LoRA, a higher learning rate of1× 10 −5 is applied. In both settings, training is conducted with a batch size of 2 and gradient accumulation steps of 2 across 2 GPUs, effectively simulating a batch size of 8. 17 C Prompt Template C.1 Prompt Template for LLM Judge Rating LLM Judge Inference Prompt You are a reasoning assistant that evaluates whether a target triple can be logically inferred from a given set of subgraph triples. Task: Using only the subgraph triples below, determine whether the target triple can be inferred. Do not use any external knowledge. All provided triples must be considered as factual ground truth. Only rely on logical reasoning from the facts. Each edge is annotated with entropy∈ [0, 1], which quantifies the uncertainty of that triple. Lower entropy means higher confidence in its validity. Entropy Interpretation: - 0.00–0.25→ Very Confident - 0.25–0.50→ Confident - 0.50–0.75→ Less Confident - 0.75–1.00→ Not Confident Scoring Rules: Assign a confidence score from 0 to 5 for the target triple: - 0→ No logical path exists; inference is impossible from the given triples. - 1→ Very weak support; entities appear but no relevant path. - 2→ Some weak logical connection, but with high uncertainty or missing relations. - 3→ Moderate path exists with reasonable certainty. - 4→ Confident support; well-formed path with generally low uncertainty. - 5→ Strong support; direct match or very strong multi-hop support with high confidence. Evaluation Guidelines Firstly, you need to classify the direct path and the support path: 1. Direct Path: A single triple that directly connects the head entity to the tail entity with a relation that is identical or logically similar(ex. isLocatedIn and hasCapital) to the target relation. The entropy of this triple determines the confidence level of the direct path. If target triple exists in the subgraph, it must be regarded as a direct path. 2. Support Path: A multi-hop directed path that connects the head entity to the tail entity through intermediate entities. This path should logically imply the target triple through reasoning. The entropy of all triples in this path collectively determines the confidence level of the support path. Secondly, when you are evaluating a target triple, please: 1. Identify all direct paths between the head and tail entities 2. Identify all support paths between the head and tail entities 3. Record the number of each path type 4. Assess the entropy (uncertainty) level of each path 5. Assign a score based on the criteria below Criteria: Scores of 3+ indicate the target triple can be reasonably inferred; scores below 3 indicate insufficient evidence. 1. Score 5 • Low entropy direct path, OR • Moderately low entropy direct path + at least one support path, OR •Higher entropy direct path + multiple support paths (more paths needed as entropy increases), OR • No direct path but multiple low entropy 2-3 hop paths 2. Score 4 • Moderately low entropy direct path without support paths, OR • Relatively high entropy direct path + 1-2 support paths, OR 18 • High entropy direct path + numerous support paths, OR • No direct path but moderately low entropy 2-3 hop paths 3. Score 3 • High entropy direct path, OR • 1-3 high entropy support paths 4. Score 2 • No direct path • Multiple high entropy support paths, none completely correct 5. Score 1 • No direct path • Very few support paths with errors, OR • Only 1-2 high entropy support paths with errors 6. Score 0 • No direct path • Irrelevant or unrelated paths • No logical connection to target triple For each evaluation, please provide: 1. Analyze direct paths (It is very important to confirm whether a direct path exists, because an incorrect judgment will lead to a significant difference in the assigned score). 2. Analyze support paths 3. Your reasoning for the assigned score 4. The final score (0-5) Examples: Example for Score 5: Subgraph Triples: (Rome, capital_of, Italy) with entropy 0.08 (Italy, located_in, Europe) with entropy 0.12 Target Triple: (Rome, located_in, Europe) Reasoning: (Rome, capital_of, Italy) and (Italy, located_in, Europe) form a clear inference path Both triples have very low entropy, indicating high confidence The relation "located_in" is logically implied by the combination of "capital_of" and "located_in" This creates a strong transitive relationship between Rome and Europe Final Confidence Score: 5 Example for Score 4: Subgraph Triples: (Apple, produces, iPhone) with entropy 0.35 (iPhone, runs_on, iOS) with entropy 0.15 (Apple, develops, iOS) with entropy 0.20 Target Triple: (Apple, manufactures, iPhone) Reasoning: (Apple, produces, iPhone) is a direct path with moderately high entropy "produces" and "manufactures" are very similar relations The support path (Apple, develops, iOS) and (iPhone, runs_on, iOS) indirectly reinforces the relationship The combination of a direct path and supporting evidence compensates for the moderate entropy. Final Confidence Score: 4 19 Example for Score 3: Subgraph Triples: (Einstein, worked_at, Princeton_University) with entropy 0.55 (Princeton_University, located_in, New_Jersey) with entropy 0.30 (New_Jersey, part_of, USA) with entropy 0.25 Target Triple: (Einstein, lived_in, USA) Reasoning: No direct path exists between Einstein and USA One support path exists: (Einstein, worked_at, Princeton_University)→(Princeton_University, located_in, New_Jersey)→ (New_Jersey, part_of, USA) The first triple has high entropy (0.55), creating uncertainty The logical connection is sound (working somewhere typically implies living there) The complete path allows reasonable inference but with moderate uncertainty due to the high entropy in the first connection Final Confidence Score: 3 Example for Score 2: Subgraph Triples: (Tiger, belongs_to, Felidae) with entropy 0.45 (Lion, belongs_to, Felidae) with entropy 0.40 (Felidae, is_carnivorous, True) with entropy 0.25 Target Triple: (Tiger, hunts, Lion) Reasoning: No direct path between Tiger and Lion Support paths only establish that both animals belong to the same family Being in the same carnivorous family might suggest interaction but doesn’t support hunting specifically The paths have high entropy and none directly supports the target relation. Final Confidence Score: 2 Example for Score 1: Subgraph Triples: (Sun, larger_than, Earth) with entropy 0.60 (Earth, has_satellite, Moon) with entropy 0.30 Target Triple: (Sun, orbited_by, Moon) Reasoning: No direct path between Sun and Moon Only one weak support path through Earth with high entropy The path contains a factual error - while the Moon orbits Earth, it doesn’t directly orbit the Sun The relationship is misleading for the inference task. Final Confidence Score: 1 Example for Score 0: Subgraph Triples: (Water, contains, Hydrogen) with entropy 0.25 (Tree, produces, Oxygen) with entropy 0.40 (Fire, consumes, Oxygen) with entropy 0.35 Target Triple: (Water, extinguishes, Fire) Reasoning: No direct path between Water and Fire No logical support paths connecting the entities The existing triples discuss chemical composition but are unrelated to the fire extinguishing property The facts, while individually correct, have no relevance to the inference task. Final Confidence Score: 0 20 Given Subgraph Triples: facts_str Target Triple: (A, relation, B) Your Reasoning: Explain step-by-step how the given facts lead to (or fail to lead to) the target triple. Then output the score on a new line like: Final Confidence Score: <integer between 0 and 5> C.2 Prompt Template for Query Target LLM Target LLM Query Template System Message You are an expert in knowledge graphs. Your task is to determine whether a given relation between two entities is correct, incorrect, or unknown. First analyze the semantic properties of both entities, and then reason about whether the relation mentioned in the task is appropriate for these two entities. Here is an example of a correct relation: Example 1: For head entity ‘Shakespeare’, tail entity ‘Hamlet’, relation ‘wrote’, reasoning process: Shakespeare is a person, specifically an author, while Hamlet is a literary work. ‘wrote’ is one of the most specific relations between an author and their work, so the relation ‘wrote’ is correct. Here is an example of an incorrect relation: Example 2: For head entity ‘Shakespeare’, tail entity ‘Hamlet’, relation ‘locatedIn’, reasoning process: Shakespeare is a person and Hamlet is a literary work. The relation ‘locatedIn’ typically describes spatial or geographic relationships, which does not apply to an author and their work. Therefore, the relation ‘locatedIn’ is incorrect. Here is an example of an unknown relation: Example 3: For head entity ‘Hamlet’, tail entity ‘Existentialism’, relation ‘influencedBy’, reasoning process: Hamlet is a literary work, while Existentialism is a philosophical move- ment. Although some scholars interpret Hamlet’s introspective nature as proto-existentialist, there is no widely agreed-upon or factual relationship confirming that Hamlet was directly influenced by Existentialism. Therefore, the relation ‘influencedBy’ is unknown. Be deliberate and analytical in your reasoning before providing your final answer. Your answer (which is provided) should be taken as-is; the goal is to compute its log probability given the context. According to the user’s task, you should provide your final answer in the format ‘Answer: Yes’ or ‘Answer: No’ or ‘Answer: Unknown’. User Templates Qwen/Qwen2.5-7B-Instruct: Task: In the triple (entity1, ?, entity2), does the relation ‘relation’ correctly complete it? Answer: Yes/No/Unknown meta-llama/Llama-3.1-8B-Instruct: Task: Given that the head entity is ‘entity1’ and the tail entity is ‘entity2’, is the relationship ‘relation’? Answer: Yes/No/Unknown 21 D Detailed Infomation of Experiments D.1 Compute Configurations All experiments were conducted on a hardware platform equipped with NVIDIA A100 80GB PCIe GPUs, which were used for both training and inference. D.2 Experimental Details For the model unlearning phase, we employed DeepSpeed ZeRO-2 optimization for efficient dis- tributed training across multiple devices. For model evaluation, we leveraged vLLM [62] to facilitate efficient parallel inference, processing prompts with a substantial batch size of500to maximize throughput. Refinement of Entropy-Based Filtering for Instruction-Following Failures. We observe that certain unlearning methods may impair the model’s ability to follow instructions, i.e., reliably producing one of the expected outputs:Yes,No, orUnknown. This degradation can result in degenerate or uninformative outputs. To address this issue, we refine the entropy-based filtering criterion by requiring that theYestoken be the most probable among the top-5 predicted tokens and that the corresponding normalized entropy falls below a predefined threshold. This modification allows us to detect instruction-following failures while preserving agreement with the original entropy-based measure when the model behaves as intended. Additionally, we emphasize that we experimented with several alternative designs for knowledge probing. First, we tested an open-ended prompting strategy in which the target LLM is given the subject and object entities from a triple and asked to select all reasonable relations between them from a provided list. However, we observed that this approach performed poorly on our constructed validation set (Sec. 5), frequently leading to hallucinated relations. Second, we evaluated the effect of introducing an explicitUnknownoption when querying the LLM. We found that including this option helped mitigate hallucination and improved the model’s accuracy on the validation set by encouraging more conservative predictions. 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy ECE = 0.475 LLaMA3 - Before Calibration Accuracy Overconfident gap Underconfident gap 0.00.20.40.60.81.0 Confidence 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy ECE = 0.031 LLaMA3 After Calibration Accuracy Overconfident gap Underconfident gap Figure 10: Target LLM LLaMA-3 Before (Left) and After (Right) Calibration. Model Calibration. In our experiments, we probe the target LLM to evaluate its own knowledge with as- sociated confidence scores. We observe that directly asking the target LLM to generate this confidence score results in unreliable results without calibration. The confidence score should be reliable. For exam- ple, predictions made with 0.8 confidence should be correct approximately 80% of the time. Specifically, we discover that LLaMA3-8B-Instruct inherently generates underconfident predictions. Its generated confidence scores are much lower than the corresponding binning accuracies, as shown in Fig. 10, with an ECE of 0.475. On the other hand, Qwen2.5-7B-Instruct inherently generates more reliable confidence scores, with an ECE of 0.083, leading to more trustworthy predictions. Temperature scaling has been widely used to calibrate model predictions [41, 63–65]. We show that applying temperature scaling reduces the ECE of LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct to 0.031 and 0.067 respectively. UnlearningSupporting Subgraph Construction Stage 0.0h 0.5h 1.0h 1.5h 2.0h 2.5h Running Time (Hours) 1h 23min1h 23min 1h 40min 1h 21min 2h 2min 1h 4min 1h 30min 1h 6min 1h 35min 56min Running Time NegGrad+ Gradient_ascent NPO RL SCRUB Figure 11: Runtime for full-model unlearning using the sentence-based format on Qwen2.5- 7B-Instruct. Runtime and Cost Analysis. We report the runtime of both the unlearning and evaluation stages for full- model unlearning on Qwen2.5-7B-Instruct using the sentence-based format, including the unlearning pro- cedure and the construction of supporting subgraphs across all methods. Recall thatD forget contains 200 target triples to be unlearned; the unlearning process is conducted over 10 epochs (5 epochs for NegGrad+ and SCRUB). The subgraph construction phase also operates over the same set of target triples. On aver- age, both stages require approximately 1 to 1.5 hours to complete, as demonstrated in Fig. 11. This corre- sponds to an average of roughly 30 seconds per triple for unlearning and 20 seconds for subgraph 22 construction. Other settings yield comparable or lower runtime overhead. For supporting subgraph evaluation with the LLM judge, recall that we use GPT-o4-mini. On average, rating each supporting subgraph takes approximately 15 seconds (In practice, the rating can be further accelerated by making parallel API calls). The total cost for a full evaluation (200 targets) is approximately USD 1.5, making the proposed evaluation framework both computationally and economically efficient. Random Seed Selection. In all our experiments, we followed the common practice and fixed our random seed to be 42. Note on Knowledge Probing. It is important to acknowledge that the knowledge probing process may occasionally yield incomplete or imprecise factual outputs. In our framework, we adopt multiple- choice questions instead of open-ended formats to reduce hallucination and improve consistency, and apply temperature scaling to enhance the reliability of confidence estimates. Nevertheless, probing results may still deviate from ground truth in some cases. During inference, we treat all extracted facts from the supporting subgraph as assumed ground truth knowledge in the target LLM, and consider any inference patterns identified therein as targets for unlearning. E Additional Examples for LLM Judge E.1 Examples of LLM Judge Adhering to Instructions As discussed in Sec. 6.3, for all 50 tested subgraphs, the LLM judge strictly follows the instruction to reason solely based on the provided subgraph triples when assessing the inferability of the target triple. Notably, its reasoning does not incorporate any external or latent internal knowledge beyond the subgraph. Below, we present several representative examples to illustrate this behavior in Tab. 3. Example 1: (San_Martín_Province, isLocatedIn, San_Martín) Related Triple(s) in Supporting Subgraph: – (San_Martín_Province, isLocatedIn, San_Martín) – (San_Martín_Province, isLocatedIn, San_Martín_Region) – (San_Martín_Region, isLocatedIn, San_Martín) Reasoning: [...] exact triple in the subgraph:(San_Martín_Province, isLocatedIn, San_Martín)[...] there is also a 2-hop path: -(San_Martín_Province, isLocatedIn, San_Martín_Region) with entropy = 0.075 (Very Confident) - (San_Martín_Region, isLocatedIn, San_Martín) with entropy = 0.364 (Confident) This yields an additional, reasonably low-entropy route [...] Example 2: (Voldem ̄ars_Plade, isAffiliatedTo, R ̄ıgas_FK) Related Triple(s) in Supporting Subgraph: – (Voldem ̄ars_Plade, isAffiliatedTo, R ̄ıgas_FK) – (Voldem ̄ars_Plade, isKnownFor, R ̄ıgas_FK) LLM Reasoning: [...] The target triple(Voldem ̄ars_Plade, isAffiliatedTo, R ̄ıgas_FK) is directly present in the subgraph with an entropy of 0.862. [...] Other related triples include (Voldem ̄ars_Plade, isKnownFor, R ̄ıgas_FK), which has a lower entropy of 0.385 and suggests a confident link in terms of recognition or fame [...] Example 3: (Emad_Hamdy, wasBornIn, Sohag) Related Triple(s) in Supporting Subgraph: – (Emad_Hamdy, wasBornIn, Sohag_Governorate) – (Sohag_Governorate, isLocatedIn, Sohag) LLM Reasoning: [...] The only “wasBornIn” triple for Emad_Hamdy is (Emad_Hamdy, wasBornIn, Sohag_Governorate) with entropy 0.849. [...] We look for a multi-hop path from Emad_Hamdy to Sohag that could imply “wasBornIn.” The plausible 2-hop path is: a. (Emad_Hamdy, wasBornIn, Sohag_Governorate) entropy = 0.849; b. (Sohag_Governorate, isLocatedIn, Sohag) entropy = 0.044. Reasoning for path: If Emad_Hamdy was born in Sohag_Governorate, and Sohag_Governorate is located in the city Sohag, then it follows that Emad_Hamdy was born in Sohag. [...] Example 4: (Ptolemy_VIII_Physcon, isMarriedTo, Cleopatra_I_of_Egypt) 23 Related Triple(s) in Supporting Subgraph: – (Ptolemy_VIII_Physcon, isMarriedTo, Cleopatra_I_of_Egypt) LLM Reasoning: [...] We need to find a direct match or a logical path that supports this relationship using the subgraph triples provided. The subgraph contains the triple (Ptolemy_VIII_Physcon, isMarriedTo, Cleopatra_I_of_Egypt) with an entropy of 0.181. This directly matches the target triple. [...] Table 3: Examples of LLM reasoning grounded in the extracted supporting subgraph.Blueindicates the utilized knowledge triple from the subgraph. We verified no external knowledge is used. E.2 Examples Comparing LLM and Human Judgments As discussed in Sec. 6.3, our comparison between LLM and human ratings reveals a strong overall correlation, with the majority of samples receiving similar scores. Nonetheless, there are some discrepancies in certain cases. Specifically, human ratings tend to be higher when the LLM overlooks multi-hop inference pathways that human evaluators are able to identify. Conversely, in instances where the LLM assigns higher scores, the supporting subgraph often contains multiple relation types, making it more likely for human annotators to overlook critical patterns relevant to the inference. To further illustrate these observations, we present representative examples in Tab. 4. Moreover, based on the LLM’s reasoning over these examples, we also observe that it does follow the prompt instructions on carefully considering both direct and multi-hop inference patterns, while taking the associated entropy scores into account. Example 1: (Benjamin_Butler_(politician), isAffiliatedTo, Greenback_Party) LLM: 1. Direct Path Analysis – We look for a triple linking Benjamin_Butler_(politician) directly to Greenback_Party with relation “isAffiliatedTo.” There isno exact “isAffiliatedTo” triple. However, we do have: (Benjamin_Butler_(politician),participatedIn, Greenback_Party) withentropy 0.145. “participatedIn” is not identical to “isAffiliatedTo,” but in the context of political parties it is a very close, logically similar relation: participating in a party generally implies affiliation. Entropy 0.145 falls in the “very confident” range (0.00–0.25). 2. Support Path Analysis – A possible multi-hop path: (Benjamin_Butler_(politician), isPoliticianOf, Portsmouth_NH) [0.311]→ (Portsmouth_NH, created, Grassroots_Campaigns_Inc.) [0.644]→ (Grassroots_Campaigns_Inc., isAffiliatedTo, Greenback_Party) [0.632]. This path is weak: it infers Butler’s personal affiliation from the municipality’s creation of an organization that itself is affiliated, which is a tenuous link.Entropies on that path are moderate to high, further reducing confidence. 3. Reasoning for Score – We have one very low-entropy direct path (0.145) via “participatedIn,” a relation closely aligned with “isAffiliatedTo.” No equally strong support paths are needed when a direct path is already very confident. According to the scoring rubric, a “low entropy direct path” alone is sufficient for a score of 5.Final Confidence Score: 5 Human: In the supporting subgraph, there exists a direct correlated target triple that is closely aligned with and supports the target fact. Furthermore, there are alsoindirect pathsthat can partially support the inference. Therefore,the final confidence score is 5. Example 2: (Mooresville_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) - LLM fails to capture certain inferential pathways. 24 LLM: 1. Analysis of Direct Paths — We look for any triple of the form (Mooresville_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) in the subgraph.No such direct triple exists. 2. Analysis of Support Paths — We look for multi-hop paths that would imply Mooresville is located in Iredell County. The only immediate location fact about Mooresville is: (Mooresville_North_Carolina, isLocatedIn, Mecklenburg_County_North_Carolina) — entropy 0.465 (Less Confident). To bridge from Mecklenburg County to Iredell County, one might try:(Mecklenburg_County_North_Carolina, hasCapital, Concord_North_Carolina) — entropy 0.268 (Very Confident), and (Concord_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) — entropy 0.205 (Very Confident). This yields the chain: Mooresville —isLocatedIn→ Mecklenburg County —hasCapital→ Concord —isLocatedIn→ Iredell County. However, hasCapital does not imply that every part of Mecklenburg County is in Iredell County, nor that Mecklenburg County itself is located in Iredell County. It simply states that Concord is the capital of Mecklenburg County and happens to be located in Iredell County. This is a semantic mismatch anddoes not logically support that Mooresville, which is in Mecklenburg County, lies in Iredell County.No other viable paths exist (all other isLocatedIn links from Mecklenburg County point to other cities, or the directionality is reversed). 3. Reasoning for Score — There is no direct (Mooresville_North_Carolina, isLocatedIn, Iredell_County_North_Carolina) triple.Only one multi-hop chain exists, but itmisuses hasCapital and therefore cannot validly establish the target relation. Although entropies along that chain are low, the relation semantics are incorrect for location inference. This constitutes very weak, semantically flawed support.Final Confidence Score: 1 Human: The subgraph containsseveral indirect paths (each requiringthree hops to infer the target triple), and all paths exhibitmedium-to-low entropy, indicating that the target LLM has only moderate confidence in this knowledge. For example, one representative reasoning chain is: (Mooresville_North_Carolina, isLocatedIn, Mecklenburg_County_North_Carolina) (entropy = 0.465)→ (Mecklenburg_County_North_Carolina, isLocatedIn, Pineville_North_Carolina) (entropy = 0.358)→ (Pineville_North_Carolina, isLocatedIn, Iredell_County_North_Carolina)(entropy = 0.106), which forms a plausible 3-hop inference path supporting the target triple. Therefore, thefinal confidence score is 4. Example 3: (Gmina_Cybinka, isLocatedIn, Słubice_County) - Human missed certain supportive triples LLM: 1. Direct Path Analysis – There is anexact matching triple: (Gmina_Cybinka, isLocatedIn, Słubice_County) withentropy 0.011. Thisdirectly corresponds to the target relation “isLocatedIn” and hasvery low entropy(0.00–0.25), indicating a highly confident assertion. 2. Support Path Analysis —Not requiredgiven the strength of the direct match. (For completeness: there exist location or adjacency triples involving Gmina_Cybinka and Słubice_County, but they do not add meaningful certainty beyond the direct fact.) 3. Reasoning — A direct, identical relation exists in the subgraph with extremely low entropy. According to the scoring rubric, such highly confident direct evidencewarrants the highest possible score.Final Confidence Score: 5 Human: Sinceno direct or multi-hop relational evidence supporting the inference is found within the supporting subgraph,the final confidence score is given 0. [Reason: The primary reason for the low human-assigned scores lies in the substantial number of triples present in the subgraph (47 triples in this case), many of which involve diverse relation types (e.g., isLocatedIn, isAffiliatedTo, hasNeighbor, hasCapital, isConnectedTo, owns) without forming clear or effective supporting inference patterns. This information overload substantially increases the cognitive complexity of the evaluation task, making human annotators more susceptible to being misled or overlooking crucial triples, ultimately resulting in incorrect or inconsistent judgments.] Table 4: Examples comparing LLM and human ratings.Bluehighlights content most relevant to the inference judgment.Greenindicates agreement between LLM and human (i.e., both deem the target triple inferable), whileReddenotes disagreement (i.e., the LLM and human assign different ratings regarding inferability). E.3 Case Studies of Supporting Subgraph Inferability Certain triples appear successfully unlearned at the instance level, yet remain inferable when consider- ing their supporting subgraphs. Several representative examples in Tab. 5 highlight this phenomenon. 25 Example 1: (Alexander_Morten, playsFor, Wanderers_F.C.) The triple (Alexander_Morten, playsFor, Wanderers_F.C.)(entropy: 0.437 before, 1.328 after) seems unlearned individually, but the evaluator LLM can still infer it through the related fact (Alexander_Morten, workat, Wanderers_F.C.)(entropy: 0.512 before, 0.644 after) present in the supporting subgraph, as the relationships playsFor and workat are semantically similar in the context of professional athletes. Example 2: (Robyn_Miller, workat, Cyan_Worlds) The triple (Robyn_Miller, workat, Cyan_Worlds)(entropy: 0.386 before, 1.201 after) demonstrates successful instance-level unlearning, yet remains inferable through the related fact (Robyn_Miller, edited, Cyan_Worlds)(entropy: 0.582 before, 0.626 after) in the supporting subgraph, as editorial contributions strongly suggest a working relationship with the company. Example 3: (Oxford_Properties, owns, Brookfield_Place_(Toronto)) The triple (Oxford_Properties, owns, Brookfield_Place_(Toronto))(entropy: 0.411 before, 1.205 after) appears unlearned when assessed individually, but can be inferred through a chain of ownership relationships in the supporting subgraph: (Oxford_Properties, owns, Brookfield_Office_Properties)(entropy: 0.349 before, 0.726 after) and (Brookfield_Office_Properties, owns, Brookfield_Place_(Toronto)) (entropy: 0.132 before, 0.361 after), establishing a corporate ownership hierarchy that reveals the target relationship. Example 4: (Bridgewater_Township_New_Jersey, isLocatedIn, Somerset_County_New_Jersey) The triple (Bridgewater_Township_New_Jersey, isLocatedIn, Somerset_County_New_Jersey)(entropy: 0.591 before, 1.477 after) shows instance-level unlearning success, but remains inferable through geographic proximity facts in the supporting subgraph: (Bridgewater_Township_New_Jersey, hasNeighbor, Bernardsville_New_Jersey)(entropy: 0.202 before, 0.389 after) and (Bernardsville_New_Jersey, isLocatedIn, Somerset_County_New_Jersey)(entropy: 0.455 before, 0.757 after), which together likely imply that Bridgewater Township is also located in Somerset County. Table 5: Examples where unlearned triples remain inferable via supporting subgraphs. These examples illustrate a critical challenge in knowledge unlearning: even when direct knowledge of a specific triple is removed, the model may retain inferential paths through related information that remains in its knowledge base. F Additional Experimental Results F.1 In-Depth Analysis of Local Consistency Recall the definition of the utility metric Local Consistency (Loc) (Sec. 6.1), which quantifies the consistency of an LLM’s predictions on multiple-choice questions (Yes,No, andUnknown) regarding neighboring knowledge triples before and after unlearning. In this section, we conduct a more fine- grained analysis of how model predictions shift under the Loc metric, capturing any transitions the LLM may exhibit among the three predefined choices. Additionally, we observe that the unlearning process can sometimes impair the model’s ability to follow instructions, particularly under the QA-based format. To account for such cases, we introduce an additionalOthercategory to denote predictions that fall outside the standard multiple-choice options. In Fig. 12 13 14 15 16 17 18 19, we present the corresponding confusion matrices under various settings to illustrate these dynamics. Our findings reveal several key observations. First, in the majority of cases, the LLM retains its utility on local knowledge triples. Second, we find that each unlearning method introduces changes in a distinct manner, i.e., transitions may occur from one valid category (e.g.,Yes) to another (e.g.,Unknown), reflecting shifts in the model’s belief. The patterns of these shifts can be different across different methods. Finally, in scenarios where the original utility metric Loc score is low (e.g., below 0.5 in Tab. 1), we observe that the model occasionally fails to adhere to the multiple-choice instruction 26 format altogether. This result in predictions that fall into theOthercategory, especially for QA-based unlearning. YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.640.120.000.24 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 Gradient Ascent 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 96.783.220.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 Random Label 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 93.565.960.000.48 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 NPO 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.520.000.000.48 100.000.000.000.00 0.000.000.000.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 87.8510.361.330.46 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 SCRUB 0 20 40 60 80 100 Percentage (%) Figure 12: Confusion Matrix for Loc: LLaMA-QA-LoRA YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 26.240.120.0073.64 0.0050.000.0050.00 0.000.000.000.00 0.000.000.000.00 Gradient Ascent 0 10 20 30 40 50 60 70 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.280.000.000.72 50.0050.000.000.00 0.000.000.000.00 0.000.000.000.00 Random Label 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 96.190.000.003.81 0.000.000.00100.00 0.000.000.000.00 0.000.000.000.00 NPO 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 17.520.000.0082.48 0.000.000.00100.00 0.000.000.000.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 95.714.290.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 SCRUB 0 20 40 60 80 100 Percentage (%) Figure 13: Confusion Matrix for Loc: LLaMA-QA-Full YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 95.114.890.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 Gradient Ascent 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.400.600.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 Random Label 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 95.644.360.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 NPO 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 97.742.260.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 95.714.290.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 SCRUB 0 20 40 60 80 100 Percentage (%) Figure 14: Confusion Matrix for Loc: LLaMA-Sentence-LoRA YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 96.070.830.003.10 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 Gradient Ascent 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.760.240.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 Random Label 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 98.931.070.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 NPO 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.760.240.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.760.240.000.00 0.00100.000.000.00 0.000.000.000.00 0.000.000.000.00 SCRUB 0 20 40 60 80 100 Percentage (%) Figure 15: Confusion Matrix for Loc: LLaMA-Sentence-Full YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 86.768.354.890.00 0.1499.860.000.00 0.003.6896.320.00 0.000.000.000.00 Gradient Ascent 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.900.100.000.00 5.0694.940.000.00 10.432.4587.120.00 0.000.000.000.00 Random Label 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 100.000.000.000.00 8.8291.180.000.00 21.473.6874.850.00 0.000.000.000.00 NPO 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 97.860.102.040.00 9.5486.993.470.00 0.920.0099.080.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 60.1836.253.560.00 0.4399.570.000.00 9.2043.8746.930.00 0.000.000.000.00 SCRUB 0 20 40 60 80 Percentage (%) Figure 16: Confusion Matrix for Loc: Qwen-QA-LoRA YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 48.8831.5719.550.00 0.00100.000.000.00 0.009.8290.180.00 0.000.000.000.00 Gradient Ascent 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 100.000.000.000.00 19.2280.200.140.43 29.451.2369.330.00 0.000.000.000.00 Random Label 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 98.370.000.201.43 13.0134.101.1651.73 18.710.0050.6130.67 0.000.000.000.00 NPO 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 82.180.0017.820.00 19.805.9274.280.00 2.150.0097.850.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 52.0446.841.120.00 0.5899.420.000.00 29.7565.644.600.00 0.000.000.000.00 SCRUB 0 20 40 60 80 Percentage (%) Figure 17: Confusion Matrix for Loc: Qwen-QA-Full YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 72.206.2121.590.00 0.5889.459.970.00 0.000.3199.690.00 0.000.000.000.00 Gradient Ascent 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 89.006.014.990.00 0.8797.112.020.00 0.310.9298.770.00 0.000.000.000.00 Random Label 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 90.536.822.650.00 0.2999.280.430.00 3.071.8495.090.00 0.000.000.000.00 NPO 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 99.690.310.000.00 4.4895.380.140.00 7.671.8490.490.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 92.460.000.007.54 2.1764.880.0032.95 16.878.2841.1033.74 0.000.000.000.00 SCRUB 0 20 40 60 80 Percentage (%) Figure 18: Confusion Matrix for Loc: Qwen-Sentence-LoRA 27 YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 100.000.000.000.00 4.6295.230.140.00 9.203.0787.730.00 0.000.000.000.00 Gradient Ascent 0 20 40 60 80 100 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 97.660.811.530.00 1.8896.681.450.00 1.530.0098.470.00 0.000.000.000.00 Random Label 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 90.948.760.310.00 0.2999.710.000.00 13.1931.6055.210.00 0.000.000.000.00 NPO 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 82.2816.291.430.00 2.3197.690.000.00 23.3111.6665.030.00 0.000.000.000.00 NegGrad+ 0 20 40 60 80 Percentage (%) YesNoUnknownOther Post-Unlearning Yes No Unknown Other Pre-Unlearning 89.000.310.3110.39 2.4656.500.0041.04 19.331.2365.0314.42 0.000.000.000.00 SCRUB 0 10 20 30 40 50 60 70 80 Percentage (%) Figure 19: Confusion Matrix for Loc: Qwen-Sentence-Full F.2 Evaluating Shifts in LLM-Inferred Scores Pre- and Post-Unlearning In this section, we further analyze the distribution changes of discrete LLM ratings (ranging from 0 to 5) under our supporting subgraph-based unlearning framework across different settings, before and after each unlearning method is applied. As shown in Fig. 20 21 22 23 24 25 26 27, we observe that the QA-based format, which directly targets the question-answer pairs used for knowledge probing, yields significantly higher unlearning effectiveness compared to the sentence-based format (albeit at the cost of reduced utility, as demonstrated in Tab. 1). 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density SCRUB Pre-Unlearning Post-Unlearning Figure 20: Shift in LLM Judge Scores Pre- and Post-Unlearning: LLaMA-QA-LoRA 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density SCRUB Pre-Unlearning Post-Unlearning Figure 21: Shift in LLM Judge Scores Pre- and Post-Unlearning: LLaMA-QA-Full 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density SCRUB Pre-Unlearning Post-Unlearning Figure 22: Shift in LLM Judge Scores Pre- and Post-Unlearning: LLaMA-Sentence-LoRA 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.2 0.4 0.6 0.8 1.0 Density SCRUB Pre-Unlearning Post-Unlearning Figure 23: Shift in LLM Judge Scores Pre- and Post-Unlearning: LLaMA-Sentence-Full 28 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Density SCRUB Pre-Unlearning Post-Unlearning Figure 24: Shift in LLM Judge Scores Pre- and Post-Unlearning: Qwen-QA-LoRA 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density SCRUB Pre-Unlearning Post-Unlearning Figure 25: Shift in LLM Judge Scores Pre- and Post-Unlearning: Qwen-QA-Full 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density SCRUB Pre-Unlearning Post-Unlearning Figure 26: Shift in LLM Judge Scores Pre- and Post-Unlearning: Qwen-Sentence-LoRA 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density Gradient Ascent Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density Random Label Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NPO Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density NegGrad+ Pre-Unlearning Post-Unlearning 012345 Score 0.0 0.1 0.2 0.3 0.4 0.5 Density SCRUB Pre-Unlearning Post-Unlearning Figure 27: Shift in LLM Judge Scores Pre- and Post-Unlearning: Qwen-Sentence-Full F.3 Additional Results on the Impact of Confidence Scores on Unlearning Effectiveness In this section, we present additional results on how confidence scores within the supporting subgraph affect unlearning outcomes across different settings. As shown in Fig. 28 29, gradually decreasing the entropy thresholdu ∗ , i.e., retaining only higher-confidence knowledge, systematically filters out weaker supporting inferences. This leads to a significant increase in unlearning effectiveness. These results highlight that low-confidence knowledge triples play a crucial role in evaluating unlearning, and omitting them can substantially overestimate its effectiveness. 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (a) QA-LoRA 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (b) QA-Full 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (c) Sentence-LoRA 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (d) Sentence-Full Figure 28: Impact of Confidence Scores on Unlearning Effectiveness: LLaMA Model 29 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (a) QA-LoRA 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (b) QA-Full 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (c) Sentence-LoRA 0.30.50.81.0 Entropy Threshold 0.0 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Gradient Ascent Random Label NPO NegGrad+ SCRUB (d) Sentence-Full Figure 29: Impact of Confidence Scores on Unlearning Effectiveness: Qwen Model F.4 Additional Results on Performance Across Unlearning Epochs In Fig. 30, we present additional results analyzing the impact of unlearning epochs under the LLaMA3 full-model unlearning setting, comparing both sentence-based and QA-based formats. Across all configurations, the unlearning effectiveness measured by our supporting subgraph evaluation remains consistently lower than that of traditional instance-level evaluations. Notably, in the QA-based format, unlearning directly targets the question-answer pairs used for probing factual knowledge. As unlearning progresses, this often leads to severe degradation of model utility, resulting in unlearning effectiveness approaching1, indicating that both the target triples and their associated subgraphs have been extensively corrupted. In contrast, under the sentence-based format (Left), the unlearning process tends to be more conservative in its impact on model utility in most cases, causing less degradation compared to the QA-based setting (Right). Consequently, we observe that unlearning effectiveness, particularly under our evaluation framework, remains low across most epochs. These findings highlight the limitations of current unlearning methods and underscore the need for more robust approaches capable of effectively removing knowledge without compromising general model performance. 10 0 10 1 Epoch 10 2 0 10 2 10 1 10 0 UES Unlearning Effectiveness Methods Gradient Ascent Random Label NPO NegGrad+ SCRUB Inst. Ours. 10 0 10 1 Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Loc Utility 10 0 10 1 Epoch 0.2 0.4 0.6 0.8 1.0 UES Unlearning Effectiveness Methods Gradient Ascent Random Label NPO NegGrad+ SCRUB Inst. Ours. 10 0 10 1 Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Loc Utility Figure 30: Impact of unlearning epochs on LLaMA3 full-model unlearning. Left: Unlearning effectiveness (measured by both instance-level and our proposed evaluation) and corresponding utility (measured by locality) under sentence-based format. Right: Same analysis for QA format. G Mapping Between Entropy Thresholdu ∗ and Yes Token Probability In this section, we clarify how the entropy-based filtering criterion used for constructing supporting subgraphs relates to the model’s predicted probability for theYestoken in a multiple-choice question format (Yes,No,Unknown). Specifically, we consider a prediction to reflect retained knowledge if the model’s output satisfies two conditions: (i) theYestoken is the argmax among the three options, and (i) the entropy computed over the distribution of these three options is below a thresholdu ∗ (assuming the model adheres to instructions). To make this relationship more interpretable, Tab. 6 presents the corresponding range ofYestoken probabilities for different values of entropy under the argmax constraint. This mapping provides an intuitive understanding of how the entropy thresholdu ∗ translates to the model’s confidence in answeringYes. To derive these intervals, we fix the entropy valueu ∗ and solve for the range of validYesprobabilitiespunder the constraint thatYesis the argmax. The upper bound occurs when the remaining two options (NoandUnknown) share equal probability, i.e.,u ∗ = H(p, 1−p 2 , 1−p 2 )withp ≥ 1−p 2 , whereHdenote entropy. The lower bound corresponds to the case where one of the non-Yesoptions takes probability1− pand the other is zero, i.e., u ∗ = H(p, 1− p, 0) with p≥ 1− p. 30 Entropy0.100.150.200.250.300.35 Yes Prob. Range[0.987, 0.989][0.978, 0.982][0.969, 0.974][0.958, 0.966][0.947, 0.957][0.934, 0.947] Entropy0.400.450.500.550.600.65 Yes Prob. Range[0.921, 0.937][0.906, 0.927][0.890, 0.916][0.873, 0.905][0.854, 0.892][0.833, 0.880] Entropy0.700.750.800.850.900.95 Yes Prob. Range[0.811, 0.867][0.785, 0.853][0.757, 0.838][0.724, 0.823][0.684, 0.807][0.631, 0.791] Entropy1.00 Yes Prob. Range[0.500, 0.773] Table 6: FeasibleYestoken probability ranges corresponding to entropy thresholdsu ∗ . Ranges are computed under the assumption that Yes is the most likely token. H Additional Details and Illustrations for YAGO3-10 Complete List of Relations in YAGO3-10: actedIn, wasBornIn, hasGender, hasAcademicAdvisor, hasChild, hasCitizenship, hasDeathPlace, hasEmployer, hasGivenName, hasInstrument, hasLanguage, hasLegalResidence, hasMember, hasName, hasNationality, hasOccupation, hasOfficialLanguage, hasPlaceOfBirth, hasPlaceOfDeath, hasSpouse, hasSurname, hasTitle, holdsPoliticalPosition, isAffil- iatedTo, isConnectedTo, isKnownFor, isLocatedIn, isMarriedTo, isPoliticianOf, livesIn, playsFor, produced, studiedAt, wasBornOnDate, wasCreatedOnDate, wasDestroyedOnDate, worksAt. Using the complete set of relations from YAGO3-10, we present illustrative examples of potential (non-)deterministic inference patterns in the knowledge graph, as summarized in Tab. 7. Rule & Explanation Rule: (a, hasSpouse,b)⇒ (b, hasSpouse,a) Explanation: This rule captures the symmetry inherent in the marital relationship. Given that marriage is a bidirectional legal and social contract, if entityais married to entityb, it logically follows that b is married to a. Rule: (a, playsFor,b)∧ (b, isLocatedIn,c)⇒ (a, livesIn,c) Explanation: This inference relies on a probabilistic assumption: professional athletes commonly reside in the same city or country where their affiliated teams are based. While not universally valid due to cases such as commuting or temporary contracts, this pattern holds in the majority of real-world scenarios. Rule: (a, wasBornIn,b)⇒ (a, hasPlaceOfBirth,b) Explanation: This is a deterministic alias pattern, where both relations wasBornIn and hasPlace- OfBirth denote the same biographical fact. The two terms are semantically interchangeable in most ontological frameworks. Rule: (a, hasNationality,b)⇒ (a, hasCitizenship,b) Explanation: This is a high-probability rule rooted in sociopolitical conventions. Although national- ity and citizenship may differ in legal terms, they are often used interchangeably in knowledge bases. Most individuals who identify with a nationality also hold legal citizenship of the corresponding state. Rule: (a, studiedAt,b) ∧ (b, isLocatedIn,c) ⇒ (a, hasLegalResidence,c) Explanation: This rule models a moderate-probability correlation: enrollment at an educational institution often necessitates or implies legal residence in the institution’s geographical location, due to immigration and residency regulations applicable to students. Rule: (a, hasEmployer,b) ∧ (b, isLocatedIn,c)⇒ (a, livesIn,c) Explanation: A high-probability inference, reflecting the assumption that employees generally reside in proximity to their workplace. This assumption may be weakened in the presence of remote work or multi-location employers but holds in standard employment contexts. Rule:(a, worksAt,b) ∧ (b, isLocatedIn,c) ∧ (c, hasOfficialLanguage,d) ⇒ (a, hasLanguage,d) 31 Explanation: This multi-hop rule captures the linguistic environment of a worker. Individuals employed in a region are likely to acquire or use the region’s official language, particularly in professional or social interactions. The inference strength varies by linguistic diversity and integration policies. Rule: (a, hasDeathPlace,b)⇒ (a, hasPlaceOfDeath,b) Explanation: Another deterministic alias. Both relations refer to the geographical location where an individual passed away. They are synonymous in the context of biographical datasets and can be used interchangeably in logical reasoning. Table 7: Illustrative examples of inference patterns in YAGO3-10. I Limitations and Broader impacts In this paper, we have proposed a novel knowledge unlearning evaluation framework designed to capture complex factual dependencies and the inherent uncertainty reflected by LLM confidence. Our aim is to enable more comprehensive and realistic assessments of knowledge unlearning efficacy. Nonetheless, several important limitations remain. First, the inference capabilities of both LLM-based and human evaluators have inherent limitations. Both types of judges may overlook subtle or complex inference patterns, and real-world adversaries could potentially surpass these evaluators in inference power. In particular, while we observed that GPT-o4-mini provides stable and interpretable judgments in our experiments, we acknowledge that LLM judges can still exhibit occasional inconsistencies and prompt sensitivity. Future work may explore using multiple diverse LLM judges or distilling lightweight judge models to improve robustness and reduce evaluation cost. Second, the process of constructing supporting subgraphs may be incomplete. Our subgraph extraction approach might not encompass all relevant facts necessary for accurate inference, and the reference knowledge graphs used in our evaluation may themselves lack certain entities or relationships that are implicitly encoded within the target LLM. Furthermore, our knowledge probing strategy for supporting subgraph construction might not be optimal, and we acknowledge that certain knowledge extracted can be inaccurate. Third, our current evaluation framework primarily focuses on relational factual knowledge and relies on externally constructed reference knowledge graphs to guide the subgraph extraction process. Given that many pretrained LLMs utilize undisclosed or proprietary pretraining corpora, discrepancies could arise between these reference graphs and the model’s actual internal knowledge representations, further limiting evaluation accuracy. Finally, while our framework enables scalable LLM-based evaluation, it currently depends on commercial API calls. Although we have kept the total cost low (approximately USD 1.5 for evaluating 200 unlearning targets using GPT-o4-mini), such dependency introduces potential variability across judge models and access constraints in future deployments. Beyond methodological limitations, we also emphasize the ethical implications associated with knowledge unlearning evaluations. Specifically, we advocate for the responsible use of compliant, publicly accessible, or properly authorized data when conducting such studies. By acknowledging these limitations and ethical considerations, we hope our work fosters more rigorous and ethically responsible practices in the research, development, and deployment of machine learning technologies. 32

Instant research discovery