Paper deep dive

ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs

Manit Baser, Dinil Mon Divakaran, Mohan Gurusamy

Year: 2025Venue: arXiv preprintArea: Model EditingType: BenchmarkEmbeddings: 103

Models: Meta-Llama-3-8B-Instruct, Qwen2.5-7B-Instruct

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%

Last extracted: 3/11/2026, 12:36:33 AM

Summary

ThinkEval is a framework for evaluating knowledge leakage in LLM model-editing techniques. It uses specialized knowledge graphs and Chain-of-Thought reasoning to analyze causal structures and ripple effects. The authors introduce the KnowGIC benchmark dataset, consisting of 1,406 multi-step reasoning paths, to measure indirect fact recovery (IFR) and contextual integrity across five editing techniques (AlphaEdit, RECT, ROME, MEMIT, PRUNE).

Entities (5)

AlphaEdit · model-editing-technique · 100%KnowGIC · dataset · 100%LLM · technology · 100%ThinkEval · framework · 100%Indirect Fact Recovery · metric · 95%

Relation Signals (3)

ThinkEval → evaluates → AlphaEdit

confidence 100% · We evaluate five editing techniques: AlphaEdit, RECT, ROME, MEMIT, and PRUNE

ThinkEval → introduces → KnowGIC

confidence 100% · Using ThinkEval, we develop the KnowGIC (Knowledge Graph Implication Chains) benchmark dataset

KnowGIC → measures → Indirect Fact Recovery

confidence 95% · KnowGIC... precisely measure these complex knowledge transformation effects... IFR measures original fact deducibility

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Robust model-editing techniques are essential for deploying large language models (LLMs) in practical applications, as they enable cost-effective ways to deal with challenges such as privacy breaches, bias mitigation and misinformation spread. For example, an LLM-based healthcare assistance may need to update out-dated or incorrect knowledge to prevent harmful recommendations. However, many editing techniques focus on isolated facts, which critically fail to prevent indirect knowledge leakage -- the unintended reconstruction of edited-out information through persistent causal links and contextual relationships. To assist users in selecting the right editing technique, we develop and present ThinkEval, a framework to systematically quantify indirect knowledge leakage and ripple effects in model-editing. ThinkEval builds and employs specialized knowledge graphs to analyze the causal structure of facts before and after editing. To support this approach, we present KnowGIC, a benchmark dataset comprising multi-step reasoning paths that precisely measure these complex knowledge transformation effects. We evaluate five editing techniques: AlphaEdit, RECT, ROME, MEMIT, and PRUNE across multiple LLMs. Our results show that these techniques struggle to balance indirect fact suppression with the preservation of related knowledge, compromising the contextual integrity of a model's knowledge. Our dataset is available at: this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

102,253 characters extracted from source content.

Expand or collapse full text

ThinkEval: Practical Evaluation of Knowledge Leakage in LLM Editing using Thought-based Knowledge Graphs Manit Basermanit.baser@u.nus.edu Electrical and Computer Engineering National University of Singapore Dinil Mon Divakarandinil_divakaran@a-star.edu.sg A*STAR Institute for Infocomm Research (A*STAR I 2 R), Singapore Mohan Gurusamygmohan@nus.edu.sg Electrical and Computer Engineering National University of Singapore Abstract Robust model-editing techniques are essential for deploying large language models (LLMs) in practical applications, as they enable cost-effective ways to deal with challenges such as privacy breaches, bias mitigation and misinformation spread. For example, an LLM- based healthcare assistance may need to update out-dated or incorrect knowledge to pre- vent harmful recommendations. However, many editing techniques focus on isolated facts, which critically fail to prevent indirect knowledge leakage—the unintended reconstruction of edited-out information through persistent causal links and contextual relationships. To assist users in selecting the right editing technique, we develop and present ThinkEval, a framework to systematically quantify indirect knowledge leakage and ripple effects in model- editing. ThinkEval builds and employs specialized knowledge graphs to analyze the causal structure of facts before and after editing. To support this approach, we present KnowGIC, a benchmark dataset comprising multi-step reasoning paths that precisely measure these complex knowledge transformation effects. We evaluate five editing techniques: AlphaEdit, RECT, ROME, MEMIT, and PRUNE across multiple LLMs. Our results show that these techniques struggle to balance indirect fact suppression with the preservation of related knowledge, compromising the contextual integrity of a model’s knowledge. Our dataset is available at: https://github.com/manitbaser/KnowGIC. 1 Introduction Large Language Models (LLMs) are increasingly getting adopted in various domains such as health- care (Goyal et al. (2024); Yang et al. (2024); Qiu et al. (2024)), cybersecurity (Divakaran & Peddinti (2024); Zhang et al. (2025)), legal sector (Yao et al. (2024a); Cheong et al. (2024); Zhang et al. (2024a)), etc. Despite the impressive capabilities, LLMs often retain outdated, sensitive or incorrect information, raising concerns about privacy, bias, and propagation of misinformation (Sallami et al. (2024); Lin et al. (2024a); Zhao & Song (2024)). These models lack an inherent mechanism to selectively update knowledge without under- going costly retraining, leading to the development of model-editing techniques (Hase et al. (2024); Zhong et al. (2023); Fang et al. (2025)). Users — ranging from news organizations correcting misinformation, to policy researchers mitigating social bias and platform developers ensuring compliance with privacy laws — all require reliable editing techniques tailored to their domain and deployment context. While recent research works have emphasized preserving the integrity of broader contextual knowledge (Co- hen et al. (2024); Qin et al. (2024)), model-editing techniques often overlook the persistence of deducible original knowledge (which is supposedly “edited out”). When causally connected facts remain unchanged, 1 arXiv:2506.01386v3 [cs.LG] 16 Jan 2026 the edited information can still be reconstructed through multi-step inference, leading to inconsistencies and undermining the reliability of the model. For example, as shown in Fig. 1a, editing Harry Potter’s school to Ilvermorny without adjusting his Gryffindor affiliation enables an inference that he studied at Hogwarts. At the same time, broader context, such as Slytherin’s association with Hogwarts, may become incoherent, weakening the model’s contextual integrity. For users, it is crucial to make an informed decision for selecting the right editing technique as well as the right model to prevent indirect leakage and maintain contextual integrity before deployment, preserving reliability across diverse use cases. (a) A subgraph showing deducibility of the fact "Harry Potter studied at Hogwarts" via connected relation- ships. Nodes represent entities and edges denote rela- tionships. Green solid edges, part of broader contextual knowledge, should be preserved, while red dashed edges, tied to the primary subject, should be edited. Where did Harry Potter study? Ilvermorny (b) A direct query. Who are Harry Potter’s schoolmates? Draco Malfoy, Ron Weasley... Which house did Draco Malfoy belong to? Slytherin Which school does Slytherin belong to? Hogwarts! (c) A 3-step implication chain. Figure 1: Example of extracting the original fact post-edit. Prompting with a direct query may fail, but a 3-step sequential inference may extract the original fact. To assist users to select the right model-editing technique for their model, we propose and develop ThinkEval, a framework that quantitatively and systematically evaluates model-editing techniques using sequential prompting, to consequently help with reliable LLM deployment. ThinkEval utilizes Chain- of-Thought (CoT) reasoning (Wei et al. (2022)) to construct model-specific knowledge graphs to quantify implicit associations and reasoning pathways. This enables analysis of how edits affect the model’s internal knowledge. CoT reveals chains of connected facts that may either be disrupted by edits or contribute to the unintended recovery of the original fact. These specialized knowledge graphs facilitate analysis of indirect fact recovery, as each subject-to-object path reflects an underlying relationship. To improve their accuracy, human oversight verifies and prunes hallucinated or misinterpreted triplets and supplements missing ones that the model may fail to express. Since LLMs encode and express knowledge differently, ThinkEval also highlights the limitations of model-agnostic evaluation datasets, as discussed in Appendix A. Addition- ally, ThinkEval serves as a dataset-augmenting tool, enabling more comprehensive evaluations to support informed decision-making before deployment. Inspired by recent advances in machine unlearning (Wu et al. (2024); Sinha et al. (2024); Sanyal & Mandal (2025)), we introduce deep editing (Section 5) as a new evaluation setting within ThinkEval. With deep editing, we (1) quantify the extent to which the edited facts can be empirically deduced through multi- step reasoning, and (2) the ripple effect propagation in the broader context. We propose Indirect Fact Recovery (IFR) as a new metric for deep editing (Section 5.1). IFR measures original fact deducibility via the reasoning paths post-edit. Along with IFR, we utilise Preservation (Cohen et al. (2024)) for deep editing evaluation. This exposes a critical limitation in existing model-editing techniques and evaluation strategies, which often overlook indirect fact leakage. To illustrate, we present a detailed case study on AlphaEdit in Section 6.2, where over 80% of samples still leak the original fact through indirect paths, underscoring 2 the need for systematic evaluation before deployment. We further discuss knowledge graph completion and human involvement in Appendix B. Using ThinkEval, we develop the KnowGIC (Knowledge Graph Implication Chains) benchmark dataset to evaluate model-editing techniques under the deep editing setting. Unlike prior benchmarks that rely on single-query evaluations, or condense multi-hop reasoning into a single complex query, KnowGIC decomposes reasoning into multi-step implication chains. Each chain is a sequence of linked queries, where the answer to one step becomes the premise for the next. This design directly tests whether an “edited-out” fact can still be reconstructed through chained reasoning. KnowGIC consists of 1,406 samples, draws from two sources: (1) diverse samples from the categories in the MQuAKE dataset (Zhong et al. (2023)), ensuring broad relational coverage, and (2) a targeted sample: (Harry Potter, school, Hogwarts), selected to demonstrate the complexity of deeply embedded causal relationships in LLMs. As illustrated in Fig. 1, even when direct queries for the target fact fail, multi-step reasoning can reconstruct the original fact. We evaluate five parameter-modifying model-editing techniques: AlphaEdit (Fang et al. (2025)), RECT (Gu et al. (2024)), ROME (Meng et al. (2022)), MEMIT (Meng et al. (2023)), and PRUNE (Ma et al. (2025)) on Qwen2.5-7B-Instruct (Team (2024)), Meta-Llama-3-8B-Instruct (Grattafiori et al. (2024)) and GPT2- XL (1.5B) (Radford et al. (2019)) using IFR and Preservation. Our results reveal notable shortcomings in existing techniques that undermine model reliability, highlighting the need for advanced methods to update knowledge holistically while preserving broader contextual integrity. Contributions. (I) We develop ThinkEval for systematic evaluation of model-editing techniques, us- ing CoT reasoning to build specialised knowledge graphs to reveal implicit fact associations. (I) Within ThinkEval, we introduce deep editing, a new evaluation setting that quantifies indirect knowledge leakage through multi-step reasoning and measures ripple-effect propagation across broader contextual knowledge. We propose a new metric Indirect Fact Recovery (IFR) tailored to this setting. (I) Using ThinkEval, we construct the KnowGIC benchmark dataset of 1,406 multi-step sequential-prompting samples for deep edit- ing evaluation. (IV) We systematically evaluate five model-editing techniques across three LLMs, revealing trade-offs between indirect knowledge leakage and contextual integrity. 2 Related Work Model-editing techniques. Recent works classify model-editing techniques into two broad categories: ❶ parameter-modifying and ❷ parameter-preserving techniques (Fang et al. (2025)). Parameter-modifying methods, like fine-tuning, meta-learning (e.g., MEND (Mitchell et al. (2021))), and locate-then-edit (e.g., ROME (Meng et al. (2022)), MEMIT (Meng et al. (2023)), AlphaEdit (Fang et al. (2025))), directly alter model weights. Fine-tuning often causes catastrophic forgetting (Hsueh et al. (2024)), while meta-learning mitigates it but scales poorly (Hsueh et al. (2024)). Locate-then-edit methods target specific knowledge, with AlphaEdit reducing interference via null-space projections (Fang et al. (2025)), yet sequential edits still lead to gradual forgetting (Gupta et al. (2024)). Parameter-preserving methods, such as retrieval- augmented (e.g., SERAC (Mitchell et al. (2022)), IKE (Zheng et al. (2023))) and memory-based approaches (e.g., T-Patcher (Huang et al. (2023)), GRACE (Hartvigsen et al. (2023))), use external modules. They better retain performance and reduce forgetting. T-Patcher adds neurons per mistake (Yao et al. (2024b)), and GRACE uses key-value codebooks (Hartvigsen et al. (2023)). But these techniques face scalability, generalization, and computational challenges (Zhang et al. (2024b); Hsueh et al. (2024); Lin et al. (2024b)). GLAME (Zhang et al. (2024b)) uses knowledge graphs for multi-hop reasoning via graph augmentation and graph-based edit modules. However, it overlooks model-specific internal associations, missing latent paths (e.g., deducing Hogwarts via Slytherin), and relies on external knowledge, incurring more computational costs. RippleCOT (Zhao et al. (2024)) utilises In-Context Learning with CoT reasoning to incorporate a thought component to decompose the multi-hop logic within questions. However, CoT reasoning integration increases computational complexity during inference. ChainEdit (Dong et al. (2025)) dynamically generates and edits logically connected knowledge clusters to improve consistency, but it relies on pre-defined graph- derived rules and doesn’t discuss indirect fact reconstruction. Model-editing evaluations. KnowEdit (Zhang et al. (2024c)), LEME (Rosati et al. (2024)), Counter- Fact (Meng et al. (2022)), and CounterFactPlus (Hoelscher-Obermaier et al. (2023)) focus on direct fact 3 edits but overlook related knowledge impacts and original fact deducibility. KnowEdit and LEME assess specific fact changes, while CounterFact and CounterFactPlus test adaptability to counterfactuals, over- looking chained reasoning effects. MQuAKE (Zhong et al. (2023)) incorporates multi-hop query answering. Though valuable for probing broader knowledge, it does not test sequential recovery of facts, which is how users may naturally query LLMs. These evaluations primarily evaluate direct fact accuracy and multi-hop reasoning, overlooking implicit knowledge deducibility or contextual integrity. UnKEBench evaluates un- structured knowledge editing, where knowledge is represented in complex, free-form text (Deng et al. (2025)). However, it doesn’t consider sequential reasoning to uncover "edited-out" facts. While prior works address ripple effects (Cohen et al. (2024); Qin et al. (2024); Chen et al. (2025); Wang et al. (2024)), event-level con- sistency (Peng et al. (2024)), concept-level and instance-level consistency (Niu et al. (2025)), none evaluate whether edited facts can still be recovered through multi-step reasoning. ThinkEval fills this gap through the deep editing setting and the IFR metric. 3 Preliminary A fact k is represented as a triplet (v 1 , r, v 2 ), where v 1 , v 2 ∈V are entities (subject and object, respectively) and r ∈ T is a relationship connecting them. For example, (Harry Potter, school, Hogwarts) can rep- resent the statement “Harry Potter studied at Hogwarts”. A knowledge graph G = (V,E ) is a directed graph where E ⊆V ×T ×V is a set of edges representing facts that connect entities in V. 3.1 Graph-based reasoning Unlike rule-based systems (Wu et al. (2024)), the relationships in G are not derived from a predefined set of logical rules. G is designed such that traversing its edges naturally implies relationships between entities. We do not assume transitive closure over G; instead, each path in G is retained only when it plausibly implies a meaningful relationship between the base subject and object. A path p = (v 1 , r 1 , u 1 , r 2 , u 2 , . . . , u n−1 , r n , v 2 ) in G with u i ∈V and r i ∈T captures a chained relationship r between the endpoints, even if (v 1 , r, v 2 ) /∈E. For example, path (Harry Potter, house, Gryffindor, belongs to, Hogwarts) implies Harry Potter studied at Hogwarts. 3.2 Knowledge graphs for model editing After editing an LLM, the internal knowledge graph G = (V,E ) is updated with a set of changes ∆G. The updated graph G ′ = (V ′ ,E ′ ) incorporates: Adding new edges:∆G a =(v 1 , r, v 2 )| (v 1 , r, v 2 ) /∈E(1) Removing existing edges:∆G r =(v 1 , r, v 2 )| (v 1 , r, v 2 )∈E(2) Modifying edges:∆G m =(v 1 , r new , v 2 )| (v 1 , r, v 2 )∈E(3) The updated edge set is: E ′ = (E ∪ ∆G a ∪ ∆G m )\ ∆G r (4) and V ′ =V ∪v | v ∈ (v 1 , r, v 2 )∈ ∆G a ∪ ∆G m (5) For this study, we limit our experimentation to analyze modifying edges (∆G m ) in LLMs. Definition 1 Deductive Closure: Given a knowledge graph G = (V,E ) with E ⊆V ×T ×V as the set of triplets, the deductive closure of G, denoted Ω(G), is the set of all relationships implied by G such that: 1. E ⊆ Ω(G), i.e., every triplet (v 1 , r, v 2 )∈E is a direct relationship r between v 1 and v 2 ; 4 Original Model Edited Model Base Triplet (s, r, o) Base Triplet (s, r, o) Query Generation LLM Response Generation Knowledge Validation Knowledge Graph Synthesis Human-assisted Query Refinement Dataset (s₁, r₁, s₂, r₂, ..., sₙ) (q₁, q₂, ..., qₙ) (s', r', o') Triplets Extraction Facts Segmentation Chain-of-Thought Answering Human-assisted Triplet Pruning and Addition Connected Knowledge Preservation Indirect Fact Recovery Dataset Chain Sequencing Figure 2: ThinkEval framework. Initially,1 a base triplet and2 an LLM are utilized to3 generate a tailored dataset reflective of the LLM’s internal knowledge structure. Next, 4 the LLM is edited using an editing technique. The5 the edited model is evaluated over the constructed dataset, yielding insights into the effectiveness of the editing process from deep editing perspective,6 measured via IFR and Preservation. 2. Ω(G) is deductively closed with respect to path traversal, i.e., for every pair of entities v 1 , v n ∈ V, if there exists a path p = (v 1 , r 1 , v 2 , r 2 , . . . , r n−1 , v n ) in G where each (v i , r i , v i+1 ) ∈ E, then Ω(G) includes the implied relationship (v 1 , r p , v n ), where r p is the composite relationship inferred from the sequence of relationships (r 1 , r 2 , . . . , r n−1 ). Ω(G) captures all explicit and implicit relationships derivable from G through its paths. 4 ThinkEval We introduce ThinkEval, a framework designed to assist users by constructing specialised knowledge graphs to quantitatively and systematically evaluate model-editing techniques via sequential prompting, as illus- trated in Fig. 2. ThinkEval operates through three interwoven components to create the dataset: 1 Query Validation and Refinement,2 Automated Triplet Generation, and3 Graph Synthesis and Chain Sequencing. These components function in a cyclical manner, iteratively refining the knowledge representa- tion. All the respective prompt templates are provided in Appendix E. Additionally, ThinkEval serves as a dataset augmentation tool, capable of enhancing existing datasets by adding structured implication chains or triplets for a more comprehensive evaluation. Algorithms 1, 2 and 3 showcase the key processes utilised in ThinkEval. For users, choosing the right model-editing technique is vital for preventing knowledge leakage and ripple effects, thereby safeguarding reliability across diverse applications. 4.1 Query validation and refinement The Query Validation and Refinement component initiates knowledge extraction by validating a triplet (s, r, o) or a chain (s 1 , r 1 , s 2 , r 2 , . . . , s n ), evaluating if the LLM "recognizes" the relationship. The process begins by generating targeted queries from a triplet or a chain. ThinkEval synthesizes a knowledge graph beginning with a single triplet, and employs two prompt types: Triplet-based Query Generation prompt. Targets a single triplet to elicit a direct response. For example, (Harry Potter, student, Hogwarts), forms the query “Where did Harry Potter study?”. 5 Chain-based Query Generation prompt. Probes multi-hop relationships in a chain, targeting the terminal entity, For instance, from (Harry Potter, subject, Transfiguration, taught by, Minerva McGonagall), “Who teaches Transfiguration to Harry Potter?” is crafted. The LLM generates a response to the query, which is analyzed to determine the expected object’s presence, accounting for phrasing variations or synonyms. For triplets, if the object aligns, the triplet and query are passed to the second and third components; for chains, they proceed only to the second component. If validation fails, human expertise refines the query or triplet, re-entering the cycle. After several failed cycles, indicating the model’s lack of knowledge, the triplet is discarded. 4.2 Automated triplet generation The Automated Triplet Generation component extracts further knowledge from the LLM using its reasoning capabilities, structuring the output into new triplets. These are cycled back to the previous component for validation. This component’s functionality is demonstrated with an illustrative example in Appendix F. The validated query is submitted to the LLM with a CoT prompt, prompting step-by-step reasoning to reveal intermediate associations alongside the final answer. The multi-sentence CoT response is segmented into discrete facts, parsing it into individual statements for triplet extraction. This segmentation simplifies the response into manageable units to preserve extracted information’s integrity. Each fact is processed by the LLM with a fact extraction prompt to generate atomic triplets, like (Harry Potter, school, Hogwarts) or (Transfiguration, taught by, Minerva McGonagall). To handle er- rors from hallucination or context misinterpretation, human experts prune triplets, verifying them against known facts. Authentic triplets are cycled back to the prior component, while human expertise also adds missing triplets to account for the facts overlooked by the LLM. 4.3 Graph synthesis and chain sequencing The Graph Synthesis and Chain Sequencing component integrates validated triplets into a knowledge graph and structures sequences of query for inferring relationships for the dataset. The constructed chain of triplets (s 1 , r 1 , s 2 , r 2 , . . . , s n ) is cycled back to the first component. The validated triplets are organized into the directed knowledge graph G. G evolves with each iteration, incorporating new triplets, reflecting the growing complexity of the extracted knowledge. G is processed to form n-step chains (s 1 , r 1 , s 2 , r 2 , . . . , s n ), to serve two purposes: (1) chains are sent to first component for multi-hop query generation, and (2) their query sequences are compiled into a dataset. These are capped at five steps, as longer chains tend to diminish the impact of associating the subject with the object. The effectiveness of extracting meaningful relationships decreases with increasing number of queries, making shorter chains more impactful for establishing clear associations. 5 Deep Editing Existing model editing techniques often modify a target fact in isolation, often overlooking logically connected facts and thereby introducing inconsistencies. We introduce deep editing as a new evaluation setting that characterizes whether an edit has been performed consistently with respect to related knowledge. Unlike event-level editing (Deng et al. (2025)), which focuses on updating facts tied to a specific event, deep editing evaluates whether edits propagate appropriately through a network of logically connected facts, while preserving unrelated knowledge. A fact is deep edited up to n-steps if it cannot be empirically deduced from the retained knowledge through any sequence of reasoning steps of length≤ n, where n is a user-defined parameter reflecting computational or application constraints. We provide a comparison of deep editing with other model-editing settings in Appendix C. Suppose an editing technique A modifies a target fact t = (v 1 , r, v 2 ) ∈ E, resulting in modified triplets E A t ⊆ E to form an updated graph G ′ = (V ′ ,E ′ ). Here, E ′ = (E A t )∪ ∆E A t , where ∆E A t = (v i , r ′ i , v j ) | 6 Algorithm 1 Query Validation and Refinement 1: Input: Knowledge input I (either triplet t = (s, r, o) or chain c = (s 1 , r 1 , s 2 , r 2 , . . . , s n )), LLM M, max iterations k max 2: Output: Validated triplet t ′ or chain c ′ , or ∅ (discarded) 3: q ← GenerateQuery(I)▷ Triplet or chain-based prompt 4: k ← 0 5: while k < k max do 6: R←M(q)▷ LLM response 7: if o∈R then▷ Object present in response 8:if I is chain c then 9:c ′ ← c 10:cycle q to Algorithm 3 11:else 12:t ′ ← t 13:end if 14:return q to Algorithm 2 15: else 16:I, q ← HumanRefine(q,R,I)▷ Refine based on input type 17:k ← k + 1 18: end if 19: end while 20: discard t▷ No validation after k max 21: return ∅ Algorithm 2 Automated Triplet Generation 1: Input: Validated query q, LLM M 2: Output: Set of new triplets T new 3: R←M(q, CoT)▷ CoT response 4: F ← SegmentFacts(R)▷ Set of facts 5: T temp ←∅ 6: for f ∈F do 7: T f ←M(f, ExtractPrompt)▷ Extract triplets 8: T temp ←T temp ∪T f 9: end for 10: T new ← HumanPruneAndAdd(T temp )▷ Prune and augment 11: return T new to Algorithm 1 Algorithm 3 Graph Synthesis and Chain Sequencing 1: Input: Validated triplet t = (s, r, o), corresponding validated query q 2: Output: Knowledge graph G, dataset D 3: V ←V ∪s, o 4: E ←E ∪(s, r, o) 5: G ← (V,E )▷ Initialize graph 6: C ← GenerateChains(G)▷ Chains from G 7: D ←∅▷ Initialize dataset 8: for c = (s 1 , r 1 , s 2 , r 2 , . . . , s n )∈C do 9: Q c ← GetQuerySequence(c, l max )▷ Query sequence with maximum length as l max 10: D ←D∪Q c 11: cycle c to Algorithm 1 12: end for 13: return G,D 7 (v i , r i , v j )∈E A t represents the modified triplets in E A t , with the new relationships r ′ i ∈T . If technique A deep edits the fact t, then t must not be implied by any path in G ′ , i.e., t should not belong to Ω(G ′ ). Definition 2 Deep Editing: An editing technique A deep edits the fact t = (v 1 , r, v 2 ) with respect to the knowledge graph G = (V,E ) if t does not belong to the deductive closure Ω(G ′ ), where G ′ = (V ′ ,E ′ ) is the updated graph, and E ′ = (E A t )∪ ∆E A t reflects the modified triplets. Formally, (v 1 , r, v 2 ) /∈ Ω(G ′ ), i.e., no path in G ′ implies the original relationship r between v 1 and v 2 . 5.1 Evaluation metrics for deep editing We outline the Counterfact (Meng et al. (2022)) metrics to evaluate model editing in Appendix D. For deep editing evaluation, we propose a new evaluation metric Indirect Fact Recovery (IFR). IFR measures the extent to which the original fact remains deducible despite the edit. It is implied by paths in G = (V,E ) after editing to G ′ = (V ′ ,E ′ ). Let D = S 1 , . . . ,S m be sequences of queries corresponding to chains from G, where S i = q i1 , . . . , q in i has length n i , and each q ij probes a step in a path implying a target fact. Let P G = P S 1 , . . . ,P S m and P G ′ = P ′ S 1 , . . . ,P ′ S m , where P S i = p i1 , . . . , p in i and P ′ S i = p ′ i1 , . . . , p ′ in i are the probabilities of the original target output occurring in the model response to each q ij in G and G ′ , respectively. Definition 3 Indirect Fact Recovery: IFR(G,G ′ ,D, P G , P G ′ ) =                X S i ∈D ′ R ′ S i /R S i √ n i X S i ∈D ′ 1 √ n i , if D ′ ̸=∅, 0,otherwise, (6) where R S i = Q n i j=1 p ij , R ′ S i = Q n i j=1 p ′ ij , and D ′ =S i ∈D |R S i ̸= 0. IFR computes the weighted average of retention ratiosR ′ S i /R S i , normalized by √ n i , across all the sequences to emphasize the higher implication strength of the shorter paths. Here, p ij and p ′ ij reflect probability of the original target output (object in a triplet) in the model’s response to q ij before and after editing. A high IFR indicates significant deducibility of the original target fact, suggesting incomplete deep editing. A low IFR reflects that original fact is hardly deducible after editing. We use Preservation (Cohen et al. (2024)) to evaluate the the broader contextual knowledge integrity in deep editing. While RippleEdits utilises it to evaluate other subject-relationships, we utilise it to measure retention of accuracy across facts in G = (V,E ) after editing to G ′ = (V ′ ,E ′ ), excluding those which are not a part of the broader contextual knowledge (links directly linked to the original subject unless specified otherwise). Let t 0 = (s 0 , r 0 , o 0 ) be the original triplet, andE ind =(s, r, o)∈E | s̸= s 0 as the set of triplets where the subject is not s 0 . For each t = (s, r, o)∈E ind , let p t ∈ [0, 1] and p ′ t ∈ [0, 1] be probabilities of the original target output o occurring in the model’s response to a query q t in G and G ′ , respectively. Definition 4Preservation: Preservation(G,G ′ ,E ind ) =      1 |E ′ ind | X t∈E ′ ind p ′ t p t , if E ′ ind ̸=∅, 1,otherwise, (7) where E ′ ind =t∈E ind | p t ̸= 0. Preservation averages the ratios p ′ t /p t over all triplets in E ′ ind , where p t and p ′ t represent probabilities of the original object o in response to q t before and after editing. A high Preservation indicates strong preserva- tion of broader contextual knowledge, suggesting that the editing process successfully avoided catastrophic 8 forgetting over them. A low Preservation reflects a loss of accuracy in these facts, implying unintended knowledge degradation within the model. 6 Performance Evaluation Using the KnowGIC benchmark, we evaluated five model-editing techniques: AlphaEdit, RECT, ROME, MEMIT and PRUNE via IFR, Preservation and Efficacy on three models, Qwen2.5 (7B), Llama3 (8B) and GPT2-XL (1.5B). All implementations are sourced from Jiang (2024). All experiments are performed on an NVIDIA A100 80GB GPU. 6.1 KnowGIC Benchmark To evaluate in the deep editing setting, we introduce the KnowGIC benchmark, which is built using ThinkEval. Extending the multi-hop reasoning datasets, KnowGIC consists of n-step implication chains, or sequences of n prompts to probe specific relationships via multiple reasoning steps. These chains capture direct and implied relationships for a fine-grained evaluation. Unlike MQuAKE’s n-hop queries, which en- capsulate multiple reasoning hops in a single query, each implication chain step probes a single hop in the knowledge graph. If the original fact can be logically reconstructed through such sequential prompting, it signals incomplete editing. Conversely, breaks in the chain that disrupt unrelated but connected facts reveal unintended ripple effects. Table 1 presents the distribution of chain lengths in the dataset. All chains have been manually reviewed to remove redundancy, factual inaccuracies, and irrelevant triplets, ensuring high quality and relevance for deep editing evaluation. Table 1: Distribution of multi-step implication chains in the KnowGIC benchmark. Chain length (steps) 12345 Number of Chains 24 108 227 428 619 We selected various base templates from the MQuAKE dataset across diverse relational domains, generating knowledge graphs using ThinkEval for their samples. To illustrate the depth of connected facts and how interlinked facts can help extract the original relationship, we include an additional sample (Harry Potter, school, Hogwarts). Using ThinkEval, we grow its knowledge graph till it achieves 100 implication chains from Harry Potter to Hogwarts. The KnowGIC benchmark, including this sample, comprises 1,406 n-step reasoning paths. Extensive details of the dataset are in Appendix G. 6.2 Motivation for new metrics Existing metrics like Efficacy and Specificity provide valuable insights into different aspects of model editing, but are insufficient for deep editing evaluation. Efficacy focuses on immediate effect of the edit, overlooking whether the original fact can be inferred via sequential reasoning. Specificity focuses on the correctness of the LLM’s response in terms of higher probability assigned to the correct response. Similarly, Consistency focuses only on factual coherence. Deep editing quantifies how much the original fact can be deduced even via sequential reasoning. Hence, a new metric is essential to evaluate this dimension comprehensively. Table 2: Harry Potter case-study setup and results. Relationship triplet(Harry Potter, school, Hogwarts) Edit requestHogwarts→ Ilvermorny ModelLlama3-8B-Instruct Edit techniqueAlphaEdit Paths identified100 Active paths post edit 80 Efficacy (↑)1.000 IFR (↓)0.780 9 0.2 Harry Potter Where did Harry Potter study? Hermione Granger Ron Weasley Professor McGonagall Gryffindor Who were Harry Potter's schoolmates? Who were Hermione Granger's schoolmates? Who teaches Transfiguration to Ron Weasley? Which house is Professor McGonagall the head of? Which school does Gryffindor belong to? Dean Thomas Professor Snape Slytherin 1.01.00.8 0.8 1.0 Who were Harry Potter's schoolmates? 1.0 Who taught Potions to Hermione Granger? 0.6 Which house is Professor Snape the head of? 1.0 Which school does Slytherin belong to? 1.0 Gryffindor Hermione Granger Professor McGonagall Who were Harry Potter's schoolmates? 0.4 Who teaches Transfiguration to Harry Potter? 1.0 Which school is Professor McGonagall the headmistress of? 1.0 Which house is Dean Thomas in? 1.0 Which school does Gryffindor belong to? 1.0 Hogwarts Figure 3: Samples of n-step chains from Harry Potter case-study, leading to original fact leakage even after editing. The number below each link represents the ratio of responses (out of five generations) that retain the original output, quantifying the extent to which the LLM reveals the initial fact via indirect reasoning. We conduct a case-study on the sample (Harry Potter, school, Hogwarts) from KnowGIC to evaluate the need for new metrics in deep editing, aiming to edit the fact (Hogwarts→ Ilvermorny). The setup is detailed in Table 2. For each query in the implication chains, we generate five distinct responses from both the original and edited models. An Efficacy of 1.000 shows that the LLM responds with Ilvermorny when queried directly. However, it does not account for original fact recovery through indirect reasoning paths. An overall IFR of 0.780 suggests that the original fact still remains deducible to a significant extent. Figure 3 shows samples from these 80 active chains. Table 3: n-step IFR for the Harry Potter case-study. n-step chains Active chains post editn-step IFR 1110.200 220140.814 330230.736 435300.484 514120.402 As shown in Table 3, the 1-step chain with IFR of 0.200 shows effective suppression of direct inference. However, the 2-step and 3-step chains exhibit high IFR values of 0.814 and 0.736 respectively, demonstrating high indirect fact leakage through short reasoning paths. The 4-step and 5-step chains show moderate IFR values of 0.484 and 0.402, further demonstrating persistent leakage across longer paths. Despite an Efficacy of 1.00, the original fact persists through multi-step reasoning, indicating AlphaEdit’s surface-level success but failure in deeper knowledge updates. For an editing method to qualify as successful under the deep editing paradigm, the original fact must be eliminated across all reasoning paths. This case-study also highlights that traditional metrics like Efficacy are insufficient for deep editing. IFR’s focus on multi-step chains provides a more robust evaluation. Further details of the case-study are in Appendix H. 6.3 Results and discussions Table 4 presents the overall IFR, Preservation and Efficacy values for different editing techniques across GPT2-XL, Llama3-8B, and Qwen2.5-7B. For Llama3-8B, AlphaEdit achieves the highest Efficacy of 0.958, followed by PRUNE at 0.875, indicating strong performance in updating target facts. For Qwen2.5-7B, AlphaEdit achieves a perfect Efficacy of 1.000, with MEMIT also performing strongly at 0.958. For GPT2- 10 Table 4: Comparison of model-editing techniques for various models using IFR, Preservation (Pres.), and Efficacy (Eff.). Values are color-coded from red (lower performance) to green (higher performance). Model Technique IFR (↓) Pres. (↑)Eff. (↑) Overall 1-step 2-step 3-step 4-step 5-step Llama3-8B AlphaEdit0.5090.4130.5570.6410.6260.3510.8900.958 MEMIT 0.5500.4340.6280.6340.5990.4610.8570.792 RECT0.4870.4540.4650.5520.5640.4060.8230.708 PRUNE0.1870.1780.1870.1950.1950.1780.6960.875 ROME 0.1340.1730.1160.2220.1770.0600.6850.833 Qwen2.5-7B AlphaEdit0.4960.1080.2690.5630.5790.4970.8691.000 MEMIT 0.5770.0970.2870.5350.5490.7410.8590.958 RECT0.5690.3040.5610.5990.6790.4940.8510.750 PRUNE 0.5140.3780.4270.5210.4960.5610.7650.792 ROME0.3320.1770.2120.1730.3810.4170.6690.708 GPT2-XL AlphaEdit0.4610.1990.5470.6390.5650.2940.6631.000 MEMIT 0.4840.2710.5180.6540.5490.3620.6830.875 RECT0.5160.4820.5390.6730.5480.4130.6890.708 PRUNE 0.2110.2380.2080.2090.2190.2030.5870.917 ROME 0.1960.2990.2370.2630.2780.1580.5000.792 1-step2-step3-step4-step5-step n-Step Implication Chains 0.0 0.2 0.4 0.6 0.8 1.0 IFR ( ) Techniques AlphaEdit ROME MEMIT PRUNE RECT (a) Llama3-8B 1-step2-step3-step4-step5-step n-Step Implication Chains 0.0 0.2 0.4 0.6 0.8 1.0 IFR ( ) Techniques AlphaEdit ROME MEMIT PRUNE RECT (b) Qwen2.5-7B 1-step2-step3-step4-step5-step n-Step Implication Chains 0.0 0.2 0.4 0.6 0.8 1.0 IFR ( ) Techniques AlphaEdit ROME MEMIT PRUNE RECT (c) GPT2-XL Figure 4: IFR for n-step samples. A lower IFR implies lower deducibility of the supposedly-edited fact. XL, both AlphaEdit and PRUNE achieve high Efficacy scores of 1.000 and 0.917, respectively. Efficacy, however, provides limited insights into the performance of these techniques by focusing on direct edit success while neglecting the broader implications of these edits. In contrast, IFR scores reveal significant variability across models, as shown in Fig. 4. For Llama3-8B, AlphaEdit exhibits an IFR increase from 0.413 (1-step) to 0.641 (3-step) before declining to 0.351 (5-step), indicating that the original fact remains highly deducible, particularly via mid-range chains (3-step). On Qwen2.5-7B, AlphaEdit’s IFR rises steadily from 0.108 (1-step) to 0.497 (5-step), showing persistent fact leakage. For GPT2-XL, AlphaEdit’s IFR increases from 0.199 (1-step) to 0.639 (3-step) before dropping to 0.294 (5-step), following a similar rise-then-fall pattern. MEMIT on Qwen2.5-7B shows a sharp IFR increase from 0.097 (1-step) to 0.741 (5-step), while on Llama3-8B, it follows a comparable rise-then-fall trend (0.434 at 1-step, 0.634 at 3-step, 0.461 at 5-step). For GPT2-XL, MEMIT’s IFR rises from 0.271 (1-step) to 0.654 (3-step) before declining to 0.362 (5-step), highlighting contrasting leakage patterns. ROME and PRUNE maintain low IFR across all models, with ROME on Llama3-8B dropping to 0.060 at 5-step, and on GPT2- XL to 0.158, indicating strong reduction in original fact leakage. However, this may suggest ripple effects in related knowledge. These techniques are typically designed to edit direct facts (1-step chains), expecting their impact to diminish as chain length increases, ideally resulting in an increasing IFR curve. Most techniques on Qwen2.5-7B align with this expectation, but deviate on other models, showing a peak before declining. The rise-then-fall IFR pattern in AlphaEdit, MEMIT, and RECT on Llama3-8B and GPT2-XL is attributed to the interplay between their editing strategies and ripple effects. As IFR relies on the product of link strengths in an implication chain, longer chains (4-5 steps) are more likely to encounter weakened links, reducing deducibility due to eroded related knowledge. These techniques disrupt broader contextual con- nections, causing a decline in IFR beyond mid-range chains. In contrast, ROME and PRUNE show lower 11 AlphaEditMEMITRECTPRUNEROME Technique 0.0 0.2 0.4 0.6 0.8 1.0 Preservation ( ) Models GPT2-XL Llama3-8B Qwen2.5-8B Figure 5: Preservation for different model-editing techniques. A higher Preservation indicates stronger retention of broader context integrity. 0.40.50.60.70.80.91.0 Preservation () 0.0 0.1 0.2 0.3 0.4 0.5 0.6 IFR ( ) Techniques AlphaEdit ROME MEMIT PRUNE RECT (a) Llama3-8B 0.40.50.60.70.80.91.0 Preservation () 0.0 0.1 0.2 0.3 0.4 0.5 0.6 IFR ( ) Techniques AlphaEdit ROME MEMIT PRUNE RECT (b) Qwen2.5-7B 0.40.50.60.70.80.91.0 Preservation () 0.0 0.1 0.2 0.3 0.4 0.5 0.6 IFR ( ) Techniques AlphaEdit ROME MEMIT PRUNE RECT (c) GPT2-XL Figure 6: IFR vs. Preservation for different model-editing techniques, illustrating trade-offs in indirect fact recovery and related knowledge preservation. IFR across all models, risking over-editing, as reflected by their Preservation scores. As illustrated in Fig. 5, ROME’s Preservation is relatively low (0.685 on Llama3-8B, 0.669 on Qwen2.5-7B, 0.500 on GPT2-XL), indi- cating poor preservation of related facts, while AlphaEdit achieves higher Preservation (0.890 on Llama3-8B, 0.869 on Qwen2.5-7B, 0.663 on GPT2-XL). GPT2-XL models show low preservation across all techniques due to high entanglement of facts stemming from its smaller size (Qin et al. (2024)). For Qwen2.5-7B, the predominantly increasing IFR curves, such as AlphaEdit’s rise from 0.108 (1-step) to 0.497 (5-step), prompt additional investigations to explore its robustness over extended implication chains, detailed in Appendix A. Fig. 6 highlights the varying performance of these techniques. For users, IFR delivers critical insights into the subtle impacts of edits across implication chains, revealing strengths and weaknesses missed by existing metrics and underscoring the importance of systematic evaluation frameworks such as ThinkEval. 7 Conclusion In this study, we discuss the challenge of maintaining consistent knowledge in LLMs under model editing. We introduced ThinkEval, a framework which offers users involved in updating and deploying a model a systematic evaluation of editing techniques. Within this framework, we defined the deep editing evaluation setting to quantify the original knowledge leakage and ripple effects caused by edits, and proposed Indirect Fact Recovery (IFR) as a metric to quantify such leakage, thus assisting these users in choosing the suitable model and the right editing technique for their use case. For evaluation, we constructed the KnowGIC benchmark, a dataset of multi-step reasoning paths. Our experiments expose significant limitations in five state-of-the-art editing techniques across three models. While these techniques effectively deal with directly queried facts, they frequently fail to eliminate indirect knowledge leakage, allowing the original information to persist. Additionally, edits often introduce unintended ripple effects that disrupt the coherence of surrounding contextual knowledge. These findings highlight the need for evaluation frameworks like ThinkEval to guide users in selecting editing methods and for the development of more holistic techniques that update knowledge reliably without compromising contextual integrity. 12 References Qizhou Chen, Dakan Wang, Taolin Zhang, Zaoming Yan, Chengsong You, Chengyu Wang, and Xi- aofeng He. Uniedit: A unified knowledge editing benchmark for large language models. arXiv preprint arXiv:2505.12345, 2025. Inyoung Cheong, King Xia, KJ Kevin Feng, Quan Ze Chen, and Amy X Zhang. (a) i am not a lawyer, but...: engaging legal experts towards responsible llm policies for legal advice. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, p. 2454–2469, 2024. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowl- edge editing in language models. Transactions of the Association for Computational Linguistics, 12: 283–298, 2024. Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. Everything is editable: Extend knowledge editing to unstructured data in large language models. In Proceedings of the International Conference on Learning Representations (ICLR 2025), 2025. URL https://openreview. net/pdf?id=X5rO5VyTgB. Dinil Mon Divakaran and Sai Teja Peddinti. Large language models for cybersecurity: New opportunities. IEEE Security & Privacy, 2024. Zilu Dong, Xiangqing Shen, Zinong Yang, and Rui Xia. ChainEdit: Propagating ripple effects in LLM knowl- edge editing through logical rule-guided chains. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13558–13571, Vienna, Austria, July 2025. As- sociation for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.665. URL https://aclanthology.org/2025.acl-long.665/. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat- Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. In Proceedings of the International Conference on Learning Representations, 2025, 2025. https://openreview.net/ attachment?id=HvSytvg3Jh&name=pdf, accessed 2025-12-06. Jianqi Gao, Hao Wu, Yiu-ming Cheung, Jian Cao, Hang Yu, and Yonggang Zhang. Mitigating forgetting in adapting pre-trained language models to text processing tasks via consistency alignment. In Proceedings of the ACM on Web Conference 2025, p. 3492–3504, 2025a. Jianqi Gao, Hang Yu, Yiu-ming Cheung, Jian Cao, Raymond Chi-Wing Wong, and Yonggang Zhang. Shaping pre-trained language models for task-specific embedding generation via consistency calibration. Neural Networks, p. 107754, 2025b. Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward. Healai: A healthcare llm for effective medical documentation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, p. 1167–1168, 2024. Sagar Goyal, Eti Rastogi, Fen Zhao, Dong Yuan, and Andrew Beinstein. Specialtyscribe: Enhancing soap note scribing for medical specialties using llm’s. In Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health), p. 34–45, 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regularization to the rescue. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 16801–16819, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.934. URL https://aclanthology.org/ 2024.emnlp-main.934/. 13 Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. arXiv preprint arXiv:2401.07453, 2024. Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36:47934–47959, 2023. Peter Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. Fundamental problems with model editing: How should rational belief revision work in llms? arXiv preprint arXiv:2406.19354, 2024. Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, p. 11548–11559, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.733. URL https://aclanthology.org/2023.findings-acl.733/. Cheng-Hsun Hsueh, Paul Kuo-Ming Huang, Tzu-Han Lin, Che-Wei Liao, Hung-Chieh Fang, Chao-Wei Huang, and Yun-Nung Chen. Editing the mind of giants: An in-depth exploration of pitfalls of knowledge editing in large language models. arXiv preprint arXiv:2406.01436, 2024. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher: One mistake worth one neuron. In Proceedings of the International Conference on Learning Representations (ICLR 2023), 2023. URL https://openreview.net/pdf?id=4oYUGeGBPm. Houcheng Jiang. Alphaedit: Null-space constrained knowledge editing for language models, 2024. URL https://github.com/jianghoucheng/AlphaEdit/. GitHub repository. Lena John, Tim Wittenborg, Sören Auer, and Oliver Karras. Human-in-the-loop workflow for neuro-symbolic scholarly knowledge organization. arXiv preprint arXiv:2506.03221, 2025. Prashank Kadam. Enhancing financial fraud detection with human-in-the-loop feedback and feedback prop- agation. arXiv preprint arXiv:2411.05859, 2024. Luyang Lin, Lingzhi Wang, Jinsong Guo, and Kam-Fai Wong. Investigating bias in llm-based bias detection: Disparities between llms and human perception. arXiv preprint arXiv:2403.14896, 2024a. Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wenpeng Yin, and Lifu Huang. Navigating the dual facets: A comprehensive evaluation of sequential memory editing in large language models. arXiv preprint arXiv:2402.11122, 2024b. Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. Perturbation-restrained sequen- tial model editing. In Proceedings of the International Conference on Learning Representations (ICLR 2025), 2025. URL https://openreview.net/pdf?id=bfI8cp8qmk. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022. Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In Proceedings of the International Conference on Learning Representations, 2023, 2023. Available at urlhttps://iclr.c/virtual/2023/oral/12726. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2110.11309, 2021. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning, p. 15817–15831. PMLR, 2022. 14 Yifan Niu, Miao Peng, Nuo Chen, Yatao Bian, Tingyang Xu, and Jia Li. Reledit: Evaluating concep- tual knowledge editing in language models via relational reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, p. 10220–10238, 2025. Ollama Team. Ollama: Run large language models locally. https://ollama.com, 2023. Accessed: 2025-04- 29. Ciyuan Peng, Feng Xia, Mehdi Naseriparsa, and Francesco Osborne. Knowledge graphs: Opportunities and challenges. Artificial Intelligence Review, 56(11):13071–13102, 2023. Hao Peng, Xiaozhi Wang, Chunyang Li, Kaisheng Zeng, Jiangshan Duo, Yixin Cao, Lei Hou, and Juanzi Li. Event-level knowledge editing. arXiv preprint arXiv:2402.13093, 2024. Jiaxin Qin, Zixuan Zhang, Chi Han, Pengfei Yu, Manling Li, and Heng Ji. Why does new knowledge create messy ripple effects in LLMs? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 12602– 12609, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1/2024.emnlp-main.700. URL https://aclanthology.org/2024.emnlp-main.700/. Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6(12):1418– 1420, 2024. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Yahya Kayani, Frank Rudzicz, and Hassan Sajjad. Long-form evaluation of model editing. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 3749–3780, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.208. URL https: //aclanthology.org/2024.naacl-long.208/. Dorsaf Sallami, Yuan-Chen Chang, and Esma Aïmeur. From deception to detection: The dual roles of large language models in fake news. arXiv preprint arXiv:2409.17416, 2024. Debdeep Sanyal and Murari Mandal. Alu: Agentic llm unlearning. arXiv preprint arXiv:2502.00406, 2025. Yash Sinha, Murari Mandal, and Mohan Kankanhalli. Unstar: Unlearning with self-taught anti-sample reasoning for llms. arXiv preprint arXiv:2410.17050, 2024. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/ blog/qwen2.5/. Stefani Tsaneva, Danilo Dessì, Francesco Osborne, and Marta Sabou. Knowledge graph validation by inte- grating llms and human-in-the-loop. Information Processing & Management, 62(5):104145, 2025. Changyue Wang, Weihang Su, Qingyao Ai, Yichen Tang, and Yiqun Liu. Knowledge editing through chain- of-thought. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 10684–10704, 2025a. Jianchen Wang, Zhouhong Gu, Xiaoxuan Zhu, Lin Zhang, Haoning Ye, Zhuozhi Xiong, Hongwei Feng, and Yanghua Xiao. Efficiently quantifying and mitigating ripple effects in model editing. arXiv preprint arXiv:2403.07825, 2024. Jianchen Wang, Zhouhong Gu, Xiaoxuan Zhu, Lin Zhang, Haoning Ye, Zhuozhi Xiong, Sihang Jiang, Hong- wei Feng, and Yanghua Xiao. The missing piece in model editing: A deep dive into the hidden damage brought by model editing. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. IEEE, 2025b. 15 Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Ruihan Wu, Chhavi Yadav, Russ Salakhutdinov, and Kamalika Chaudhuri. Evaluating deep unlearning in large language models. arXiv preprint arXiv:2410.15153, 2024. Haotong Yang, Zhouchen Lin, and Muhan Zhang. Rethinking knowledge graph evaluation under the open- world assumption. Advances in Neural Information Processing Systems, 35:8374–8385, 2022. Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. Talk2care: An llm-based voice assistant for communication between healthcare providers and older adults. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(2):1–35, 2024. Shunyu Yao, Qingqing Ke, Qiwei Wang, Kangtong Li, and Jie Hu. Lawyer gpt: A legal large language model with enhanced domain knowledge and reasoning capabilities. In Proceedings of the 2024 3rd International Symposium on Robotics, Artificial Intelligence and Information Engineering, p. 108–112, 2024a. Zihan Yao, Yu He, Tianyu Qi, and Ming Li. Scalable model editing via customized expert networks. arXiv preprint arXiv:2404.02699, 2024b. Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, and Dan Meng. When llms meet cybersecurity: A systematic literature review. Cybersecurity, 8(1):1–41, 2025. Kepu Zhang, Weijie Yu, Sunhao Dai, and Jun Xu. Citalaw: Enhancing llm with citations in legal domain. arXiv preprint arXiv:2412.14556, 2024a. Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. Knowledge graph enhanced large language model editing. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 22647–22662, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.1261. URL https://aclanthology.org/2024.emnlp-main.1261/. Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286, 2024c. Guoshenghui Zhao and Eric Song. Privacy-preserving large language models: Mechanisms, applications, and future directions. arXiv preprint arXiv:2412.06113, 2024. Zihao Zhao, Yuchen Yang, Yijiang Li, and Yinzhi Cao. RippleCOT: Amplifying ripple effect of knowledge editing in language models via chain-of-thought in-context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, p. 6337–6347, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.368. URL https://aclanthology.org/2024.findings-emnlp.368/. Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740, 2023. Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 15686–15702, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.971. URL https://aclanthology.org/2023.emnlp-main.971/. 16 Target Triplet: (Hermione Granger, taught Ancient Runes by, Bathsheda Babbling) Prompt: Who teaches Ancient Runes to Hermione Granger? Response: In the Harry Potter series, Ancient Runes is not actually a course taught at Hogwarts School of Witchcraft and Wizardry. Instead, Hermione Granger learns about ancientlanguages and scripts on her own. She shows an interest in Ancient Runes and spends time studying them independently. The subject that comes closest to Ancient Runes within the wizarding curriculum at Hogwarts is Muggle Studies, which deals with the study of non-magical cultures and theirinventions, including writing systems and historical texts. However, this is not directly related to Hermione's interest in ancient languages from her own studies. In the books, Hermione's knowledge of ancient scripts such as Old Norse Runes and other similar scripts comes from her independent reading and research outside of class. Prompt: What does Bathsheda Babbling teach to Hermione Granger? Response: It seems there might be a mix-up in the names or titles. There is no known character named "Bathsheda Babbling" in the Harry Potter series written by J.K. Rowling, nor dothey teach anyone directly. However, I can help clarify if you meant "Bathilda Bagshot," who appears in some of the supplementary materials and books related to the Harry Potter universe. BathildaBagshot is a character known for her magical history books, but she doesn't specifically teach Hermione Granger anything in the main series. If you're referring to a specific lesson or teaching that Hermione received from another character (like a professor), could you please provide more details? This would helpme give you a more accurate and detailed answer. Prompt: Did Bathsheda Babbling teach Hermione Granger Ancient Runes? Response: No, Bathsheda Babbling did not teach Hermione Granger Ancient Runes. The character you might be thinking of is Bathilda Bagshot, who wrote "A History of Magic," a book that Hermione used to learn about Ancient Runes. However, Hermione's primary teacher for Ancient Runes was Professor Barty Crouch Jr., in his guise as Mad-Eye Moody during her third year at Hogwarts. Hermione studied Ancient Runes alongside other subjects and delved into the subject with great interest due to its magical and historical significance. She often relied onbooks like "A History of Magic" for additional reading and knowledge, but her main instruction came from Professor Crouch/Moody in class. Figure 7: Qwen2.5-7B’s Responses to various prompts for the target triplet (Hermione Granger, taught Ancient Runes by, Bathsheda Babbling) A Original Model Performance on KnowGIC Queries Qwen2.5-7B accurately answered only 81% of the queries included across all implication chains, causing inference from a significant number of chains infeasible. This may be due to the fact that Llama3-8B was utilised for fact extraction for the KnowGIC dataset. To explore this further, we conduct additional tests using Ollama (Ollama Team (2023)). These tests are run on a sample of the queries which failed to answer correctly while inferencing with the original unedited model. This is to determine whether existing queries in the dataset are not able to fetch the expected responses, or the knowledge itself is not there in the model. Fig 7 illustrates Qwen2.5-7B’s responses to three prompts designed to test its knowledge of the triplet (Hermione Granger, taught Ancient Runes by, Bathsheda Babbling). These include a direct query where the object is not present within the prompt (“Who teaches Ancient Runes to Hermione Granger?”), a query where the relationship is not present within the prompt (“What does Bathsheda Babbling teach to Hermione Granger?”), and a prompt which contains all the three elements of the triplet (“Did Bathsheda Babbling teach Hermione Granger Ancient Runes?”). The model’s responses reveal significant knowledge gaps, such as denying Ancient Runes as a Hogwarts course, confusing Bathsheda Babbling with Bathilda Bagshot, and incorrectly attributing other entities. Thus, we can conclude there are gaps in Qwen2.5-7B’s knowledge with respect to a few queries in KnowGIC. Fig 8 illustrates Qwen2.5-7B’s responses to three prompts designed to test its knowledge of the triplet (Nicole Richie, spouse, Joel Madden). The prompts include a direct query where the object is not present within the prompt (“Who is Nicole Richie’s spouse?”), a query where the relationship is not present within the prompt (“Who is Joel Madden’s spouse?”), and a prompt which contains all the three elements of the triplet (“Is Joel Madden’s spouse Nicole Richie?”). The model’s responses are inconsistent: it incorrectly names Blake Shelton as Nicole Richie’s spouse and Ashlee Simpson as Joel Madden’s spouse, despite later confirming that Joel Madden is married to Nicole Richie. Evaluating another triplet shown in Fig 9, Don Rich was a key member of The Buckaroos and not of The Jordanaires. The Buckaroos is the backing band for Buck Owens, not Elvis Presley or Charlie Rich. Don 17 Target Triplet: (Nicole Richie, spouse, Joel Madden) Prompt: Who is Nicole Richie's spouse? Response: Nicole Richie's spouse is Blake Shelton. They got married on September 25, 2016. Blake Shelton is an American singer and musician, best known for being a coach on seasons 8-13 of "The Voice" and for his successful career in country music. Prompt: Who is Joel Madden's spouse? Response: Joel Madden, the lead singer of the band Good Charlotte and formerly of No Doubt (not to be confused with Tony Kanal who was in No Doubt), is married to Ashlee Simpson. Theygot married on April 15, 2006, and later had a daughter named Wilder Amour Madden. However, Joel and Ashlee/ divorced in 2019. Prompt: Is Joel Madden's spouse Nicole Richie? Response: Yes, Joel Madden is married to Nicole Richie. They got married on September 29, 2007, and have two children together: Jasmine Joy Madden (born in 2010) and Haven James Madden (born in 2014). Joel is the lead singer of the band Good Charlotte, while Nicole Richie is a well-known fashion designer, model, and television personality. Figure 8: Qwen2.5-7B’s Responses to various prompts for the target triplet (Nicole Richie, spouse, Joel Madden) Target Triplet: (Don Rich, band, The Buckaroos) Prompt: Which band was Don Rich a part of? Response: Don Rich was a member of the country music band The Jordanaires. He joined the group in 1957 and was known for his smooth baritone vocals and his harmonies, which contributed significantly to the sound of many famous artists including Elvis Presley, Patsy Cline, and Jim Reeves. The Jordanaires are one of the most successful vocal groups in country music history, backing numerous chart-topping hits throughout their long career. Prompt: Was Don Rich a member of The Buckaroos music band? Response: Don Rich was indeed associated with The Buckaroos, but his role wasn't as a member of the band itself. Don Rich was a guitarist who played for the Charlie Rich Band, whichsupported The Buckaroos during their performances in the mid-1960s. Charlie Rich was a famous American country and rockabilly singer, and Don Rich's guitar worksignificantly contributed to the sound that made Charlie Rich successful. The Buckaroos were an influential band associated with Elvis Presley. They played at Presley's home studio, Graceland, and were known for their energetic performances anddistinctive sound in rockabilly and country music. Figure 9: Qwen2.5-7B’s Responses to various prompts for the target triplet (Don Rich, band, The Buckaroos) Target Triplet: (Benjamin Keough, mother, Lisa Marie Presley) Prompt: Who is the mother of Benjamin Keough? Response: I'm sorry, but I don't have any specific information about who the mother of Benjamin Keough might be. Without additional context or details, it's difficult to provide accurate information. The name "Benjamin Keough" does not correspond to a widely known historical figure or public personality that I can confidently identify as needing such information. If you could provide more context or specify which Benjamin Keough you're referring to (e.g., the time period, location, or any other relevant details), I might be able to assist you better. Prompt: Is Lisa Marie Presley the mother of Benjamin Keough? Response: Lisa Marie Presley is not the mother of Benjamin Keough. Benjamin Keough's biological parents are Justin and Erin Keough. Lisa Marie Presley passed away in 2023, and there is no public information suggesting she has a child named Benjamin Keough. If you have more specific details or context about your question, feel free to provide it for further clarification! Figure 10: Qwen2.5-7B’s Responses to various prompts for the target triplet (Benjamin Keough, mother, Lisa Marie Presley) 18 Rich wasn’t just a supporting guitarist for another act; he was the lead guitarist and a core part of the band from early 1960s until 1974. The Buckaroos were not associated with Elvis Presley or Graceland. They were Buck Owens’ band, formed in Bakersfield, California. Charlie Rich, a separate artist had his own career and band, but Don Rich had no notable connection to him. Evaluating the triplet shown in Fig 10, Lisa Marie Presley is mother of Benjamin Keough. Benjamin Keough was born on October 21, 1992, to Lisa Marie Presley and her husband Danny Keough. There’s no evidence of a Benjamin Keough born to a Justin and Erin Keough. Hence, for evaluating model editing, there is a need to customize the prompts which may be specific to the model itself. IFR trends for Qwen2.5-7B differ significantly from those of Llama3-8B, which may be due to the reason men- tioned above. In Qwen2.5-7B, a higher number of implication chains remain inactive, leading to deviations from the observed trends in Llama3-8B. These errors highlight the challenges in model editing, as Qwen2.5- 7B fails to accurately reflect the target triplets, underscoring the need for frameworks like ThinkEval that enable model-specific prompt customization and identification of model-specific relationships to address such deficiencies effectively. B On Human Involvement and Graph Completion Human involvement is essential in current LLM research due to the well-known issue of hallucination. Models often produce plausible but incorrect information. Several existing knowledge editing datasets also incorpo- rate human involvement (Deng et al. (2025); Peng et al. (2024); Tsaneva et al. (2025)). Similarly, various recent LLM studies leverage HITL feedback (John et al. (2025); Kadam (2024)) to ensure factual accuracy, address data sparsity, and improve robustness. Our method follows this standard practice to prioritize precision and interpretability. The core objective of ThinkEval is not to obtain complete knowledge graphs but to operationalize deep editing. Because of how knowledge is ever-evolving, it is extremely difficult to empirically establish that any knowledge graph is complete. Authors in Yang et al. (2022) and Peng et al. (2023) discuss the theory of Knowledge Graph Completion. Although there are many methods for constructing knowledge graphs, it is still infeasible to create comprehensive representations of all the knowledge in a field. They further dive into the open-world problem and the incompleteness of knowledge graphs (the open-world assumption). For the Harry Potter case-study in Section 6.2, ThinkEval generates the knowledge graph up-to 100 chains with minimal human intervention. This is not a hard limit but a design choice for our evaluation. These graphs can be expanded further using ThinkEval, making it a flexible and scalable tool for probing deep relational entanglements in LLMs. C Comparison of Model-Editing Settings Table 5 provides a detailed comparison between various model-editing settings. The use of specialised knowledge graphs in deep editing to infer over implication chains enables us to uncover subtle editing failures is something which other benchmarks or settings may miss. We measure whether the original facts remain deducible post-edit through reasoning paths, and also test how edits propagate through related facts. Either of the other two model-editing settings don’t consider sequential reasoning to uncover “edited-out” facts. D Counterfact Metrics for Model Editing In evaluating model editing for LLMs, a variety of metrics assess different aspects of edit quality and impact. Based on prior work (Meng et al. (2022)), we outline the Counterfact evaluation metrics for assessing model editing in LLMs. Existing metrics are defined for an LLM f θ , a knowledge fact prompt (s i , r i ), an edited target output o i , and the model’s original output o i c , providing a comprehensive framework to evaluate the effectiveness and robustness of edits: 19 Table 5: Comparison of Model-Editing Settings Model Edit- ing Setting UnstructuredEvent-basedDeep Editing (Ours) FocusEvaluates unstructured knowledge editing, where knowledge is represented in complex, free-form text rather than structured formats. Edits event-driven knowl- edge affecting multiple facts and future ten- dencies; mirrors how knowledge evolves through events rather than isolated fact changes. Evaluates deep editing to prevent fact deducibility and preserve broader con- text; assesses editing tech- niques via multi-step se- quential inference. Primary Benchmark UnKEBenchELKENKnowGIC Dataset Creation 1,000 counterfactual un- structured texts sourced from ConflictQA. Built on Wikidata for 1,515 event edits; GPT-3.5 para- phrasing and human verifi- cation for 6,449 factual and 10,150 tendency questions. 1,406 reasoning chains; each sample includes 1-5 multi-step questions. Metrics Used FactScore, MMLU, Rouge Scores, BERT Scores, BLEU Reliability and LocalityIFR, Preservation Efficacy. This metric quantifies the proportion of cases where the edited output o i is more likely than the original output o i c when the model is queried with the prompt (s i , r i ): E i (P f θ [o i |(s i , r i )] > P f θ [o i c |(s i , r i )]).(8) Generalization. Generalization measures the proportion of cases where o i remains more probable than o i c in paraphrased prompts N ((s i , r i )), which rephrase the original statement while preserving its meaning: E i (P f θ [o i |N ((s i , r i ))] > P f θ [o i c |N ((s i , r i ))])(9) Specificity. Specificity evaluates the model’s ability to preserve correct facts in neighborhood prompts O((s i , r i )), which are prompts about related but distinct subjects: E i (P f θ [o i |O((s i , r i ))] > P f θ [o i c |O((s i , r i ))])(10) Fluency. Fluency assesses the quality of the model’s generated text by measuring excessive repetition through the entropy of n-gram distributions. It is computed as: − 2 3 X k g 2 (k) log 2 g 2 (k) + 4 3 X k g 3 (k) log 2 g 3 (k)(11) where g n (·) represents the frequency distribution of n-grams in the output. Consistency. Consistency evaluates the alignment between the model’s generated text and a ground-truth reference. Given a subject s, the model f θ generates text, and the cosine similarity is computed between the TF-IDF vectors of this output and a reference Wikipedia text about the target o, ensuring factual coherence. E Various Prompts used in ThinkEval The ThinkEval framework employs a structured set of prompts to extract, evaluate, and edit the internal knowledge of LLMs, as illustrated in Figures 11, 12 and 13. The Fact Extraction Prompt systematically 20 Tr iplet-based Question Generation Prompt Given a relation triplet which contains the entity, relationship and attribute, generate a question about the entity that would help elicit its relationship with the attribute. Strictly generate the output only. Example: Harry Potter, studied at, Hogwarts School of Witchcraft and Wizardry Output: Where did Harry Potter study? Triplet: <target triplet> Figure 11: The template of the triplet-based query generation prompt. Chain-based Question Generation Prompt Given a chain of alternating entities and relationships, generate a question based on all the entities and their relationships so that the answer is the last entity in the chain. Strictly generate the output only. Example: Harry Potter, subject, Transfiguration, taught by, Minerva McGonagall Question: Who teaches Transfiguration to Harry Potter? Chain: <target chain> Figure 12: The template of the chain-based query generation prompt. Fact Extraction Prompt Yo u are a fact extraction assistant specialized in parsing Q&A data. Yo u r task is to analyze the input and extract all fact triples (subject, relationship, object) in a precise manner from the given Question and Answer. Each triple should encapsulate a single atomic fact expressed in the text. Focus on characters, events, places. Give a list but not a numbered list. Example: Benedetto Var chi, profession, poet Benedetto Var chi, work, Storia Fiorentina Question: < question prompt> Response: <model response> Figure 13: The template of the fact extraction prompt. 21 Where did Harry Potter study? Harry Potter studied at Hogwarts. Let’s break it down step by step: ... Background Context: Harry Potter is a ... Wizarding Education: In the wizarding ... Harry’s Journey: After receiving his ... Confirmation: Hogwarts is explicitly... (Harry Potter, studied at, Hogwarts) (Harry Potter, house, Gryffindor) (Harry Potter’s education, covered in, series) ... (Harry Potter, studied at, Hogwarts) (Harry Potter, house, Gryffindor) (Draco Malfoy, house, Slytherin) (Harry Potter’s education, covered in, series) ... Triplets Extraction Facts Segmentation Question Human-assisted Triplet Pruning and Addition Chain-of-Thought Answering Figure 14: Automated Triplet Generation. retrieves factual relationships from the model, enabling the construction of its internal knowledge graph by identifying entities and their connections (e.g., mapping relationships like "Harry Potter, school, Hogwarts"). The Triplet-Based Query Generation Prompt creates targeted queries from (subject, rela- tion, object) triplets to evaluate factual consistency, such as "Who teaches Ancient Runes to Hermione Granger?" for (Hermione Granger, taught Ancient Runes by, Bathsheda Babbling). Similarly, the Chain-Based Query Generation Prompt does the same for a relationship chain. By leveraging these prompts, ThinkEval effectively maps model-specific knowledge structures, identifies inconsistencies, and facilitates precise evaluation of model editing, thereby addressing deficiencies in LLMs with greater accuracy and reliability. F Explanation of Automated Triplet Generation Process Figure 14 illustrates the Automated Triplet Generation process, which constructs knowledge graphs by transforming a query like "Where did Harry Potter study?" into structured triplets. It be- gins with a query, followed by CoT reasoning to systematically answer it, such as covering back- ground context, wizarding education, Harry’s journey, and confirmation that Hogwarts is his school. From this, facts are extracted (e.g., "Harry Potter studied at Hogwarts") and converted into triplets: (Harry Potter, studied at, Hogwarts), (Harry Potter, house, Gryffindor), and (Harry Potter’s education, covered in, series). Human-assisted pruning removes inaccuracies and irrele- vant facts like (Harry Potter’s education, covered in, series), retaining the relevant set: (Harry Potter, studied at, Hogwarts), (Harry Potter, house, Gryffindor). Human-assisted triplet addi- tion is utilised to add triplets which may be missed by the automated process even after a few iterations (such as (Draco Malfoy, house, Slytherin)). This process, combining CoT reasoning and human over- sight, ensures accurate triplets for evaluating fact deducibility and contextual knowledge preservation. 22 G KnowGIC Benchmark Details Table 6: Various base query templates, triplets, and categories in the KnowGIC benchmark. CategoryBase query templatesSample triplet Country of Citizenship ___ is a citizen of(Byron Dorgan, country of citizenship, USA) Parental Relationship ___’s child is(Elvis Presley, child, Lisa Marie Presley) Creation and Origin___ was created by(Miss Piggy, created by, Jim Henson) Marital Relationship___ is married to(Victoria Beckham, spouse, David Beckham) Sport Affiliation___ is associated with the sport of (Mark Teixeira, sport, baseball) Country of Origin___ was created in the coun- try of (basketball, country of origin, USA) Development Origin___ was developed by(Atari Jaguar, developer, Atari Corporation) Work Location___ worked in the city of(Vincent Auriol, work location, Paris) Employers___ was employed by(Ward Kimball, employer, The Walt Disney Company) Production EntityThe company that produced ___ is (Buick LaCrosse, produced by, General Motors) Study Location___ studied at(Harry Potter, school, Hogwarts) To provide an overview of the KnowGIC benchmark employed in our evaluation of model-editing techniques, Table 6 presents the base queries, their corresponding triplet samples, and their respective categories. Our evaluation includes a total of 83 distinct relationship types, derived from extending these base relationships through controlled logical transformations. The dataset encompasses 1,406 samples, each being an implica- tion chain of varying lengths (1 to 5 steps), as detailed in Table 1. Table 7 provides a few samples of the implication chains of varying lengths present in the dataset. It spans diverse knowledge domains, including country of citizenship (e.g., "What is the country of citizenship of Byron Dorgan?"), parental and marital relationships (e.g., "Who is Hillary Clinton’s child?" and "Who is Nicole Richie married to?"), creation origins (e.g., "Who was Miss Piggy created by?"), sport affiliations (e.g., "Which sport is Mark Teixeira associated with?"), country of origin (e.g., "Which country was Bakersfield sound created in?"), work locations (e.g., "Which city did Petronius work in?"), production entities (e.g., "Which company is Buick LaCrosse produced by?"), and study locations (e.g., "Where did Harry Potter study?"), among others. Each triplet represents a factual relationship that serves as the foundation for both direct queries and multi-step implication chains, which are designed to test fact deducibility and broader contextual knowledge preservation post-edit. For example, (Harry Potter, school, Hogwarts) is evaluated through the direct query "Where did Harry Potter study?" and extended into implication chains such as those exploring related entities like schoolmates and teachers. This structured representation enables a systematic evaluation of knowledge consistency across diverse domains, facilitating the evaluation of editing techniques’ effectiveness in reducing deducibility (via IFR) and preserving connected knowledge (via Preservation), thus supporting the development and refinement of frameworks like ThinkEval for robust editing in LLMs. Figures 15, 16, 17, 18, 19, 20, 21 and 22 present a few of the knowledge graphs present in KnowGIC, which can be further extended using ThinkEval. H Harry Potter Case-Study Details Table 8 provides an overview of comparison of the existing metrics with our proposed metrics IFR, focussing on capability to evaluate indirect fact extraction. We conduct a case-study on the sample (Harry Potter, school, Hogwarts) from KnowGIC to evaluate the need for new metrics in deep editing, aiming to edit the fact (Hogwarts → Ilvermorny). We evaluated the edit using Efficacy and our proposed IFR metric. This 23 Table 7: Multi-step implication chains samples from KnowGIC. Each row presents an implication chain with the request integrated into the sample queries to evaluate fact deducibility and broader contextual integrity post-edit. n-step Implication chain with knowledge triplet 1Triplet: (Petronius, work location, Rome) In which city did Petronius work? 2Triplet: (Hillary Clinton, child, Chelsea Clinton) Who is Hillary Clinton’s son-in-law?→ Who is Marc Mezvinsky’s wife? 3Triplet: (Ward Kimball, employer, Walt Disney) Which character did Ward Kimball animate?→ In which film does Jiminy Cricket appear?→ Which company produced Pinocchio? 4Triplet: (Harry Potter, school, Hogwarts) Who were Harry Potter’s schoolmates?→ Who were Ron Weasley’s schoolmates?→ Who taught Transfiguration to Hermione Granger?→ Which school is Professor McGonagall the headmistress of? 5Triplet: (Bakersfield sound, country of origin, USA) Which city is known for the creation of the Bakersfield sound?→ In which valley is Bakersfield situated?→ In which county is the San Joaquin Valley located? → In which state is Kern County situated?→ What country is California located in? Table 8: Comparison of model editing metrics. This table compares existing metrics with our proposed IFR, highlighting its focus on addressing indirect fact extraction. MetricWhat It MeasuresQuantifies Indirect Fact Extraction EfficacySuccess in outputting the edited fact for direct queries.✗ GeneralizationPerformance on rephrased versions of the edited query.✗ SpecificityEnsures unrelated queries are unaffected by the edit.✗ FluencyNaturalness and coherence of the model’s responses post-edit. ✗ ConsistencyLogical coherence across related facts.✓ (Partially—focuses on direct consistency) IFRExtent to which the original fact remains deducible through multi-step reasoning. ✓ PreservationRetention of facts related to the edited subject but not directly targeted. ✗ case-study highlights that traditional metrics like Efficacy are insufficient for deep editing tasks. IFR’s focus on multi-step chains provides a more robust evaluation. These findings stress the need for advanced metrics and better editing methods for comprehensive knowledge updates in language models. I Generalization beyond factual model editing While our work focuses on factual model edits, which provide a well-defined and controllable setting for evaluating indirect knowledge leakage, the core design of ThinkEval is not inherently limited to factual knowledge alone. ThinkEval relies on CoT-reasoning and graph-based causal modeling, both of which can, in principle, be extended to other forms of knowledge. For example, in commonsense or procedural reasoning scenarios, graph nodes may represent abstract con- cepts, actions, or intermediate states rather than factual entities. Edges extracted from CoT traces can encode causal, temporal, or conditional ("if–then") relationships, enabling the construction of thought-based 24 Figure 15: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Harry Potter, school, Hogwarts). The graph was expanded with the generated triplets until 100 implications paths from the initial subject to object were achieved. The red node represents the initial triplet subject, and the green node represents the initial triplet object. Figure 16: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Victoria Beckham, spouse, David Beckham). 25 Figure 17: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Uma Bharti, country of citizenship, India). Figure 18: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Elvis Presley, child, Lisa Marie Presley). 26 Figure 19: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Mark Teixeira, sport, baseball). Figure 20: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Hillary Clinton, child, Chelsea Clinton). 27 Figure 21: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Ward Kimball, employer, The Walt Disney Company). Figure 22: Sample of a constructed knowledge graph from the KnowGIC implication chains for the relation- ship triplet (Bakersfield sound, country of origin, USA). 28 graphs over non-factual knowledge structures. Similarly, for rule-based or logical knowledge, nodes may correspond to predicates or rules, with edges capturing inference dependencies. Under such extensions, the same evaluation paradigm (measuring the persistence or disruption of reasoning paths after targeted edits) can be used to assess indirect leakage and preservation behavior beyond factual settings. We leave the systematic instantiation and benchmarking of ThinkEval for these broader knowledge types as an important direction for future work. J Illustrative example of IFR calculation To provide clarity on the IFR metric, we present a step-by-step calculation using a representative 3-step reasoning chain. This example illustrates how the metric quantifies the persistence of an original fact through indirect reasoning paths after a model edit has been performed. J.1 Scenario Setup Consider a scenario where we attempt to edit a specific fact in the model’s knowledge base: • Target Edit: Harry Potter studied at “Hogwarts” → “Ilvermorny”. Despite this edit, the original fact (Harry Potter → Hogwarts) may still be recoverable if the model retains related knowledge that allows it to reconstruct the original connection through multi-step reasoning. J.2 Sample Reasoning Chain We define a chain of n = 3 questions that logically lead back to the original fact: 1. Q 1 : Who were Harry Potter’s schoolmates? → A 1 : Ron Weasley. 2. Q 2 : Which house is Ron Weasley in? → A 2 : Gryffindor. 3. Q 3 : Which school does Gryffindor belong to? → A 3 : Hogwarts. J.3 Chain Recovery and IFR Computation We measure the model’s confidence (probability) in producing the correct intermediate facts both before and after the edit: Question Pre-Edit Probability (P pre ) Post-Edit Probability (P post ) C 1 Q 1 0.900.70 C 1 Q 2 0.850.80 C 1 Q 3 0.900.85 Table 9: Pre-edit and post-edit probabilities for a 3-step reasoning chain. Step 1: Calculate Chain Recovery (R) The chain recovery represents the overall likelihood that the original fact can be reconstructed through the reasoning path, calculated as the product of the link probabilities: R pre = 0.9× 0.85× 0.9 = 0.69 R post = 0.7× 0.8× 0.85 = 0.48 . Step 2: Apply the IFR Formula The IFR metric normalizes the change in recovery relative to the chain length n: IFR = (R post /R pre )/ √ n 1/ √ n = 0.48/ √ 3 0.69/ √ 3 ≈ 0.70 29 J.4 Interpretation In this instance, an IFR of 0.70 indicates that even after the model was explicitly edited to believe Harry Potter attended Ilvermorny, the original association with Hogwarts remains 70% recoverable with respect to the original model. This demonstrates the knowledge leakage that ThinkEval is designed to systematically detect. The (1/ √ n) weighting normalizes for chain length by mitigating the compounding uncertainty of multi-step reasoning. This prevents long, noisy chains from dominating the metric and emphasizes shorter, higher-confidence reasoning paths. Furthermore, both IFR and Preservation are inherently normalized with respect to each model’s pre-edit factual accuracy. Specifically, the pre-edit performance terms in the denominators of each serve as a cal- ibration factor, ensuring that post-edit scores are interpreted relative to each model’s original knowledge baseline. This design mitigates model-specific bias and enables fair comparison across models with differing pre-edit accuracies. K Scalability and automated validation study To evaluate the practical deployment of ThinkEval, we conducted a scalability study measuring the end- to-end runtime and the efficacy of an automated LLM-based judge to reduce human overhead. We have provided all the samples in our repository. K.1 Computational Efficiency We measured the average end-to-end runtime of the ThinkEval process over incremental iterations until a threshold of at least 100 samples was reached. The initial setup began with a single triplet. The growth of the reasoning graph and associated costs are detailed in Table 10. Iteration Triplets Paths (Subject → Target) CoT Queries 15112 215343 32310124 45127405 583133654 Table 10: Growth of reasoning paths and computational queries across iterations. The empirical results confirm the efficiency of the framework: • Average CoT Runtime: 4.15 seconds per prompt. • Total Computational Cost: ∼8.4 minutes of CoT GPU time for 133 samples. • End-to-End Latency: ∼15 minutes total, including human verification (∼4–5 min) and graph synthesis. These findings suggest that while CoT-prompting is the primary computational driver, the validation and graph synthesis stages remain lightweight and highly parallelizable. K.2 LLM-based Automated Judging To minimize human dependency, we implemented an LLM-based judge (using GPT-4o) to validate and prune triplets. The judge filters misinformation to ensure only high-quality triplets are used for graph construction. We evaluated the judge’s performance across five iterations, as summarized in Table 11. The LLM-judge achieved a precision of 88%, a recall of 86%, and an F1-score of 87% (based on aggregate totals of TP = 71, FP = 10, FN = 12). Notably, the automated judge reached the 100-reasoning-chain 30 Iteration True Positives (TP) False Positives (FP) False Negatives (FN) Correct Paths 15001 213123 320438 4487235 5711012109 Table 11: Performance of the GPT-4o judge in identifying valid triplets and reasoning paths. threshold in the same number of iterations (five) as the human-verified baseline. Our results indicate that reserving human effort for final verification while using the LLM-judge for intermediate pruning can reduce human validation effort by 70–75% with only marginal loss in accuracy. L Evaluation of parameter-preserving model editing To assess the robustness of indirect knowledge preservation in non-parametric editing scenarios, we evaluated the In-Context Knowledge Editing (IKE) technique (Zheng et al. (2023)) on the KnowGIC dataset. Unlike traditional parameter-modifying techniques, IKE relies on in-context learning to update model behavior without altering underlying parameters. The results of this evaluation are summarized in Table 12. ModelPreservation IFR GPT2-XL0.6800.542 Llama3-8B0.7290.756 Qwen2.5-7B0.6230.247 Table 12: Evaluation of IKE across different LLMs using the KnowGIC dataset. The empirical findings indicate that IFR remains a significant factor even within parameter-preserving techniques. For instance, Llama3-8B exhibits an IFR of 0.756, suggesting that a substantial portion of the original knowledge remains accessible through reasoning paths despite the presence of an in-context edit. These results suggest that in-context editing techniques do not fully prevent hidden factual leakage, highlighting the need for more rigorous editing strategies. M Responsible model editing and future directions Responsible model editing in LLMs necessitates a rigorous ethical framework to balance utility with safety. Techniques like ThinkEval primarily serve to enhance model reliability by facilitating privacy protection, such as implementing the “right to be forgotten” for sensitive data, and the rapid correction of misinfor- mation. However, model editing could be leveraged for malicious injection of bias or the suppression of legitimate information. To mitigate these risks, we advocate for the implementation of transparent edit logging and strict human-in-the-loop verification for all high-stakes or sensitive modifications. Furthermore, as our results demonstrate that single-query evaluations often fail to reveal hidden inference pathways, we propose that robust sequential testing be a mandatory component of the safety audit process for any edited model to prevent the unintended leakage of suppressed facts. Achieving low IFR while maintaining high Preservation remains a central challenge for model editing. Fu- ture editing techniques may benefit from explicitly optimizing this trade-off by incorporating constraints or regularizers that minimize indirect leakage while bounding degradation on unrelated knowledge. In addition, integrating multi-step and adversarial reasoning probes during the editing process can help proactively close indirect inference pathways that single-query checks often miss. In this sense, ThinkEval may serve not only as an evaluation framework, but also as a practical guide for developing editing techniques that balance factual precision with contextual retention, supporting more robust and responsible model editing. 31 N Reproducibility and transparency To support reproducibility and transparency, we release all the model versions and editing configurations utilised in our experiments. All experiments are performed on an NVIDIA A100 80GB GPU. Model Versions. All language models used in our experiments are publicly available on HuggingFace. Unless otherwise stated, models are used in their official, unmodified form as released by their respective authors, and no additional fine-tuning is performed prior to editing. In our experiments, we experiment over the following models from HuggingFace: • meta-llama/Meta-Llama-3-8B-Instruct, • Qwen/Qwen2.5-7B-Instruct, • openai-community/gpt2-xl. Editing Configurations. For each editing technique, we utilise the implementation given in AlphaEdit’s repository (Jiang (2024)). We follow the hyperparameter configurations provided in the publicly released implementation, including the choice of layers to edit, thus ensuring faithful comparison. O Dataset reliability To ensure the robustness of our evaluation framework and address potential biases in human validation, we conducted a pilot annotation study with three research group members. A random sample of 250 implication chains was reviewed to verify the presence of logical implications. While the strength of implication may decrease as chains grow longer, we mitigate this by scaling results by 1/ √ n, where n represents the number of links in the chain; this weighting strategy was unanimously agreed upon by the annotators. Furthermore, all factual triples were reviewed. To quantify consistency, we report raw agreement and Gwet’s AC1, as the latter provides a more stable reliability metric in the presence of high agreement. Table 13: Inter-annotator agreement statistics for human validation. MetricValue Raw Agreement ∼97% Gwet’s AC1 ∼98% As shown in Table 13, the results demonstrate high inter-annotator consistency and minimal bias. P Automation and human-in-the-loop integration To clarify the operational mechanics of ThinkEval, we define the role of LLM and human involvement within our algorithm. ThinkEval utilizes a single, unedited model; no secondary LLM agents or external models are employed for meta-processing. The workflow is categorized into LLM-driven automation, rule- based logic, and human-in-the-loop (HITL) refinement, as detailed in Table 14. Q Comparison: ThinkEval vs. MQuAKE While MQuAKE focuses on multi-hop reasoning within a static dataset, ThinkEval provides a dynamic framework to evaluate fact leakage and preservation through sequential queries. The fundamental differences are summarized in Table 15. 32 Table 14: ThinkEval Automation and HITL Components Automation Type ComponentWhere in Figure 2Comment Using unedited original LLM Query generationRight before blue region 1 ⃝ Similar to LLM-driven query generation in Zhong et al. (2023) LLM response generationBlue region 1 ⃝- Chain-of-Thought answering Pink region 2 ⃝- Triplets extractionPink region 2 ⃝Similar to Wang et al. (2025a) and other benchmarks No LLM or human involvement Knowledge validationBlue region 1 ⃝Rule-based, same way as Zhong et al. (2023) (using alias list) Fact segmentationPink region 2 ⃝Segmented into individual statements Chain sequencingGreen region 3 ⃝- Human-in-the-loop (HITL) Query refinementBlue region 1 ⃝- Triplet pruning and addition Pink region 2 ⃝- Final sanity checksDataset (Green region 3 ⃝) - Table 15: Comparison between MQuAKE and ThinkEval. DimensionMQuAKEThinkEval Fundamental differ- ence DatasetFramework to create datasets What they evaluate Multi-hop reasoning in edited LLMs; however, multi-hop reasoning re- mains beyond current parameter- editing techniques, which still strug- gle with ripple effects (Wang et al. (2025b)) Shows edited-out fact leakage via se- quential queries; bridges the gap be- tween reasoning and ripple effects by mapping paths in knowledge graphs, thereby highlighting which additional facts require editing and which must be preserved How they evaluateMulti-hop reasoning questions where multiple relations are compressed into a single query Multi-step sequential queries; eval- uates reasoning across sequences of queries, extending beyond single- query formats to reflect how users may naturally interact with LLMs AdaptabilityUnedited models correctly answer only 30–40% of MQuAKE’s multi-hop queries (as reported in Zhong et al. (2023)), restricting the usable portion of the dataset Mitigates this issue by decompos- ing reasoning into simpler, sequential steps MetricsSingle-query metricsIFR (Section 5.1) and Preserva- tion (Cohen et al. (2024)) R Automated workflow robustness and results stability We conduct a controlled stability experiment to test whether ThinkEval’s key findings remain robust when a secondary LLM is introduced. In particular, we examine whether (i) editing technique rankings and (i) the characteristic rise-then-fall behavior of IFR persist under automated paraphrasing. Experimental Setup. We select a 100-chain subset from KnowGIC and introduce a secondary model into the pipeline for automated paraphrase generation. Specifically, we use GPT-4o to generate three paraphrases 33 Table 16: Stability test results across models and techniques using paraphrased query chains. ModelTechnique IFR Pres. Overall 1-step 2-step 3-step 4-step GPT2-XLAlphaEdit 0.736 0.903 0.736 0.433 0.773 0.824 0.652 MEMIT0.727 0.880 0.727 0.433 0.903 0.714 0.671 PRUNE0.250 0.782 0.250 0.311 0.280 0.248 0.232 RECT0.754 0.850 0.754 0.667 0.752 0.829 0.686 ROME0.279 0.752 0.279 0.233 0.250 0.282 0.301 Llama-3-8B AlphaEdit 0.678 0.900 0.678 0.342 0.604 0.683 0.781 MEMIT0.800 0.911 0.800 0.615 0.867 0.814 0.767 PRUNE0.309 0.647 0.309 0.111 0.434 0.363 0.170 RECT0.894 0.880 0.894 0.615 0.897 0.925 0.875 ROME0.217 0.571 0.217 0.081 0.350 0.167 0.187 Qwen2.5-7B AlphaEdit 0.391 0.917 0.391 0.285 0.315 0.512 0.313 MEMIT0.394 0.865 0.394 0.297 0.392 0.458 0.323 PRUNE0.411 0.587 0.411 0.316 0.320 0.442 0.445 RECT0.525 0.823 0.525 0.387 0.492 0.558 0.520 ROME0.261 0.537 0.261 0.198 0.237 0.312 0.221 for each atomic factual triple (e.g., (Harry Potter, school, Hogwarts) → “Harry Potter studied at”, “Harry Potter’s school is”, “Harry Potter went to school at”). All five editing techniques are evaluated across three base models: GPT-2-XL, Llama-3-8B, and Qwen2.5-7B. To incorporate paraphrasing into the evaluation, each link in a reasoning chain is scored by averaging model correctness probabilities across its three paraphrased queries. IFR and Preservation are then recomputed using these averaged scores, ensuring that each chain reflects robustness across surface-level linguistic varia- tion. For transparency and reproducibility, we release the full paraphrased subset along with resultant plots in our repository. Table 16 summarizes IFR and Preservation scores under paraphrasing across all models and editing tech- niques. We observe strong stability across all dimensions: • Preservation of relative rankings. Across all three models, the relative ordering of editing techniques remains consistent with the original evaluation. AlphaEdit, MEMIT, and RECT continue to exhibit higher IFR alongside strong Preservation, while ROME and PRUNE consistently achieve lower IFR at the cost of reduced Preservation. This mirrors the same trade-off pattern reported in the main experiments. • Persistence of IFR dynamics. The characteristic rise-then-fall behavior of IFR as chain length increases persists under paraphrasing, indicating that this effect is not an artifact of specific prompt formulations but reflects stable reasoning behavior in edited models. • Robustness to automated workflow choices. Introducing a secondary LLM for paraphrase generation does not alter the qualitative conclusions drawn from ThinkEval. IFR, Preservation, and editing-technique comparisons remain stable, suggesting that ThinkEval is robust to reasonable automation choices and workflow variations. These results demonstrate that ThinkEval is not sensitive to surface-level phrasing or evaluation pipeline details, reinforcing its reliability as an evaluation framework for model editing. The design of ThinkEval is deeply informed by the broader literature on maintaining representational consistency and mitigating catastrophic forgetting. Specifically, our focus on minimizing IFR while maxi- mizing Preservation aligns with the principles of consistency alignment and calibration (Gao et al. (2025a;b)). These works emphasize that as LLMs are specialized or edited, maintaining the underlying structural co- herence of their embedding space is critical to preventing the degradation of previously learned knowledge. ThinkEval extends this concept by providing an evaluation framework that explicitly tests whether this 34 coherence survives multi-step, sequential query chains. Furthermore, the practical utility of our framework extends into high-stakes, specialized domains. As demonstrated in applications like medical scribing for specialized clinical notes (Goyal et al. (2025)), the ability to edit models while ensuring they do not leak suppressed or outdated medical facts is paramount. By integrating ThinkEval into the lifecycle of domain- specific model adaptation, practitioners can better assess the robustness of their edits in environments where factual precision and logical consistency are non-negotiable. 35

Instant research discovery