Paper deep dive

The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models

Ryosuke Takahashi, Go Kamoda, Benjamin Heinzerling, Keisuke Sakaguchi, Kentaro Inui

Year: 2024Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 34

Models: GPT-2

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 5:20:40 PM

Summary

This paper investigates the side effects of knowledge deletion in language models (LMs) by using synthetic knowledge graphs (Erdős-Rényi and Barabási-Albert). The study demonstrates that deleting knowledge associated with popular entities (high-degree nodes) in models trained on power-law distributed graphs (BA) leads to catastrophic side effects, whereas this correlation is absent in uniform (ER) graphs.

Entities (5)

ROME · algorithm · 99%Barabási-Albert graph · graph-model · 97%Erdős-Rényi graph · graph-model · 97%Knowledge Deletion · process · 96%Language Models · technology · 95%

Relation Signals (3)

Language Models → trainedon → Synthetic Knowledge Graphs

confidence 98% · We train LMs using synthetic knowledge graphs.

ROME → performs → Knowledge Deletion

confidence 95% · We apply the knowledge editing technique to the LM to delete a specific knowledge instance.

Barabási-Albert graph → exhibits → Catastrophic Side Effects

confidence 92% · In the BA graph, there is a clear relationship between the degree of the entity and the impact of its deletion.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Language models (LMs) encode world knowledge in their internal parameters through training. However, LMs may learn personal and confidential information from the training data, leading to privacy concerns such as data leakage. Therefore, research on knowledge deletion from LMs is essential. This study focuses on the knowledge stored in LMs and analyzes the relationship between the side effects of knowledge deletion and the entities related to the knowledge. Our findings reveal that deleting knowledge related to popular entities can have catastrophic side effects. Furthermore, this research is the first to analyze knowledge deletion in models trained on synthetic knowledge graphs, indicating a new direction for controlled experiments.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

33,623 characters extracted from source content.

Expand or collapse full text

The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models Ryosuke Takahashi1 Go Kamoda1 Benjamin Heinzerling2,1 Keisuke Sakaguchi1,2 Kentaro Inui3,1,2 1Tohoku University 2RIKEN 3MBZUAI ryosuke.takahashi, go.kamoda@dc.tohoku.ac.jp benjamin.heinzerling@riken.jp keisuke.sakaguchi@tohoku.ac.jp kentaro.inui@mbzuai.ac.ae Abstract Language models (LMs) encode world knowledge in their internal parameters through training. However, LMs may learn personal and confidential information from the training data, leading to privacy concerns such as data leakage. Therefore, research on knowledge deletion from LMs is essential. This study focuses on the knowledge stored in LMs and analyzes the relationship between the side effects of knowledge deletion and the entities related to the knowledge. Our findings reveal that deleting knowledge related to popular entities can have catastrophic side effects. Furthermore, this research is the first to analyze knowledge deletion in models trained on synthetic knowledge graphs, indicating a new direction for controlled experiments. The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models Ryosuke Takahashi1 Go Kamoda1 Benjamin Heinzerling2,1 Keisuke Sakaguchi1,2 Kentaro Inui3,1,2 1Tohoku University 2RIKEN 3MBZUAI ryosuke.takahashi, go.kamoda@dc.tohoku.ac.jp benjamin.heinzerling@riken.jp keisuke.sakaguchi@tohoku.ac.jp kentaro.inui@mbzuai.ac.ae 1 Introduction Language models (LMs) can store knowledge in their internal parameters through training (Petroni et al., 2019), and research focusing on analyzing the knowledge stored inside LMs has gained attention (Jiang et al., 2020; Heinzerling and Inui, 2021). Although this capability is essential in building human-aiding assistants, challenges related to reliability and safety are also reported. For example, LMs possess knowledge only up to the point when their training data was collected, making them not robust to the constantly changing real-world knowledge (Kasai et al., 2022). Additionally, there is a risk that LMs could leak personal and confidential information contained in the training data, raising privacy concerns (Huang et al., 2022). To address these challenges, several studies have been conducted on knowledge editing (Feng et al., 2023; Zhang et al., 2024; Dai et al., 2022; Meng et al., 2022, 2023; Li et al., 2023) and knowledge deletion (Jang et al., 2023; Ishibashi and Shimodaira, 2023) in LMs. These studies have reported some success in knowledge editing and deletion, yet failures, challenges, and difficulties have also been identified. Figure 1: Overview of the analysis flow for the side effects of knowledge deletion using a synthetic knowledge graph: 1. First, create a synthetic knowledge graph. 2. Train the LM on the created knowledge graph. 3. Apply the knowledge editing method, ROME, to delete a specific knowledge instance. 4. Analyze the side effects of the deleted knowledge by comparing the model’s accuracy on the trained knowledge before and after the deletion. As a result, we reveal that deleting knowledge related to popular entities has catastrophic side effects. Our study aims to understand better when and why knowledge editing and deletion work as intended. We hypothesize that analysis of what kind of knowledge is being deleted is essential to fulfill the objective. This paper focuses on the frequency of entities related to knowledge in the training corpus. To test this hypothesis, we design controlled experiments to analyze the impact of knowledge deletion on LMs (Figure 1). We formalize the notion of “kind of knowledge” regarding the structural properties of the knowledge graph trained by the model. We synthesize knowledge graphs with various properties, pre-train LMs on these graphs, perform knowledge deletion of specific facts, and then observe the side effects of deletion on entities related to the deleted knowledge. Analyses reveal that deleting knowledge concerning frequently occurring entities results in significant and catastrophic side effects in LMs trained on knowledge graphs with similar properties to the real world. Furthermore, this study is the first to analyze knowledge deletion using models with controlled knowledge structures, presenting a new direction in the analysis of knowledge deletion using synthetic knowledge graphs. 2 Experimental Design and Approach LMs distribute and encode knowledge across many model parameters, forming complex interdependencies among various knowledge instances. This interconnected structure may lead to unintended side effects when deleting specific knowledge. Furthermore, not all knowledge is equally essential; “more important” knowledge might be connected to many others, and “less important” knowledge with fewer associations. However, accurately identifying what knowledge has been stored in pre-trained LMs is challenging (Jiang et al., 2020), making it difficult to conduct detailed analyses considering the properties of the knowledge structure held internally by existing pre-trained LMs. In light of this, we have designed the following experiment. First, we create a synthetic knowledge graph (Section 3.1) and train the LM on the created knowledge graph (Section 3.2). This enables us to control the knowledge stored in the LM through training precisely. Then, we apply the knowledge editing technique to the LM to delete a specific knowledge instance (Sections 3.3 and 3.4) and analyze side effects to test our hypotheses (Section 4). This experimental design allows us to conduct analyses focusing on the properties of the knowledge structure concerning knowledge deletion. 3 Experimental Setup 3.1 Knowledge Graphs This work deals with relational knowledge represented in triples, such as (s, r, o). We refer to a representation where the subject s and object o correspond to vertices, and the relation r corresponds to an edge, as a knowledge graph. Here, s and o are elements of the entity set ℰEE with |ℰ|=200ℰ200|E|=200| E | = 200, and r is an element of the relation set ℛRR with |ℛ|=50ℛ50|R|=50| R | = 50. Previous studies on knowledge editing and deletion have assumed the existence of knowledge graphs composed of relational knowledge between entities expressed in natural language. This study introduces an innovative approach by creating a synthetic knowledge graph, allowing for precise control over the information LMs acquire during training. We created two synthetic knowledge graphs with different characteristics. The first is an Erdős-Rényi (ER) graph (Erdös and Rényi, 1959), structured to ensure the probability of forming edges between vertices is uniform. The second is a Barabási-Albert (BA) graph (Barabasi and Albert, 1999), characterized by a vertex degree distribution that follows a power law, thus resembling the structure of the real world more closely (Figure 2). One advantage of synthetic knowledge graphs is that, for instance, it is possible to create entities with extremely high or low degrees, enabling the analysis of knowledge deletion for such entities in the BA graph. Figure 2: Two synthetic knowledge graphs we created. (Left) Erdős-Rényi graph: Features a relatively uniform degree distribution of vertices, representing a simple structure. (Right) Barabási-Albert graph: The degree distribution of vertices follows a power law, reflecting the properties of complex networks in the real world. 3.2 Storing Knowledge Graphs in LMs We train LMs using synthetic knowledge graphs. With the knowledge graphs created in Section 3.1, we first assign five names to each entity eisuperscripte^ieitalic_i (0≤i<|ℰ|;i∈ℕ)formulae-sequence0ℰℕ(0≤ i<|E|\,;i )( 0 ≤ i < | E | ; i ∈ blackboard_N ), denoted as ejisubscriptsuperscripte^i_jeitalic_iitalic_j (0≤j<5;j∈ℕ)formulae-sequence05ℕ(0≤ j<5\,;j )( 0 ≤ j < 5 ; j ∈ blackboard_N ). Similarly, we assign five names to each relation risuperscriptr^iritalic_i (0≤i<|ℛ|;i∈ℕ)formulae-sequence0ℛℕ(0≤ i<|R|\,;i )( 0 ≤ i < | R | ; i ∈ blackboard_N ), denoted as rjisubscriptsuperscriptr^i_jritalic_iitalic_j (0≤j<5;j∈ℕ)formulae-sequence05ℕ(0≤ j<5\,;j )( 0 ≤ j < 5 ; j ∈ blackboard_N ). The LMs include these names in their vocabulary. Hereafter, names referring to the same entity (or relation) are termed “paraphrases.” We then create a corpus composed of sentences with three words (e.g., “e00subscriptsuperscript00e^0_0e00 r11subscriptsuperscript11r^1_1r11 e41subscriptsuperscript14e^1_4e14”) and train models with GPT-2 (Radford et al., 2019) architecture, initialized with 6, 12, and 24 layers. During inference, we input two words into the model, and the model predicts one word. A prediction is correct if the outputted word represents any paraphrases indicating the gold entity. During training, we use 20%percent2020\%20 % of the entire knowledge base, which includes paraphrased knowledge, intending to achieve generalization across all paraphrased knowledge (for further details, refer to Appendix A). Then, we use the entire knowledge as full data to verify whether the model has successfully learned the knowledge graph. After training, the model achieves an accuracy rate of approximately 99%percent9999\%99 % not only on the training data but also on the full data, indicating that it has successfully memorized the provided knowledge (see Table 1). Additionally, principal component analysis on the embeddings of entities and relations in the trained model suggests that the model recognizes the representations of paraphrases (Figure 3). Knowledge Graph Model Accuracy on Training Data Accuracy on Full Data ER 06 layers .9998 .9941 12 layers .9985 .9983 24 layers .9900 .9905 BA 06 layers .9991 .9987 12 layers .9991 .9989 24 layers .9972 .9964 Table 1: Accuracy of training data and full data for each model after training the RA graph and the BA graph. Figure 3: Results of the principal component analysis (PCA) on the embedding representations in the LM after training ER graphs. The left side represents the PCA results for entity embeddings, while the right represents the PCA results for relation embeddings. Each entity and relation has five paraphrases and paraphrases about the same entity or relation are illustrated in the same color (here, paraphrases of six entities and relations are highlighted). The PCA results indicate that the embeddings of paraphrases cluster together, suggesting that the LM recognizes paraphrases. 3.3 Knowledge Editing While our experimental design is agnostic to knowledge editing and deletion methods, here we opt for ROME (Meng et al., 2022), one of the primary knowledge editing methods for causal LMs. ROME updates model weight through the following two steps to achieve the editing of a specific knowledge instance. Step 1: Causal Tracing First, we identify the model components that play a crucial role in the process of knowledge association by the model. This is achieved by analyzing the contribution of each hidden state of the model when predicting information related to knowledge111Here, predicting o for an input x=(s,r)x=(s,r)x = ( s , r ) is referred to. (for details, see Appendix B). The results of Causal Tracing revealed that the initial Feed-Forward (F) layers significantly contribute to knowledge association, supporting research indicating that F layers serve as key-value memory storage for knowledge (Geva et al., 2021). Step 2: Rank-One Model Editing During editing, we add a rank-1 matrix to the weights W2subscript2W_2W2 of the second layer in the F layer (Equation 1) identified by Causal Tracing. FFN⁢(x)=σ⁢(x⁢W1+b1)⁢W2+b2FFNsubscript1subscript1subscript2subscript2 (x)=σ(xW_1+b_1)W_2+b_2FFN ( x ) = σ ( x W1 + b1 ) W2 + b2 (1) Specifically, we treat W2subscript2W_2W2 as an associative memory for existing key-value pairs (K,V)(K,V)( K , V ), and perform edits to insert a new key-value pair (k∗,v∗)subscriptsubscript(k_*,v_*)( k∗ , v∗ )222K=[k1⁢|k2|⁢…]delimited-[]subscript1subscript2…K=[k_1|k_2|…]K = [ k1 | k2 | … ] and V=[v1⁢|v2|⁢…]delimited-[]subscript1subscript2…V=[v_1|v_2|…]V = [ v1 | v2 | … ], where k and v represent vectors.. This reduces to solve a constrained least-squares problem, and the updated weight W^2subscript^2 W_2over start_ARG W end_ARG2 is given as follows. W^2=W2+Λ⁢(C−1⁢k∗)⊤subscript^2subscript2Λsuperscriptsuperscript1subscripttop W_2=W_2+ (C^-1k_*) over start_ARG W end_ARG2 = W2 + Λ ( C- 1 k∗ )⊤ (2) Here, C−1=K⁢K⊤superscript1superscripttopC^-1=K C- 1 = K K⊤ represents the uncentered covariance of K, and Λ Λ is a vector proportional to the residual error of the new key-value pair (k∗,v∗)subscriptsubscript(k_*,v_*)( k∗ , v∗ ). 3.4 Knowledge Deletion from LMs Generally, knowledge editing in LMs refers to reassociating an entity with a different entity from one already connected to. This means updating a knowledge instance from (s,r,o)(s,r,o)( s , r , o ) to (s,r,o∗)superscript(s,r,o^*)( s , r , o∗ ). In this work, we define knowledge deletion as the process of reassociating an entity with a different entity from one it is already connected to, by introducing an entity edeletedsubscriptdeletede_deletededeleted to represent deletion. By treating edeletedsubscriptdeletede_deletededeleted entity as o∗superscripto^*o∗, we achieve knowledge deletion using a pre-existing knowledge editing method. 4 Experiment Procedure We analyze the side effects of deleting a specific knowledge instance. This means measuring the impact of deleting a knowledge instance on the overall set of knowledge instances. The impact is determined as follows: 1. Measure the accuracy of knowledge at the end of pre-training, denoted as accpre-delsubscriptaccpre-delacc_pre-delaccpre-del. 2. Delete one piece of knowledge related to a subject s. This means editing the knowledge from (s,r,o)(s,r,o)( s , r , o ) to (s,r,edeleted)subscriptdeleted(s,r,e_deleted)( s , r , edeleted ) to delete the knowledge instance. 3. After deleting the knowledge instance related to the subject s, measure the accuracy, accpost-del⁢(s)subscriptaccpost-delacc_post-del(s)accpost-del ( s ), for all knowledge instances in the model. 4. We calculate the impact I⁢(s)I(s)I ( s ) as the difference in accuracy before and after deleting a knowledge instance related to the subject s. I⁢(s)=accpre-del−accpost-del⁢(s)subscriptaccpre-delsubscriptaccpost-del I(s)=acc_pre-del-acc_post-del(s)I ( s ) = accpre-del - accpost-del ( s ) (3) Figure 4: The relationship between the degree of entities, or subjects, and the impact of their deletion in a 6-layer GPT model trained on the knowledge graph. The left vertical axis indicates the degree of entities, while the horizontal axis represents the corresponding entities. The right vertical axis shows the impact (amount of side effects) on other knowledge when deleting a knowledge instance related to an entity. When it comes to deleting knowledge related to a specific entity, we have observed that there is no relationship between the degree (i.e., number of connections) of the entity in the ER graph and the impact of its deletion. However, in the BA graph, there is a clear relationship between the degree of the entity and the impact of its deletion. Since the impact of deleting knowledge in LMs can have significant side effects on related knowledge, it is recommended to avoid deleting knowledge related to frequent entities in LMs trained on knowledge structures that are closer to the real world, as doing so may have catastrophic consequences. Results and Discussion Figure 4 shows the relationship between the degree of entities and the impact of deleting the knowledge instance related to the entity in the 6-layer model trained on the knowledge graphs. From Figure 4, it can be seen that, in the case of the ER graph, there is no relationship between the degree of entities and the impact. On the other hand, in the case of the BA graph, a relationship between the degree of entities and the impact is observed. This means that in the model trained on the ER graph, which has structural properties different from the real world, there is no significant difference in the impact of deleting knowledge related to each entity. However, in the model trained on the BA graph, which has structural properties closer to the real world, the deletion of knowledge related to entities with higher degrees has a more significant impact. In comparison, the deletion of knowledge related to entities with lower degrees has less impact. Although the absolute value of the impact is small, given the vast amount of knowledge stored in actual language models, the overall impact on the entire knowledge base can be considered significant. The impact indicates the extent of side effects on other knowledge caused by deleting knowledge related to a specific entity. Therefore, as our hypothesis, the deletion of important knowledge (related to entities with higher degrees) results in significant side effects. In comparison, the deletion of less important knowledge (related to entities with lower degrees) leads to minor side effects. Additionally, as part of an analysis that includes models with different numbers of layers, Table 2 presents the Pearson correlation coefficient between the degree of entities and their impact in each model trained on the knowledge graphs. Table 3 shows the statistical values of the impact of entities in each model trained on the knowledge graphs. From Table 2 and Table 3, it is observed that for models with 12 and 24 layers, similar to the 6-layer model, the correlation coefficient values between the degree of entities and the impact are small for the ER graph and significant for the BA graph. This indicates that the behavior of LMs in knowledge deletion varies depending on the structural properties of the knowledge being trained, suggesting the importance of analysis focused on the structural properties of knowledge. 5 Conclusion This paper analyzes the deletion of knowledge in LMs trained on a synthetic knowledge graph, revealing that the side effects of deleting knowledge related to popular entities can be catastrophic. Furthermore, it is demonstrated that LMs trained on synthetic knowledge graphs recognize paraphrases of entities and relationships, as well as relational knowledge between entities, similar to models trained on real-world knowledge. This suggests the effectiveness of studies that employ synthetic knowledge graphs to control the knowledge trained by LMs and to analyze the deletion of knowledge in LMs. Knowledge Graph Model Correlation Coefficient ER 06 layers .2093 12 layers .2634 24 layers .3045 BA 06 layers .9348 12 layers .9892 24 layers .7540 Table 2: Pearson correlation coefficients between the degree of entities and the impact of their deletion in models trained on the knowledge graph. Future work will extend the analysis of the relationship between the degree of entities involved in the knowledge deletion and the side effects across different structures of synthetic knowledge graphs and knowledge editing methods. In this study, we utilized ROME, a method of knowledge editing, to analyze the deletion of knowledge. Still, since ROME is originally a method of editing knowledge, it is essential to conduct a detailed analysis to see if any traces of the deleted knowledge remain. Additionally, focusing on the challenges associated with completely deleting knowledge, using synthetic knowledge graphs to control the knowledge in LMs, and analyzing the conditions under which knowledge can be deleted presents a compelling area of research. Unraveling what constitutes the state of “knowledge being deleted” in LMs and how to delete knowledge genuinely is deemed highly significant. Furthermore, while we validated using synthetic knowledge graphs in this work, conducting analyses with knowledge graphs represented in natural language is also essential. Knowledge Graph Model Max Min Mean SD ER 06 layers 0.20 0.00 0.05 0.04 12 layers 0.20 0.00 0.07 0.04 24 layers 0.21 0.00 0.07 0.05 BA 06 layers 0.60 0.00 0.05 0.08 12 layers 1.08 0.00 0.09 0.18 24 layers 1.13 0.00 0.09 0.17 Table 3: Statistical values of the impact I (in units of 10−2superscript10210^-210- 2) of deleting a knowledge instance related to an entity in models trained on the knowledge graph. Acknowledgements This work was supported by JSPS KAKENHI Grant Number JP22H00524, JP21K21343, JP21K17814. References Barabasi and Albert (1999) Albert-Laszlo Barabasi and Reka Albert. 1999. Albert, r.: Emergence of scaling in random networks. science 286, 509-512. Science (New York, N.Y.), 286:509–12. Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Erdös and Rényi (1959) P. Erdös and A. Rényi. 1959. On random graphs i. Publicationes Mathematicae Debrecen. Feng et al. (2023) Zhangyin Feng, Weitao Ma, Weijiang Yu, Lei Huang, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. Trends in integration of knowledge and large language models: A survey and taxonomy of methods, benchmarks, and applications. arXiv preprint arXiv:2311.05876. Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Heinzerling and Inui (2021) Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Huang et al. (2022) Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022. Ishibashi and Shimodaira (2023) Yoichi Ishibashi and Hidetoshi Shimodaira. 2023. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852. Jang et al. (2023) Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics. Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. RealTime QA: What’s the answer right now? arXiv preprint arXiv:arXiv:2207.13332. Li et al. (2023) Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2023. Pmet: Precise model editing in a transformer. arXiv preprint arXiv:2308.08742. Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems. Curran Associates, Inc. Meng et al. (2023) Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR). Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Zhang et al. (2024) Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. 2024. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286. Figure 5: PCA results of the embedding representations in the LM before training the ER graph. Appendix A Recognition of Paraphrased Representation in LMs As indicated in Section 3.2, the synthetic knowledge graphs created in this study possess a structure where each entity and relation has paraphrases, aiming to replicate real-world scenarios more closely. When training the model on the knowledge graphs with such a structure, we utilized a set of sampled knowledge instances for the training data. This approach is based on the consideration that if the LM learned all relational knowledge, it might result in the LM merely memorizing each instance of relational knowledge, thereby hindering its generalization ability to recognize paraphrased expressions. The embedding representations in Figure 3 from Section 3.2 show the results of training the model by sampling 20%percent2020\%20 % of all knowledge instances. It demonstrates that the embedding representations of paraphrases for each entity and relation cluster together. On the other hand, in the model before training, the embeddings of paraphrases for each entity and relationship are dispersed (Figure 5). This indicates that LMs are not inherently capable of recognizing paraphrases; instead, they acquire the ability to recognize them through learning. Figure 6: Example of analysis results for the contribution of F layers in each layer by Causal Tracing (in the case of a 12-layer model). The horizontal axis represents the layers of the LM, and the vertical axis corresponds to each input token. The values indicate the difference in the probability of generating the correct token before and after the corrupted-with-restoration run. The darker areas indicate that restoring the corresponding F layers allows the model to generate the correct token again, demonstrating that those F layers contribute to knowledge prediction. Appendix B Supplemental Information on the Knowledge Editing Method In Section 3.3, we briefly introduced ROME, an existing knowledge editing method used in our experiments. This section provides a more detailed explanation of Causal Tracing, a step crucial for locating the parts that play a significant role when the LM associates knowledge. Causal Tracing analyzes the contribution of each hidden state of the LM during inference through the following procedure333Manipulations to the hidden states of LMs can also be conceptualized as to the F layers or attention layers.. 1. clean run: First, predictions related to knowledge are made to obtain the normally hidden states of the model. Specifically, all hidden states hi(l)|i∈[1,T],l∈[1,L]conditional-setsuperscriptsubscriptℎformulae-sequence11\h_i^(l)\,|\,i∈[1,T],l∈[1,L]\ hitalic_i( l ) | i ∈ [ 1 , T ] , l ∈ [ 1 , L ] are determined when the model predicts o from input x=(s,r)x=(s,r)x = ( s , r ). Here, T is the length of the input x (in this work, T=22T=2T = 2), and L is the number of layers in the model. 2. corrupted run: Next, when making predictions related to knowledge, the hidden states of the corrupted model are determined by hiding information about the subject. Specifically, when input x is provided, noise is added to the embedding representation h1(0)superscriptsubscriptℎ10h_1^(0)h1( 0 ) corresponding to the subject (h1(0):=h1(0)+ϵassignsuperscriptsubscriptℎ10superscriptsubscriptℎ10italic-ϵh_1^(0):=h_1^(0)+ 1( 0 ) := h1( 0 ) + ϵ). Afterward, predictions related to knowledge are made, and the corrupted hidden states hi⁣∗(l)|i∈[1,T],l∈[1,L]conditional-setsuperscriptsubscriptℎformulae-sequence11\h_i*^(l)\,|\,i∈[1,T],l∈[1,L]\ hitalic_i ∗( l ) | i ∈ [ 1 , T ] , l ∈ [ 1 , L ] are determined. As a result, the correct output that could be output during the clean run can no longer be output during the corrupted run. 3. corrupted-with-restoration run: Finally, for the model with the corrupted hidden states obtained from the corrupted run, specific hidden states hi⁣∗(l)superscriptsubscriptℎh_i*^(l)hitalic_i ∗( l ) are restored to the normally hidden states hi(l)superscriptsubscriptℎh_i^(l)hitalic_i( l ) obtained during the clean run. This process is performed for each hidden state individually, and predictions related to knowledge are made. When the correct output can be output again by restoring a specific hidden state, it indicates that the hidden state contributes to knowledge prediction. Figure 6 presents an example result of analyzing the contributions of the F layers at each layer using Causal Tracing. In this case, the F layer in the first layer plays a significant role when the LM outputs the correct token e010superscriptsubscript010e_0^10e010 from the input token sequence (e00,r01)superscriptsubscript00superscriptsubscript01(e_0^0,r_0^1)( e00 , r01 ). Thus, Causal Tracing facilitates the identification of parts that play crucial roles in the LM’s knowledge prediction.

Instant research discovery