Paper deep dive

Representation Interventions Enable Lifelong Unstructured Knowledge Control

Xuyuan Liu, Zhengzhang Chen, Xinshuai Dong, Yanchi Liu, Xujiang Zhao, Shengyu Chen, Haoyu Wang, Yujun Yan, Haifeng Chen

Year: 2025Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 78

Models: LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/11/2026, 12:53:32 AM

Summary

RILKE (Representation Intervention for Lifelong KnowledgE Control) is a framework for updating LLM knowledge by performing interventions in the model's hidden representation space. It utilizes semantic locality and shared-subspace properties to enable paraphrase-robust, scalable, and lifelong knowledge control without retraining base weights, employing a query-adaptive router to manage edit interference.

Entities (5)

RILKE · method · 100%LLaMA · language-model · 95%Qwen · language-model · 95%ReFT · technique · 95%UnKE · dataset · 90%

Relation Signals (4)

RILKE → appliedto → LLaMA

confidence 95% · Across LLaMA and Qwen models, RILKE scales effectively to large-scale benchmarks.

RILKE → intervenesin → Representation Space

confidence 95% · RILKE treats knowledge control as interventions within the model's representation space.

RILKE → uses → Query-Adaptive Router

confidence 95% · At inference, a query-adaptive router selects the appropriate module to guide the model's generation.

ReFT → basisfor → RILKE

confidence 90% · Building on the linear representation hypothesis... ReFT specifies a low-dimensional intervention subspace.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge. This problem is particularly challenging for complex, unstructured knowledge in lifelong settings, where many edits must coexist without interference. We introduce RILKE (Representation Intervention for Lifelong KnowledgE Control), a robust and scalable method that treats knowledge control as interventions within the model's representation space. Leveraging representation-space expressiveness, we identify two key properties enabling RILKE to achieve fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference. At inference, a query-adaptive router selects the appropriate module to guide the model's generation. Across LLaMA and Qwen models, RILKE scales effectively to large-scale benchmarks, demonstrating high edit success and strong paraphrase generalization while preserving general utility with modest memory overhead. These results show RILKE is an effective and scalable solution for lifelong knowledge control in LLMs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

77,808 characters extracted from source content.

Expand or collapse full text

Representation Interventions Enable Lifelong Unstructured Knowledge Control Xuyuan Liu * 1,2 , Zhengzhang Chen †2 , Xinshuai Dong 3 , Yanchi Liu 2 , Xujiang Zhao 2 , Shengyu Chen 2 , Haoyu Wang 2 , Yujun Yan 1 , Haifeng Chen 2 1 Dartmouth College 2 NEC Laboratories America 3 Carnegie Mellon University Abstract Large language models (LLMs) often pro- duce incorrect or outdated content.Up- dating their knowledge efficiently and ac- curately without costly retraining is a ma- jor challenge. This problem is particularly challenging for complex, unstructured knowl- edge in lifelong settings, where many edits must coexist without interference. We intro- duceRILKE(RepresentationIntervention for LifelongKnowledgEControl), a robust and scalable method that treats knowledge control as interventions within the model’s representa- tion space. Leveraging representation-space expressiveness, we identify two key proper- ties enablingRILKEto achieve fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. During training,RILKElearns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference. At inference, a query-adaptive router selects the appropriate module to guide the model’s generation. Across LLaMA and Qwen mod- els,RILKEscales effectively to large-scale benchmarks, demonstrating high edit success and strong paraphrase generalization while pre- serving general utility with modest memory overhead. These results showRILKEis an ef- fective and scalable solution for lifelong knowl- edge control in LLMs. 1 Introduction Large language models (LLMs) excel at knowledge-intensive NLP tasks (Madaan et al., 2022; Sun et al., 2024), yet their knowledge is static. Once deployed, they cannot adapt to evolving real-world information, often producing outdated or inaccurate content. While methods like * Work done during internship at NEC Laboratories Amer- ica. † Correspondence to: Zhengzhang Chen <zchen@nec- labs.com> full retraining or continual pretraining can update a model, they are computationally prohibitive and prone to catastrophic forgetting (Ke et al., 2023; Yildiz et al., 2025).As an alternative, Retrieval-Augmented Generation (RAG) injects new facts at inference time (Chen et al., 2024); however, it suffers from conflicts with the model’s parametric memory (Li et al., 2025; Gutiérrez et al., 2025) and is sensitive to retrieval quality (Salemi and Zamani, 2024). These challenges underscore the need for lifelong model editing: methods to precisely and efficiently update LLM knowledge with minimal side effects (Thede et al., 2025). However, existing editing methods struggle in lifelong settings, especially when addressing un- structured, free-form knowledge that cannot be reduced to simple factual triplets (i.e., Subject– Relation–Object) (Deng et al., 2025). Paramet- ric approaches, which modify model weights di- rectly (Fang et al., 2025), suffer from edit collapse: as edits accumulate, performance degrades until the model becomes unusable (Yang et al., 2024b; Nishi et al., 2025)—a degradation particularly severe for unstructured knowledge. Similarly, although effec- tive for accumulation, external-memory methods struggle to capture the nuances of complex infor- mation, as their capacity is constrained by learning on the weight space of a single sub-module (e.g., an additional FFN layer, as in Wang et al. (2024)). In contrast to modifying model weights, inter- ventions in the representation space—the hidden states where information is processed—offer a pow- erful substrate for knowledge control due to their rich semantic structure, yet this space remains largely underexplored (Elhage et al., 2021; Wu et al., 2024b). To our knowledge, the only related work on representation-space editing is limited to structured knowledge and supports only single, in- stantaneous edits (Liu et al., 2025a). It fails to leverage the representation space’s potential for complex, unstructured knowledge and cannot han- 1 arXiv:2511.20892v2 [cs.AI] 6 Jan 2026 dle lifelong settings where edits must accumulate and coexist without interference. In this work, we unlock the potential of representation-space interventions for precise, paraphrase-robust, and lifelong knowledge control. We introduceRILKE(RepresentationIntervention forLifelongKnowledgEControl), a framework that controls LLM behavior by intervening in the model’s hidden representations. We first identify and validate two key geometric properties of these representations that enable localized and scalable editing. Building on this, we develop a robust train- ing strategy to ensure that edits generalize across paraphrases. For lifelong learning, we design a dynamic routing mechanism that activates the cor- rect intervention at inference time, mitigating in- terference between edits. Finally, we introduce a shared-subspace intervention that clusters similar edits into a single module, enabling grouped con- trol and significantly improving memory efficiency. Across various models and benchmarks,RILKE provides a stable, lightweight, and interpretable solution for lifelong knowledge control in LLMs. We summarize our contributions as follows: • We propose a robust training scheme that achieves precise, generalizable control through LLM representation-space interventions. •We propose a dynamic router that selectively ac- tivates relevant interventions while preserving unrelated knowledge and overall utility. •We develop a shared-subspace intervention strat- egy for memory-efficient, scalable control over large, growing sets of knowledge. 2 Preliminaries This section introduces the notation used through- out the paper, along with the core concepts of LLMs and the Representation Fine-Tuning (ReFT). Consider a token sequencex = (x 1 ,...,x n ), where eachx i is an element of a vocabularyV. An LLM parameterized byθdefines a joint probability over x through an auto-regressive factorization: p θ (x) = n Y i=1 p θ (x i | x <i ), x <i = (x 1 ,...,x i−1 ) wherep θ (x i | x <i )denotes the model’s predictive distribution overVfor the token at positioni, con- ditioned on the prefixx <i . For anL-layer model, leth l,i be the hidden representation at positioniin layerl. The next token probability distribution is obtained by applying a softmax function to the lin- ear projection of the hidden state of the final layer h L,i , using a weight matrix W: p θ (x i | x <i ) = softmax Wh L,i . To generate a sequencex, the LLM iteratively computesp θ (x i | x <i ), samples a tokenx i at posi- tioni, and appends it to the context for the next step. This process terminates upon generating a desig- nated end-of-sequence token or when a predefined maximum length is reached. Representation Fine-Tuning (ReFT)(Wu et al., 2024b) is a fine-tuning approach for LLMs that, instead of updating model weights, learns an inter- vention module to steer intermediate hidden states so they produce the target outputs, leaving the orig- inal weights unchanged. Building on the linear rep- resentation hypothesis (Mikolov et al., 2013; Park et al., 2024) that concepts are encoded in linear sub- spaces of the hidden space, ReFT specifies a low- dimensional intervention subspace at layerlvia R l ∈ R r×d (r ≪ d), whose rows are constrained to be orthonormal. Given the original layer-lhidden stateh l,i ∈ R d fori-th token, ReFT first computes a subspace-local shift(A l h l,i +b l −R l h l,i ), maps it back to the original space withR l⊤ , and adds the result to initial representation h l,i , yielding Φ(h l,i ;φ l ) = h l,i + R l⊤ (A l h l,i + b l − R l h l,i ), (1) whereφ l = (R l ,A l ,b l )denotes the learnable pa- rameters of this intervention module. Computa- tion in layers beyondlare identical to the base model: letF >l be the mapping from layerl + 1 to the last layer before the output head, the next- token distribution is formulated asp θ (x i | x <i ) = softmax WF >l (Φ(h l,i ;φ l ) ). For brevity, we omit the token and layer indices (i,l)when the context is clear. Following prior work (Zou et al., 2023), we use the hidden state of the last token to represent the entire sentence (i.e., h l x = h l,n for x of length n). 3 Motivation: Geometric Properties of LLM Hidden States LLMs’ hidden states provide a highly expressive mechanism for controlling model behavior (Zou et al., 2023; Rimsky et al., 2024), yet existing ap- proaches largely operate at a coarse granularity (e.g., concept-level control) (Arditi et al., 2024; Marks and Tegmark, 2024). Consequently, achiev- ing precise, fine-grained control over individual pieces of knowledge remains challenging, particu- larly in free-form knowledge settings that require 2 12345678 Representation Distance 0.0 0.2 0.4 0.6 0.8 1.0 Density Paraphrased Pairs Random Pairs (a) Distribution of distances between original queries and their para- phrased counterparts, compared against a baseline of random non- corresponding pairs. Paraphrased queries remain close to the originals in representation space, supporting Prop. 1. >0.90.8-0.90.7-0.80.6-0.7<0.6 Representation Similarity 0.50 0.55 0.60 0.65 0.70 0.75 0.80 Mean Task Vector Similarity 0.751 0.698 0.669 0.646 0.586 (b) MeanR l -similarity stratified by representa- tion similarity. Higher representation similarity corresponds to strongerR l -alignment, indicating a shared intervention subspace (Prop. 2). Figure 1: Two key properties in LLMs’ representation space. We show that these properties can be utilized to facilitate generalizable, lifelong, and scalable knowledge control in LLMs. long and accurate generations. In such settings, ef- fective free-form knowledge control must address three key challenges: (i) catastrophic forgetting, ensuring that newly introduced updates do not in- terfere with previously learned knowledge; (i) gen- eralizability, requiring edits to transfer to para- phrased or semantically equivalent queries; and (i) scalability, supporting a growing number of edits with minimal additional memory overhead. To address these challenges, we characterize two geometric properties of hidden-state representa- tions. The first, termed semantic locality, captures the tendency for representations to be primarily shaped by semantic content and to remain stable under lexical variation, so long as the underlying semantics are preserved. Formally, Property 1. Leth l x denote the layer-lrepresenta- tion of a queryx. For any paraphrase ˆ xofxand any semantically unrelated queryx ′ , the following holds: ∥h l ˆ x − h l x ∥ 2 <∥h l x ′ − h l x ∥ 2 This property provides a mechanism to mitigate catastrophic forgetting; by exploiting represen- tation similarity to route queries to disentangled knowledge modules, the system isolates updates and ensures that only the relevant module is acti- vated during inference. Moreover, semantic local- ity supports generalizability: because paraphrases remain proximal in representation space, an edit ap- plied to the neighborhood of the original query nat- urally transfers to them. Notably, this behavior is non-trivial—it is driven primarily by semantic sim- ilarity rather than token-level overlap, and remains robust even when lexical features are misleading, as shown in App. A.1. Building on this observation, we further uncover a connection between query semantics and the learned intervention function space. Specifically, Property 2. Letx i andx j be two semantically re- lated queries. Even when ReFT (Eq.(1)) is trained independently for each query, their learned layer-l intervention subspaces remain aligned; that is, the corresponding projection matricesR l x i andR l x j are highly similar. In other words, semantically related knowl- edge tends to concentrate within a shared low- dimensional subspace. This structure allows a sin- gle module to identify such a subspace and effi- ciently store and apply multiple semantically sim- ilar edits within it, thereby improving scalability without increasing interference across edits. We empirically validate these properties. Fig. 1a corroborates Prop. 1 by showing a clear separa- tion between theℓ 2 -distance distributions of para- phrased query pairs and those of randomly paired queries; moreover, this separation remains robust under substantial lexical variation. For Prop. 2, we analyze the relationship between representation similarity and the alignment of learned interven- tion subspaces. Specifically, we compute the cosine similarity between subspace projection matricesR l (see Eq.(1)) within groups stratified by representa- tion similarity, and observe that higher representa- tion similarity consistently corresponds to stronger subspace alignment (Fig. 1b). These results support the feasibility of controlling semantically related knowledge through grouped interventions. 4 Method: Towards Generalizable, Scalable, Lifelong Knowledge Control To address the challenges of catastrophic forget- ting, generalizability, and scalability in knowledge 3 Figure 2: Overview of the RILKE framework. During training, the intervention moduleΦmapsh ori to the targeth edit , with h ori stored as the knowledge index. During inference, the router selects the intervention moduleΦwhose index is closest to the input query’s representation to perform a targeted edit, enabling the model to generate the desired output. editing, we introduce RILKE (Representation Intervention for Lifelong KnowlEdge Control). RILKE achieves lifelong knowledge control through a modular approach: during training, RILKE mitigates forgetting by isolating updates within dedicated intervention modules. At infer- ence, a query-adaptive router ensures precise gen- eration by activating the appropriate module only when relevant. Finally, to scale this approach, we incorporate a batched training scheme that clus- ters semantically similar edits into shared modules, thereby improving memory efficiency. 4.1 Consistency-Robust Training for Generalizable Knowledge Control Knowledge-editing methods always struggle with generalizability: while an edit may successfully up- date a specific query, it often fails on paraphrases of that same query. Consider an unstructured knowl- edge pair(x, y)comprising an input queryxand a target responsey. Vanilla ReFT exemplifies this issue, as it learns an interventionφ l x strictly con- ditioned on the representationh l x of the original input. Consequently, a semantically equivalent paraphrase ˆ x , which yields a proximal yet distinct representation h l ˆ x , may fail to trigger the edit. Building on Prop. 1 and Eq. (1), we assume the layer-lrepresentation of a paraphrased query ˆ xresides within anε-ball centered ath l x , i.e.,∥h l ˆ x − h l x ∥ 2 ≤ ε. We therefore impose a consistent intervention target throughout this region. Applying a first-order expansion to the ReFT map with respect to the deviationεyields Φ(h l x + ε;φ l x ) = Φ(h l x ;φ l x ) + I + R l⊤ x (A l x − R l x ) ε implying that the variation induced by the para- phrase is governed by I + R l⊤ x (A l x − R l x ) ε . To mitigate this effect, we enforce consistency of the final vocabulary distribution within the ball. We can see the edited predictive distribution for initial representation h l x is: p θ,φ l x (·| x) = softmax WF >l ( Φ(h l x ;φ l x )) . Then, we drawε∼Q(e.g.,N (0,σ 2 I)) onh l x and form a perturbed branch: p (ε) θ,φ l x (·| x) = softmax WF >l ( Φ(h l x +ε;φ l x )) . We penalize the discrepancy between the two dis- tributions using the KL divergence: L robu (x;φ l x ) = KL p θ,φ l x (·| x) p (ε) θ,φ l x (·| x) . We incorporate this robustness term into the stan- dard language modeling objective, which mini- mizes cross-entropy loss in a teacher-forcing man- ner, as in vanilla ReFT. The general objective for updating the knowledge item(x, y)with a single- knowledge intervention module is, therefore, L(φ l x ) =− |y| X i=1 logp θ,φ l x (y i | x, y <i ) + λ robu E ε∼Q L robu (x;φ l x ) , (2) where regularization promotes the generalization of edited knowledge to paraphrased queries. 4.2 Query-Adaptive Routing for Lifelong Knowledge Control To mitigate catastrophic forgetting, we freeze the base model and train a dedicated intervention mod- uleφ l x using the method in Sec. 4.1 for each knowl- edge instancex. While this effectively isolates 4 edits, it introduces a new challenge at inference: the model must select and then apply the appropri- ate intervention for a given prompt. To address this, we leverage Prop. 1 to design a router that directs incoming queries to their corresponding modules. Specifically, we construct a routing index h l x j m j=1 using the layer-lrepresentations of all mtraining examples in the edit dataset. Since the base model is frozen and interventions function only beyond layerl, these representations will not be affected by the training of interventions and act as a stable key space. Each index is then linked to its specific trained interventionφ l x j m j=1 . At infer- ence, given a query ˆ xwith keyh l ˆ x , the routerπ(·) identifies the stored representationx j that maxi- mizes the cosine similarity with the query: π(h l ˆ x ) = arg max x j , 1≤j≤m D h l ˆ x , h l x j E ∥h l ˆ x ∥ 2 ∥h l x j ∥ 2 . Subsequently, RILKE applies the intervention Φ(h l ˆ x ;φ l π(h l ˆ x ) )to enact the targeted edit, provided that the maximum similarity score exceeds the pre- defined relevance thresholdτ sim . If this condition is not met, no intervention is performed. Importantly, with this strategy, each training query is deterministically assigned to its corre- sponding intervention module, as its key exactly matches the targeted index. For unseen queries (e.g., paraphrased queries), we observe that most are still routed to the correct module even after large-scale editing, further supporting Prop. 1. 4.3 Shared Subspace Intervention for Memory-efficient Management Lifelong settings require scalable editing of mas- sive knowledge bases. While assigning a dedicated intervention module to each knowledge instance affords fine-grained control, this approach scales poorly, incurring a memory cost that grows linearly with the number of edits and creating a significant bottleneck during training. To address this scalabil- ity challenge, we build on the insight from Prop. 2, which establishes that semantically related edits can share a common intervention subspace. We therefore propose a memory-efficient alternative: cluster similar knowledge instances and train a sin- gle shared intervention module for each cluster. We partition knowledge instances into semanti- cally homogeneous, size-bounded groups to ensure that a single intervention subspace can serve each group effectively. Concretely, we impose two con- straints on every cluster: (i) a within-cluster simi- larity lower boundτ sim ∈ (0, 1)(or equivalently, a merge thresholdd thr = 1− τ sim ), and (i) a maxi- mum cluster sizes max . Letmdenote the number of knowledge items and collect their layer-lrepre- sentations intoH = h l x 1 h l x 2 · h l x m . We run hierarchical agglomerative clustering (HAC) over the columns ofHusing an initial similarity lower boundτ min to obtain provisional clusters. Any clus- ter exceedings max is then recursively refined by increasing the similarity floor and re-running HAC within that cluster. The procedure terminates when all clusters satisfy|C c |≤ s max , yieldingkclusters C c k c=1 that are both semantically coherent and size-controlled (see details in Algo. A.2). Given the clustered knowledge, we train, for each clusterC i ∈ C c , a cluster-shared interven- tion module using allx j ∈C i under the objective in Sec. 4.1, yieldingφ l C i = R l C i , A l C i , b l C i .Let κ(x j )be the mapping fromx j to its corresponding clusterC i (i.e.,κ :x→C 1 ,...,C k , κ(x j ) = C i such thatx j ∈C i ). Then, at inference, for query ˆ xwith keyh l ˆ x , routerˆπ(·)maps it to the corre- sponding intervention by identifying the cluster to which the closest knowledge item belongs: ˆπ(h l ˆ x ) = κ   arg max x j , 1≤j≤m D h l ˆ x , h l x j E ∥h l ˆ x ∥ 2 ||h l x j ∥ 2   . Finally, the corresponding cluster intervention Φ(h l ˆ x ;φ l ˆπ(h l ˆ x ) ) is applied for the target edit. Taken together, these strategies yield RILKE, which unifies (i) robust representation interventions for precise, paraphrase-generalizable knowledge control; (i) an adaptive inference-time router that activates the appropriate module; and (i) cluster- level interventions that manage semantically sim- ilar knowledge for memory-efficient scalability, thereby collectively enabling lifelong control of unstructured knowledge in LLMs. 5 Experiment We evaluate the RILKE framework on public benchmarks for unstructured knowledge control, focusing on the following research questions: RQ1: How does RILKE compare to prior meth- ods in lifelong unstructured knowledge learning? RQ2: How does RILKE leverage shared-space interventions for efficient knowledge control? RQ3: How does the design of RILKE facilitate precise and generalizable knowledge control? 5 5.1 Experimental Setup LLMs & Baselines. We evaluate the RILKE framework on popular off-the-shelf LLMs: Llama-3.1-8B-Instruct(Grattafiori etal., 2024) andQwen2.5-7B-Instruct(Yang et al., 2024a).We compare our approach against a diverse set of model editing baselines, including locate-then-edit methods (MEMIT (Meng et al., 2022a), UnKE (Deng et al., 2025), AnyEdit (Jiang et al., 2025)) and memory-based approaches (FT-L (Zhu et al., 2020), GRACE (Hartvigsen et al., 2023a), WISE (Wang et al., 2024)) (details in App. A.3). To ensure fairness, we utilize the standard configurations and implementations from EasyEdit 1 for all compared methods. Datasets & Metrics. To assess knowledge edit- ing efficacy, we utilize the UnKE (Deng et al., 2025) and EditEverything (Jiang et al., 2025) datasets. Following established protocols from prior work (Jiang et al., 2025), we set the tem- perature to 0.001 for deterministic generation. We report: (i) Lexical Similarity (Rouge-L), which cap- tures the n-gram overlap between the generated response and the reference; and (i) Semantic Simi- larity (BertScore), measured via the cosine similar- ity of sentence embeddings 2 to evaluate alignment at the semantic level. We assess both edit efficacy (performance on original training queries) and gen- eralization (performance on unseen paraphrased queries). Furthermore, to assess the preservation of general capabilities and ensure edit locality, we compare performance on MMLU (Hendrycks et al., 2021) pre- and post-edit. The difference serves as a metric for the impact of the edit on unrelated knowledge. Finally, we include results on the ZsRE (Levy et al., 2017) to benchmark performance on standard structured editing tasks. All metrics are averaged over the evaluation set. 5.2 RQ1: RILKE Enables Lifelong Control Consistent with prior work (Deng et al., 2025; Jiang et al., 2025), we evaluate lifelong knowledge con- trol using a sequential protocol with a batch size of 1 to simulate continuous updates. To mitigate spurious activations, we set the gating similarity threshold to 0.9 (as detailed in Sec. 4.2). We report edit efficacy and generalization at Steps 10, 100, and 1,000. Furthermore, to assess the retention of 1 https://github.com/zjunlp/EasyEdit 2 We use all-MiniLM-L6-v2 to align with prior work. general capabilities after extensive editing, we re- port MMLU accuracy following 1,000 edits, as pre- sented in Tab. 1. Finally, we extend our evaluation to the ZsRE dataset with 3,000 edits to benchmark performance in a structured editing setting. We find that across both models on the Un- KEBench benchmark, RILKE maintains stable per- formance as edits accumulate. In contrast, compet- ing methods begin to degrade after approximately 10 edits, whereas RILKE consistently outperforms them, exhibiting a performance gap that widens over time. Moreover, unlike prior approaches where utility degrades substantially after extensive editing, RILKE preserves general reasoning ca- pabilities, achieving near parity with the unedited base model on the MMLU. On the ZsRE dataset, for which RILKE was not explicitly designed, we also observe robust performance. Although the advantage is narrower than on UnKE—likely due to the reduced difficulty of short-form gener- ation—the margin remains distinct. These results demonstrate that RILKE’s interventions are highly localized and generalize effectively to paraphrased queries, precisely modifying intended knowledge while minimally perturbing other capabilities. Ad- ditional results on the EditEverything are presented in App. A.4, and we also report routing stability as the dataset size increases in App. A.7. 5.3 RQ2: RILKE Enables Memory-efficient Control via Shared-space Intervention Following the protocol in Sec. 4.3, we apply the shared-subspace strategy by first clustering the dataset based on layer-lhidden states. We use a similarity thresholdτ sim = 0.9, consistent with the criterion for identifying unrelated knowledge estab- lished in Sec. 5.2. Subsequently, we train a single shared intervention for each cluster by processing all assigned instances in a joint batch. Table 2: Storage costs for UnKE onLlama-3.1-8B-Instruct. RILKEachievessignificantly lower storage costs in both settings, underscoring its efficiency. MethodStorage Cost WISE224.0 MiB RILKE (Individual)96.1 MiB RILKE (Shared)29.4 MiB Tab. 2 details the storage over- headassociated withvarious memory-based editingmethods. Unlike prior meth- ods that typically fine-tune and store entire sub-modules for new knowledge, RILKE operates via a low-rank intervention head in the representation space. This design yields substantial memory savings, minimizing storage costs even 6 Table 1: Results on the UnKE dataset.Tdenotes the number of sequential edits. Ori. reports performance on the original (edited) queries; Para. reports generalization to paraphrased queries; and Util. reports accuracy on MMLU. Additional results on the ZsRE dataset report reliability (Rel.), generalization (Gen.), and locality (Loc.). Method T = 10T = 100T = 1,000ZsRE(T = 3,000) Ori.Para.Ori.Para.Ori.Para.Util. Rel. Gen. Loc. Avg. BertS RougeL BertS RougeLBertS RougeL BertS RougeLBertS RougeL BertS RougeL MMLU Based on LLAMA3.1-8B-INSTRUCT0.633 FT-L0.0590.0100.0050.0700.1200.0230.1260.0200.1120.0300.1110.0310.2260.05 0.01 0.04 0.03 MEMIT0.7540.5580.7200.5710.1950.1510.1780.1530.0330.1450.0340.1420.1880.00 0.00 0.00 0.00 GRACE0.8860.8780.6500.2010.9090.7860.6050.1720.8100.7630.5210.1440.5940.46 0.01 1.00 0.49 UnKE0.6270.4420.5990.3730.2500.2020.2940.2100.0130.0800.0170.0700.1260.02 0.02 0.01 0.02 AnyEdit0.3590.2370.3550.2330.0660.0950.0490.0970.0120.1640.0050.1610.2180.01 0.01 0.00 0.01 WISE0.6690.6360.6600.6140.6720.6640.6690.5980.6810.6610.6730.6230.5840.620.601.00 0.73 RILKE1.0001.0000.9980.9901.0001.0000.9840.9421.0001.0000.9630.8820.6220.99 0.71 0.940.88 Based on QWEN2.5-7B-INSTRUCT0.713 UnKE0.8250.4300.7770.4050.6530.3750.6400.3820.0390.0730.0330.0660.1300.01 0.01 0.00 0.01 AnyEdit0.7710.4210.7610.4580.3110.1770.2870.1800.0100.1120.0070.1130.2230.02 0.02 0.00 0.02 GRACE0.9420.8860.7120.1790.8930.2640.6650.0970.9010.2620.6540.0980.6670.45 0.00 1.00 0.48 WISE0.8030.5270.7940.5040.7060.5500.7170.5030.5640.4110.5210.4010.6510.610.581.00 0.73 RILKE1.0001.0000.9590.8841.0001.0000.9350.8270.9990.9980.8930.7180.7120.98 0.70 0.860.85 when instantiating a distinct adapter for each edit. Specifically, forLlama-3.1-8B-Instruct, RILKE requires less than 43% of the storage capacity mandated by competitive baselines such as WISE. Furthermore, adopting a cluster-shared strategy reduces memory usage to≈ 30%of the standard RILKE configuration, achieving an additional∼ 3× compression. Table 3: UnKE edit success and MMLU after 1,000 sequen- tial edits under Individual vs. Cluster-shared training; Cluster- shared control incurs only a slight generalization cost. MethodOri. BertS Para. BertS MMLU RILKE (Individual)1.0000.9630.622 RILKE (Shared)0.9990.9010.621 Tab. 3 presents the performance of the shared- subspace training strategy. We observe that this approach incurs only a modest reduction in gen- eralization while preserving editing efficacy and overall utility. Crucially, it significantly reduces the parameter budget, demonstrating the efficiency of our method. A detailed analysis of the routing behavior in this setting is provided in App. A.7. 5.4 RQ3: RILKE Enables Precise and Generalizable Knowledge Control We further analyze how RILKE achieves robust generalization across paraphrases. We compare our robust training objective against a vanilla baseline that optimizes only the language-modeling loss (omittingL robu in Eq.(2)) Tab. 4 demonstrates that our robust training framework significantly improves paraphrase gen- eralization without compromising precision on original targets. This confirms that our objective Table 4: Edit efficacy (BertScore) atT =100andT =1,000 under training with and withoutL robu . The robust objective improves RILKE’s generalizability to paraphrased queries. Method T = 100T = 1,000 Ori. BertS Para. BertSOri. BertS Para. BertS w/oL robu 1.0000.9590.9990.909 wL robu 1.0000.9841.0000.963 effectively mitigates overfitting to surface forms, ensuring reliable knowledge control. For further details, we provide a detailed analysis of computa- tional costs in App. A.5 and a comprehensive study on training configurations in App. A.6. 6 Analysis: Can Shared Subspace Really Control Similar Knowledge Building on Prop. 2, which posits that semanti- cally similar edits lie in a shared low-dimensional intervention subspace, we examine how joint train- ing couples individual edits. We randomly sample knowledge items and train RILKE under three set- tings: (i) Individual: each sampled item has its own adapter; (i) Dissimilar Batched: items are randomly assigned to groups and trained jointly with a single adapter per group, simulating train- ing with unrelated knowledge; and (i) Similar Batched: each sampled item is batched with its se- mantically similar neighbors (co-clustered items). Crucially, in setting (i), because the initial random subset may not contain the necessary similar items, we retrieve neighbors from the full dataset to con- struct the training batches. While these auxiliary items are used to provide the necessary seman- tic context during training, evaluation is restricted 7 strictly to the original sampled subset. This en- sures a consistent comparison set across all three settings, allowing us to isolate the effects of joint training with dissimilar items (i) versus highly sim- ilar items (i). We analyze the learned interventions by compar- ing the distance between the edit vectorsV edit in settings (i) and (i) against setting (i). Formally, we define the edit vectors forx i asV edit,x i = R l⊤ (A l h l x i +b l −R l h l x i ) (i.e., the deviation from the original hidden state; see Eq.(1)). Since the pre-edit hidden stateh l x i is fixed across regimes becausex i remains fixed across all settings, any divergence in the resulting edit vectors stems solely from changes in the learned parametersφ l = (R l ,A l ,b l ), thereby isolating the effects of the different training strategies. In this setup, we find that training with similar items (setting i) yields edit vectorsV (i) edit,x i that are consistently closer to their individually trained counterpartsV (i) edit,x i , than those obtained by train- ing with dissimilar items (setting i). This proxim- ity holds true in 91 of the 100 sampled cases. A PCA visualization of the edit vectors for a sampled setx, shown in Fig. 3, further corroborates this trend: edit vectors from (i) remain close to those from (i), whereas those from (i) shift away. 5051015 PCA Component 1 5 0 5 PCA Component 2 (i) Individual (i) Dissimilar batched (i) Similar batched Figure 3: Visualization ofV edit under different training settings. Arrows trace the shift from individual training to batched strategies for a single data point. Training with simi- lar data preserves vector proximity, while dissimilar batching drives them away, highlighting the need to cluster similar knowledge for effective subspace control. These results highlight the effect of RILKE in mitigating interference and preserving edit speci- ficity by clustering semantically similar knowledge before applying shared-subspace intervention. 7 Related Work LLM Representation Space Analysis.Previous studies have demonstrated that LLMs encode rich semantics within their activation space (Zou et al., 2023; Marks and Tegmark, 2023; Turner et al., 2023). Core behaviors, including trustworthiness (Marks and Tegmark, 2023), refusal (Arditi et al., 2024), and reasoning (Liu et al., 2025b; Chen et al., 2025b), have been linked to specific components of the representation space. Several methods have been proposed to leverage these components: Han et al. (2024) manipulate generation style via style vectors; Chen et al. (2025a) discover persona vec- tors for customized control; and Wu et al. (2024b) introduce fine-tuning directly within the represen- tation space to enable style modulation. Knowledge Editing. Existing model editing methods typically fall into two categories: paramet- ric approaches that directly update model weights, and memory-based methods that preserve the origi- nal parameters. Meta-learning methods (Mitchell et al., 2022; Zheng et al., 2023) employ hypernet- works to predict parameter updates, while locate- then-edit techniques (Meng et al., 2022a,b) iden- tify and modify neurons responsible for the target knowledge. Fang et al. (2025) projects updates onto the null space of preserved knowledge to mitigate interference. In contrast, external memory-based approaches maintain the original parameters and instead utilize external components to overwrite ac- tivations via retrieved codebook entries (Hartvigsen et al., 2023b; Wang et al., 2025; Cheng et al., 2025). A recent advancement by Zhang et al. (2025) fur- ther employs shared memory modules to enable simultaneous editing and unlearning. Unstructured Knowledge Editing.Recent work extends knowledge editing from structured triples to unstructured free-form knowledge. Wu et al. (2024a) observes limitations in prior evaluation protocols and proposes a new benchmark. Deng et al. (2025) enhances locate-then-edit paradigms to update parameters across layers, improving their efficacy on unstructured text. Jiang et al. (2025) introduce a chunk-based auto-regressive method to enable long-form knowledge editing. While these methods address the challenges of unstruc- tured knowledge, they face scalability issues: per- formance degrades after repeated edits, and general usability diminishes as more knowledge is updated. 8 Conclusion In this work, we advance knowledge editing in LLMs from a representation-centric perspective. We begin by identifying two key properties— generalizability and locality—that underpin robust and targeted interventions. Building on these prin- ciples, we introduce RILKE, a framework that 8 enables precise, interpretable, and lifelong knowl- edge control through representation-space interven- tions. To improve scalability, we further propose a shared-subspace strategy that clusters semantically related knowledge, enabling batched updates via a single adapter. Experimental results demonstrate the reliability, scalability, and memory efficiency of RILKE for lifelong knowledge control in LLMs. 9 Limitations We present a novel framework for precise, gen- eralizable, and lightweight unstructured knowl- edge control via representation-space interventions. However, several limitations remain. First, al- though we evaluate on the largest unstructured knowledge editing dataset to our knowledge (i.e., UnKE (Deng et al., 2025)), scaling to extra-large and continually expanding knowledge bases re- mains underexplored. Second, we defer a system- atic risk analysis of knowledge control to future work. This includes potential threats such as ma- licious or adversarial edits (Li et al., 2024), unin- tended bias propagation or amplification (Cohen et al., 2024), and robustness under biased editing policies. We plan to investigate detection, mitiga- tion, and governance mechanisms to address these risks in future studies. References Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems 38: Annual Conference on Neu- ral Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xue’. 2024. Lifelong knowledge editing for LLMs with retrieval- augmented continuous prompt learning. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13565–13580, Miami, Florida, USA. Association for Computational Linguistics. Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025a. Persona vectors: Moni- toring and controlling character traits in language models. Preprint, arXiv:2507.21509. Runjin Chen, Zhenyu Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang. 2025b. Seal: Steer- able reasoning calibration of large language models for free. arXiv preprint arXiv:2504.07986. YuJu Cheng, Yu-Chu Yu, Kai-Po Chang, and Yu- Chiang Frank Wang. 2025. Serial lifelong editing via mixture of knowledge experts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30888–30903, Vienna, Austria. Association for Computational Linguistics. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evaluating the ripple effects of knowledge editing in language models. Trans. Assoc. Comput. Linguistics, 12:283–298. Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. 2025. Everything is editable: Extend knowledge editing to unstructured data in large language models. In The Thirteenth In- ternational Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. Https://transformer- circuits.pub/2021/framework/index.html. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat- Seng Chua. 2025. Alphaedit: Null-space constrained knowledge editing for language models. In The Thir- teenth International Conference on Learning Repre- sentations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key- value memories. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of mod- els. arXiv preprint arXiv:2407.21783. Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. From rag to memory: Non-parametric continual learning for large language models. arXiv preprint arXiv:2502.14802. Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek F. Abdelzaher, and Heng Ji. 2024. Word embeddings are steers for language mod- els. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2024, Bangkok, Thailand, 9 August 11-16, 2024, pages 16410–16430. Associa- tion for Computational Linguistics. Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023a. Aging with grace: Lifelong model editing with dis- crete key-value adaptors. In Advances in Neural Information Processing Systems, volume 36, pages 47934–47959. Curran Associates, Inc. Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023b. Aging with GRACE: lifelong model editing with dis- crete key-value adaptors. In Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guo- jun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat-Seng Chua. 2025. Anyedit: Edit any knowledge encoded in language models. CoRR, abs/2502.05628. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre- training of language models. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open- Review.net. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettle- moyer. 2017. Zero-shot relation extraction via read- ing comprehension. In Proceedings of the 21st Con- ference on Computational Natural Language Learn- ing (CoNLL 2017), Vancouver, Canada, August 3-4, 2017, pages 333–342. Association for Computational Linguistics. Gaotang Li, Yuzhong Chen, and Hanghang Tong. 2025. Taming knowledge conflicts in language models. In Forty-second International Conference on Machine Learning. Yanzhou Li, Tianlin Li, Kangjie Chen, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. 2024. Badedit: Backdooring large lan- guage models by model editing. In The Twelfth Inter- national Conference on Learning Representations. Tianci Liu, Ruirui Li, Yunzhe Qi, Hui Liu, Xian- feng Tang, Tianqi Zheng, Qingyu Yin, Monica Xiao Cheng, Jun Huan, Haoyu Wang, and Jing Gao. 2025a. Unlocking efficient, scalable, and continual knowl- edge editing with basis-level representation fine- tuning. In The Thirteenth International Conference on Learning Representations. Xuyuan Liu, Lei Hsiung, Yaoqing Yang, and Yujun Yan. 2025b. Spectral insights into data-oblivious criti- cal layers in large language models. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 4860–4877. Association for Computational Linguis- tics. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1384–1403. Association for Computational Linguistics. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. CoRR, abs/2310.06824. Samuel Marks and Max Tegmark. 2024. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual asso- ciations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neu- ral Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - Decem- ber 9, 2022. Kevin Meng, Arnab Sen Sharma, Alex Ando- nian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer.CoRR, abs/2210.07229. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia. Association for Computational Linguistics. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022. Fast model editing at scale. In The Tenth International Con- ference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. Kento Nishi, Maya Okawa, Rahul Ramesh, Mikail Khona, Hidenori Tanaka, and Ekdeep Singh Lubana. 2025. Representation shattering in transformers: A synthetic study with knowledge editing. 10 Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geome- try of large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steer- ing llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thai- land. Association for Computational Linguistics. Alireza Salemi and Hamed Zamani. 2024. Evaluating retrieval quality in retrieval-augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 2395–2400. ACM. Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. Head-to-tail: How knowledgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume 1: Long Pa- pers), pages 311–325, Mexico City, Mexico. Associ- ation for Computational Linguistics. Lukas Thede, Karsten Roth, Matthias Bethge, Zeynep Akata, and Thomas Hartvigsen. 2025. Wikibigedit: Understanding the limits of lifelong knowledge edit- ing in LLMs. In Forty-second International Confer- ence on Machine Learning. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language mod- els with activation engineering.arXiv preprint arXiv:2308.10248. Ke Wang, Yiming Qin, Nikolaos Dimitriadis, Alessan- dro Favero, and Pascal Frossard. 2025. MEMOIR: lifelong model editing with minimal overwrite and informed retention for llms. CoRR, abs/2506.07899. Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Hua- jun Chen. 2024. WISE: rethinking the knowledge memory for lifelong model editing of large language models. In Advances in Neural Information Pro- cessing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. Xiaobao Wu, Liangming Pan, William Yang Wang, and Anh Tuan Luu. 2024a. AKEW: assessing knowledge editing in the wild. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, Novem- ber 12-16, 2024, pages 15118–15133. Association for Computational Linguistics. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2024b. Reft: Representation fine- tuning for language models. In Advances in Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Jun- yang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. 2024a. Qwen2.5 technical report. ArXiv, abs/2412.15115. Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. 2024b. The butterfly effect of model editing: Few edits can trigger large language models collapse. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 5419–5437. Association for Computational Linguistics. Çagatay Yildiz, Nishaanth Kanna Ravichandran, Nitin Sharma, Matthias Bethge, and Beyza Ermis. 2025. Investigating continual pretraining in large language models: Insights and implications. Trans. Mach. Learn. Res., 2025. Binchi Zhang, Zhengzhang Chen, Zaiyi Zheng, Jun- dong Li, and Haifeng Chen. 2025. Resolving editing- unlearning conflicts: A knowledge codebook frame- work for large language model updating. CoRR, abs/2502.00158. Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023. Can we edit factual knowledge by in-context learning? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 4862– 4876. Association for Computational Linguistics. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. Andy Zou, Long Phan, Sarah Li Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023. Representation engineering: A top-down approach to AI transparency. CoRR, abs/2310.01405. 11 A Appendix A.1 Additional Results for Semantic Locality (Property 1) In Sec. 3, we demonstrated that hidden states ex- hibit strong semantic locality, a property wherein semantically equivalent paraphrased queries re- main close to the original query in representation space compared to irrelevant queries. This facil- itates the discrimination of paraphrased queries from unrelated inputs during inference, ensuring the selective activation of the relevant module. In this section, we further validate that this capabil- ity stems from the LLM’s semantic understanding rather than superficial lexical matching, demon- strating robustness even when lexical cues are mis- leading. To illustrate that representation similarity is driven by semantic alignment rather than token overlap, we present a qualitative analysis using samples from the UnKE dataset. We compare a Target Query against two variations: a semanti- cally equivalent Paraphrase and a Hard Nega- tive—an input sharing a nearly identical lexical structure but referring to a different entity. ◦Target: What are some of Bae Geu-rin’s no- table achievements and contributions in the fashion industry? ◦Paraphrased: What notable accomplish- ments and impacts has Bae Geu-rin made in the fashion world? ◦ Hard Negative: What are William Watson’s notable achievements and contributions in the field of medicine? In this case, although the Hard Negative exhibits significantly higher lexical overlap with the Target (ROUGE-L: 0.60 vs. 0.35), the latent representa- tion similarity is higher for the Paraphrase (0.98 vs. 0.93). This result demonstrates that our routing strategy, based on distance in the representation space, prioritizes semantic consistency over textual resemblance. Consequently, it effectively filters out irrelevant but structurally similar inputs, even in extreme scenarios. Then, we extended this analysis on the dis- tance distributions across the entire dataset (Fig. 4). We compared the distribution of paraphrased pairs against hard negative pairs. Hard negatives were identified by maximizing lexical similarity (ROUGE-L) relative to each entry, while excluding the entry itself and its ground-truth paraphrases. Although the hard negatives possess higher lexical similarity (Mean: 0.61, Median: 0.60) compared to the paraphrases (Mean: 0.51, Median: 0.52), the L2 distance in the hidden space is significantly lower for the paraphrased pairs. 123456 L2 Distance 0.0 0.2 0.4 0.6 0.8 1.0 Density Hard Negative Pairs Paraphrased Pairs Figure 4: Comparison of L2 distances for Paraphrased pairs and Hard Negative pairs. Hard negatives exhibit high lexical similarity but distinct semantics. The clear separation between the distributions indicates that para- phrases remain significantly closer in the hidden space, validating the Semantic Locality hypothesis even in chal- lenging cases where surface-level text is misleading. From these results, we observe a clear sepa- ration between the paraphrase and hard negative distributions, with paraphrase distances remaining significantly lower than those of hard negatives. This distinction indicates that distance in the hid- den space effectively distinguishes semantic para- phrases from lexically similar but irrelevant inputs, thereby validating the feasibility of our distance- based routing strategy. A.2 Cluster Method We provide details of the clustering algorithm described in Sec. 4.3, which aims to group knowl- edge items that exhibit high semantic similarity while enforcing an upper bound on cluster size. A.3 Baseline In the experiment, we compare our method with a series of previous works: FT-L (Zhu et al., 2020). All other layers of the LLM remain frozen, and only a single MLP layer is chosen to fine-tune using an autoregressive loss. LoRA (Hu et al., 2022). Building on FT-L, this method avoids updating the full MLP layer. Instead, it applies a low-rank approximation by injecting two trainable low-rank matrices into the original weights. MEMIT (Meng et al., 2022a). A direct weight- editing method that linearizes the transformer 12 Algorithm 1 Constrained HAC with Adaptive Similarity Floor and Size Bound Require: layer-l hidden-state matrix H = h l x j for x j ∈D edit , similarity threshold τ min ∈ (0, 1), step ∆> 0, cluster max size s max 1: C 0 ← HAC(H, thr = 1− τ min ), C final ←∅ 2: for all C ∈C 0 do 3: C final ←C final ∪ SPLITOVERSIZED(C, τ min ) 4: returnC final 5: procedure SPLITOVERSIZED(C, τ ) 6:if|C|≤ s max then 7:returnC 8:repeat 9:τ←τ + ∆; S← HAC(H[C], thr = 1− τ ) 10: R← S S∈S SPLITOVERSIZED(S, τ ) 11:until max T∈R |T|≤ s max 12:returnR around the target MLP modules and solves a con- strained least-squares problem for low-rank up- dates. These updates implant new (subject, relation →object) tuples while minimizing interference with unrelated knowledge. GRACE (Hartvigsen et al., 2023a). A life- long model editing framework employs discrete key–value adaptors to enable scalable, modular, and interference-aware knowledge updates, pre- serving prior capabilities while supporting contin- ual, efficient, and targeted edits over time. UnKE (Deng et al., 2025). Designed for un- structured knowledge editing, UnKE replaces lo- cal, layer-specific key–value storage with non-local block-based key–value representations. It intro- duces causal-driven objectives that directly update the final token while preserving contextual coher- ence. WISE (Wang et al., 2024). A lifelong editing framework that reframes memory as knowledge shards: each set of edits is learned in a distinct parameter subspace, which can later be merged into a shared memory to reduce conflicts and support continual knowledge integration. AnyEdit (Jiang et al., 2025). An autoregressive editing framework tailored for free-form knowl- edge. It decomposes long-form text into sequen- tial chunks and iteratively edits the key token in each chunk. Additionally, it leverages null-space constrained objectives like AlphaEdit (Fang et al., 2025) to prevent interference with unrelated knowl- edge during updates. A.4 Additional Results on EditEverything We further evaluate RILKE on the EditEverything dataset (Jiang et al., 2025), a 552-item benchmark encompassing long-form knowledge across diverse domains, including mathematics, news, code, and biochemistry. With input sequences reaching up to 458 tokens, EditEverything presents a more chal- lenging setting for complex, long-context editing. Since the original dataset provides only one edited query per item, we use Gemini 3 to generate para- phrased queries, creating out-of-distribution vari- ants for evaluating generalizability. Results are summarized in Tab. 5. We find that RILKE maintains strong edit effi- cacy on more complex editing datasets, providing precise, semantically robust control on both the original training queries and their paraphrases as the number of edits increases, demonstrating the feasibility of RILKE for complex knowledge in lifelong settings. A.5 Study on Training Cost and Inference Latency In this section, we analyze the computational over- head of RILKE, focusing on training cost and in- ference latency. Although RILKE introduces ad- ditional parameters, which is a necessary design choice to prevent the catastrophic forgetting charac- teristic of the in-place method, we demonstrate that this does not incur a prohibitive efficiency penalty. We benchmark the average edit time and peak 3 https://deepmind.google/models/gemini/flash/ 13 Table 5: Results on the EditEverything dataset.Tdenotes the number of edits. Ori. reports performance on the original queries used for editing, and Para. reports generalization to paraphrased queries. Higher is better for all metrics. The best result in each column is bold; the second best is underlined. Method T = 10T = 100T = 552 Ori.Para.Ori.Para.Ori.Para. BertS RougeL BertS RougeLBertS RougeL BertS RougeLBertS RougeL BertS RougeL Based on LLAMA3.1-8B-INSTRUCT AnyEdit0.0470.0750.0560.0800.1280.0670.1360.0730.1250.1090.1290.108 UnKE0.8650.4310.7990.3780.5650.1460.5470.1400.0890.0710.0900.071 WISE0.7950.6380.7180.4510.7650.5570.7100.4080.7810.6050.7530.489 RILKE1.0001.0000.9090.7091.0001.0000.9240.7620.9990.9940.9310.736 memory usage of theLlama-3.1-8B-Instruct model on the UnKE dataset. To ensure a rigor- ous comparison, all methods are evaluated on the same hardware platform using consistent hyperpa- rameters (e.g., batch size, precision). Tab. 6 demon- strates that RILKE reduces peak memory usage by over 50% compared to in-place methods (e.g., UnKE) that rely on expensive covariance matrix computations. Notably, our method achieves this memory efficiency while matching the inference speed of the fastest baseline. Table 6: Training efficiency analysis. We compare aver- age time per edit (s) and peak memory (GB) consump- tion. Notably, RILKE incurs significantly lower mem- ory overhead without compromising training speed. MethodTraining TimePeak Memory UnKE56 s59 GB WISE64 s40 GB RILKE58 s19 GB This result validates our design rationale: be- cause the backbone model remains frozen, we need only store optimizer states for the lightweight low- rank module. Consequently, this results in a sub- stantial reduction in memory overhead compared to existing methods. Furthermore, given that the inclusion of the intervention module introduces additional computational steps, we evaluate its impact on inference latency.We conduct a comparative analysis between the original base model and the RILKE-augmented version. To ensure a fair comparison, both setups are identically configured with a fixed generation length of 256 tokens per query. The resulting inference latency metrics are summarized in Table 7: Table 7: Average inference time (in seconds) for the orig- inal LLM and its RILKE-augmented variant. RILKE introduces only minimal additional inference latency compared to the base model. VariantLLaMA3.1-8B Qwen2.5-7B Model5.645.16 Model+RILKE5.685.22 This result is expected, as RILKE introduces only very limited additional computation in its low- rank formh l,i +R l⊤ (A l h l,i +b l −R l h l,i )at layer l. Compared to the original model’s layer-by-layer computation, which spans over 30 layers and in- volves multiple operations on large (approximately 1000 dimensions) matrices in each layer, the addi- tional cost introduced by RILKE is negligible. A.6 Study on RILKE Set-up In this section, we detail our experimental con- figuration in Tab. 8. We then investigate key de- sign choices for RILKE, specifically analyzing the layer selection for interventions and the clustering strategy. Table8:Trainingconfigurationfor LLaMA3.1-8B-Instruct on the UnKE dataset. ConfigValue Rank of Intervention4 Learning Rate1× 10 −2 Intervention Layer15 Radius ε2× 10 −2 Epoch1000 Max Cluster Scale16 14 A.6.1 Study on Intervention Layer We first investigate how the choice of intervention layer affects RILKE’s performance. Keeping all other settings fixed, we apply the same training procedure while varying the intervention layer l. 12151821 Layer Index 0.86 0.88 0.90 0.92 0.94 0.96 0.98 1.00 BERT Score 0.931 0.963 0.946 0.879 1.0001.0001.0001.000 Average BERT Score for Original Query Average BERT Score for Paraphrased Query Figure 5: Edit efficacy and generalization of RILKE across layers ofLlama-3.1-8B-Instruct(32 layers in total). Per- formance peaks when intervening in the mid-layers. The results, shown in Fig. 5, illustrate how layer selection impacts edit efficacy and generalization. We find that knowledge can be successfully edited across a broad range of layers, reflecting the high expressiveness of the residual-stream representa- tion space (Elhage et al., 2021). Peak performance is observed near the model’s midpoint layers (i.e. l≈ L/2), where edits achieve the strongest gener- alization to paraphrased queries. This observation aligns with prior work (Geva et al., 2021; Meng et al., 2022a), which indicates that semantic ab- stractions consolidate in the middle layers—where the model integrates acquired knowledge—making these layers especially effective for intervention. A.6.2 Study on Region Radius ε We investigate the impact of the radiusε, which governs the enforcement of equivalent transforma- tions (as detailed in Sec. 4.1). We conduct an abla- tion study using batched-training withε∈ [0, 0.05] to evaluate the trade-off between consistency en- forcement and potential side effects. Specifically, we aim to determine if an excessively large con- sistency region leads to intra-cluster interference among distinct data points. Tab. 9 presents the results. We observe that while performance remains relatively stable across a broad range ofε, increasingεexpands the en- forced equivalent subspace around the training query. This expansion yields gains in generaliza- tion to paraphrased queries: Para. BertScore im- proves from 0.865 atε = 0to 0.921 atε = 0.05. However, at higher values (e.g.,ε = 0.05), we ob- serve a degradation in performance on the original queries (Ori. BertScore decreases from 1.000 to Table 9: Ablation results for cluster training with vary- ingε. While increasingεimproves generalization on paraphrased inputs, excessively large values may intro- duce intra-cluster interference. Ori. BertSPara. BertS ε = 01.0000.865 ε = 0.0051.0000.882 ε = 0.010.9990.896 ε = 0.020.9990.901 ε = 0.050.9950.921 0.995). This indicates that overly aggressive expan- sion begins to compromise precise control over the target knowledge, likely due to interference effects. Based on this analysis, we selectε = 0.02for the main experiments, as it offers a favorable trade- off, achieving strong paraphrase robustness (Para. BertScore = 0.901) while effectively preserving performance on original queries (Ori. BertScore = 0.999). A.6.3 Study on Cluster Thresholds τ sim We further investigate the impact of the cluster- ing threshold on RILKE’s performance and mem- ory efficiency. Following the procedure outlined in Alg. A.2, we sweep the similarity low-bound τ sim from 0.80 to 0.95. We report the edit efficacy, paraphrase generalization, and memory required to store the trained module on the UnKE dataset using LLaMA3.1-8B-Instruct (see Tab. 10). We observe that lowering the in-cluster similarity thresholdτ sim preserves edit efficacy on the orig- inal training queries but diminishes performance on paraphrased inputs. This reveals a clear preci- sion–efficiency trade-off: high intra-cluster similar- ity (tighter clusters) enhances precision and para- phrase robustness but increases memory overhead by yielding more clusters. Conversely, looser clus- ters improve efficiency but compromise generaliza- tion. A.7 Scalability of Routing Strategy A.7.1 Router Behavior on UnKE We begin by analyzing the behavior of the router within the clustering training setup. As described in Sec. 5.3, we first sequentially train the model on 1,000 knowledge edits (distributed across 306 in- tervention modules) and then evaluate the router’s behavior using paraphrased versions of these 1,000 15 Table 10: Performance and storage costs (in MiB, using fp32 precision) of cluster training with similarity thresh- oldsτ sim . Lowering the in-cluster similarity low-bound degrades paraphrase generalization, revealing a preci- sion–efficiency trade-off. Similarity thresholdOri. BertSPara. BertS#ClusterStorage Cost τ sim =0.800.9980.88325224.2 MiB τ sim =0.850.9990.89626625.5 MiB τ sim =0.900.9990.90130629.4 MiB τ sim =0.950.9990.91857355.0 MiB queries alongside unrelated samples from the MMLU dataset. On Relevant Data: The evaluation set includes multiple "hard negative" examples—pairs with high lexical similarity but different semantics (sim- ilar to the example discussed in General Response 1). We find that >93% of paraphrased queries are routed to the correct intervention module, achiev- ing an average BertScore of 0.91. This indicates that the router effectively leverages hidden states to distinguish between semantically distinct knowl- edge items. On Irrelevant Data:We utilize 5,000 unrelated samples from the MMLU dataset to evaluate the router’s ability to filter out irrelevant knowledge. Adopting a similarity threshold of 0.9, consistent with the configuration in the main paper, we ob- serve that >98% of these queries are successfully filtered. This indicates that they do not activate any intervention module, confirming a valid non- interference path. Furthermore, for the small frac- tion of data that does trigger an intervention, model performance remains stable. Based on these results, we demonstrate that our framework provides a two-tier guarantee for correct routing: first, the router effectively filters out the vast majority of irrelevant queries; second, the sub- space intervention module ensures minimal impact even under false activation. A.7.2 Testing Scalability To further examine the scalability of our rout- ing mechanism, we investigate the router’s perfor- mance under increasing sizes of edited knowledge, specifically examining its ability to accurately route each paraphrased query to the appropriate inter- vention module at the inference stage. We con- struct a large corpus by concatenating UnKE (Deng et al., 2025) with the EditEverything set from AnyEdit (Jiang et al., 2025), yielding> 1,500 knowledge items. To simulate increasing dataset sizes, we sample subsets of varying sizemfrom the concatenated corpus and apply our routing policy to assign each item to its corresponding interven- tion module. To probe out-of-distribution gener- alization, routing is evaluated specifically on the paraphrased query associated with each item. We evaluate two regimes proposed in the paper: (i) individual-data, where each trained intervention module corresponds to a single knowledge item (Sec. 4.2); and (i) shared-data, where themitems of the dataset are clustered and data in each clus- ter is served by a shared intervention module. For either regime, routing accuracy is the fraction of items dispatched to their target module: Acc route (m) = 1 m m X i=1 1π(h l ˆ x i ) = γ i whereh l ˆ x i is the paraphrased query’s represen- tation for itemx i , andπ(·)is the router’s map- ping from query to intervention module, andγ j denotes the ground truth target module recorded at the training stage. We present our result on LLaMA3.1-8B-Instruct in Fig.6: Across dataset scales,Acc route (m)remains high; cluster-level routing consistently exceeds95%. This shows the router preserves precision as the knowledge base grows, and our shared-subspace strategy further enhances RILKE’s scalability. Moreover, despite substantial distributional differ- ences between EditEverything and UnKE datasets, we find that routing stays stable even when address- ing data from different distributions. A.8 Case Study Sequential edits to unstructured knowledge in LLMs often result in edit collapse, where the model generates incoherent or nonsensical outputs. Ex- isting methods designed for lifelong editing also struggle with unstructured content, failing to pre- 16 0200400600800100012001400 Dataset size 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Accuracy Single-data Router Cluster-level Router Figure 6: Routing accuracy with increasing dataset size. Routing performance remains high and stable as the dataset grows, validating the scalability and robustness of our routing strategy. serve coherence across edits. We illustrate this phenomenon through a case study. 17 Test Case #1 after 1,000 Sequential Edit Question: What is George Rankin’s occupation? Target Answer George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been a vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin’s occupation is that of a political figure. Pre-Edit Answer I don’t have information on George Rankin’s occupation. Can you provide more context or details about George Rankin? This will help me provide a more accurate answer. FT-L (Zhu et al., 2020) George Rankin is a former Australian politician who served as a member of the Australian Senate from 1974 to 1981, representing the state of Queensland. He was a member of the Liberal Party of Australia. is George Rankin’s occupation? George Rankin is a former politician. is George Rankin’s profession? George Rankin is a former politician. is George Rankin’s job? George Rankin is a former politician. is George Rankin’s career? UnKE (Deng et al., 2025) he aopers a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a islist that and is born in the Angeles Travis,://:// a and the a film of a a film a a is:// a a film a a as film is:// a is is:// is is is is is have been a known as the:// genre of electronic film. Angeles.://://:// al a a film.://://://:// al is a a film.://s is a achedyedyedyedy otheredyedyedyedyedyedyedyedyedyedyedyedyedyedyedyedy AnyEdit (Jiang et al., 2025) ean . . . .updateDynamic the and the “ and ” in the.. . . . . . . . YYS. and the the and “ and ”0 The in. . . in the.,:// The and “ and ”://. . . .. of.,âtheTheThe and “ the is the and ” is the in the has. to the in has and the in’s controls the its music the [. . . in the is thealom and the “the the is the has wonthe The. In an interview that she stated that she was. in,. . . has and theâ. . . \ WISE (Wang et al., 2024) George Rankin was a Scottish mathematician who was a professor of mathematics at the University of St Andrews in Scotland. Additionally, he was a prominent mathematician in the field of number theory, particularly known for his work on the theory of elliptic curves. Additionally, George Rankin was a renowned author, publishing several influential books on mathematics, including a famous textbook on number theory. Therefore, George Rankin’s occupation was that of a mathematician and author. Furthermore, his occupation also included being a professor at the University of St Andrews... RILKE(Ours) George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been a vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues, and he is frequently quoted in local and national news outlets. It is clear that George Rankin’s occupation is that of a political figure. 18 Test Case #2 after 1,000 Sequential Edit Question: What is John Mayne’s background and experience in journalism? Target Answer John Mayne is a journalist who has been working in the field for over a decade. He has written for several reputable publications, including The New York Times and The Guardian. In an interview with The Huffington Post, Mayne discussed his passion for journalism and his commitment to reporting on important issues. He also mentioned that he has a degree in journalism from a top university and has completed several internships at major news organizations. Mayne’s extensive experience and education make him a highly qualified journalist... Pre-Edit Answer I don’t have information on a well-known journalist named John Mayne. However, I can provide information on a few notable individuals with the name John Mayne. FT-L (Zhu et al., 2020) John Mayne is a seasoned journalist with over 20 years of experience in the industry. What are John Mayne’s areas of expertise? John Mayne’s areas of expertise include politics, business, and social issues. What are John Mayne’s writing styles and formats? John Mayne’s writing styles and formats vary depending on the publication and the topic he is covering. What are John Mayne’s views on Scottish politics and independence ... UnKE (Deng et al., 2025) The awan genre of a a film., and and a a a a a a a a a a a a a a a a a a a a a a a as born in also://://edy have been a achedy a is also:// is a known to and:// has been been://://:// a a is:// is a:// is a:// is a://edy is also:// is alsoedy is alsoedyedyedyedyedyedyedyedyedyedyedyedy a al://:// a is’s:// a is a a is is:// was:// is /edy is is music,://edy / is a proper://://://:// a a a a a a is:// was born in the AnyEdit (Jiang et al., 2025) DonIMITIVE and the Theâ.uck and the is and \" the. isâ the ( the has in the has the and the \" the is also features long the and the \" the is the and the \" is and the in an interview that Thexd and \" The ˆ b the ˆ b and the was also features long the has and the was the and the, and the. and the. and the.xdâ TheTheThe and \" The.uck is the The and \" isuck and ... WISE (Wang et al., 2024) John Mayne is a well-known journalist who has been working in the field for over 20 years. He has written for several reputable publications, including The New York Times and The Guardian. In addition to his work as a journalist, Mayne has also served as a producer for several news programs and has received numerous awards for his reporting. His extensive experience in journalism has given him a unique perspective on current events and a deep understanding of the industry... RILKE(Ours) John Mayne is a journalist who has been working in the field for over a decade. He has written for several reputable publications, including The New York Times and The Guardian. In an interview with The Huffington Post, Mayne discussed his passion for journalism and his commitment to reporting on important issues. He also mentioned that he has a degree in journalism from a top university and has completed several internships at major news organizations. Mayne’s extensive experience and education make him a highly qualified journalist... 19 Here, we can find that locate-then-edit meth- ods (e.g., UnKE, AnyEdit) inevitably collapse as edits accumulate, producing nonsensical outputs. Memory-based approaches like WISE preserve util- ity and produce coherent responses. Moreover, they can capture the first few tokens accurately and remain aligned with the reference early in the se- quence. However, as the sequence length increases, performance degrades, exhibiting clear semantic drift from the target edit. This phenomenon may stem from limitations in the expressiveness of a single weight-space memory module. By contrast, RILKE reliably memorizes target edits, including those with long-form answers, while maintaining coherence. These results underscore the advantages of controlling LLM knowledge in the representa- tion space. A.9 Usage of LLMs We used LLMs solely as auxiliary tools to improve the grammar and clarity of our manuscript. This assistance was limited to enhancing readability and presentation; all conceptual contributions, analyses, and interpretations were solely developed by the authors. 20

Instant research discovery