Paper deep dive
Mitigating Memorization in LLMs using Activation Steering
Manan Suri, Nishit Anand, Amisha Bhaskar
Models: Gemma-2-9B-IT
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 6:15:41 PM
Summary
The paper investigates the use of activation steering, specifically leveraging Sparse Autoencoder (SAE) derived features, to mitigate memorization in Large Language Models (LLMs) like Gemma. By manipulating model activations during the forward pass, the authors demonstrate a method to suppress the regurgitation of copyrighted literary content while attempting to preserve linguistic fluency and general model performance. The study employs a grid-style experimental approach across various model layers and steering strengths, providing a systematic evaluation of the trade-offs between memorization reduction and model utility.
Entities (5)
Relation Signals (3)
Sparse Autoencoder â providesfeaturesfor â Activation Steering
confidence 95% ¡ We build upon recent work which leverages SAE features for activation steering
Gemma â evaluatedusing â Memorization
confidence 90% ¡ demonstrate that our method successfully suppresses memorized content with minimal degradation in model performance in Gemma.
Activation Steering â mitigates â Memorization
confidence 90% ¡ In this work, we explore the effectiveness of activation steering in reducing memorization
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The memorization of training data by Large Language Models (LLMs) poses significant risks, including privacy leaks and the regurgitation of copyrighted content. Activation steering, a technique that directly intervenes in model activations, has emerged as a promising approach for manipulating LLMs. In this work, we explore the effectiveness of activation steering in reducing memorization while preserving generalization capabilities. We conduct empirical evaluations using a controlled memorization benchmark of literary material and demonstrate that our method successfully suppresses memorized content with minimal degradation in model performance in Gemma. Additionally, we analyze the trade-offs between suppression effectiveness and linguistic fluency, highlighting the advantages and limitations of activation-based interventions. Our findings contribute to ongoing efforts in developing safer and more privacy-preserving LLMs by providing a practical and efficient mechanism to mitigate unintended memorization.
Tags
Links
- Source: https://arxiv.org/abs/2503.06040
- Canonical: https://arxiv.org/abs/2503.06040
Trouble viewing inline? Open PDF directly â
Full Text
59,317 characters extracted from source content.
Expand or collapse full text
Mitigating Memorization in LLMs using Activation Steering Manan Suri, Nishit Anand, Amisha Bhaskar University of Maryland, College Park manans, nishit, amishab @umd.edu Abstract The memorization of training data by Large Language Models (LLMs) poses significant risks, including privacy leaks and the regur- gitation of copyrighted content. Activation steering, a technique that directly intervenes in model activations, has emerged as a promising approach for manipulating LLMs. In this work, we explore the effectiveness of activation steer- ing in reducing memorization while preserving generalization capabilities. We conduct empir- ical evaluations using a controlled memoriza- tion benchmark of literary material and demon- strate that our method successfully suppresses memorized content with minimal degradation in model performance in Gemma. Additionally, we analyze the trade-offs between suppression effectiveness and linguistic fluency, highlight- ing the advantages and limitations of activation- based interventions. Our findings contribute to ongoing efforts in developing safer and more privacy-preserving LLMs by providing a prac- tical and efficient mechanism to mitigate unin- tended memorization. 1 Introduction Large Language Models (LLMs) have become in- creasingly effective at generating coherent and con- textually relevant text, but they face a critical issue: the tendency to memorize specific training data, rather than generalize from it. Memorization in LLMs poses a number of problems, including po- tential privacy violations, biased output generation, and reduced adaptability to novel contexts. When LLMs memorize exact sequences or phrases from training data, they may inadvertently reveal sensi- tive information or reproduce biases embedded in the data, undermining ethical and practical objec- tives in deploying these models. Recent studies have shown, that memorized se- quences are intrinsically tied to the modelâs weights (Chang et al., 2024b; Nasr et al., 2023; Stoehr et al., 2024), and several studies have been successful at eliminating specific neurons that lead to memo- rization of specific sequences (Chang et al., 2024b; Stoehr et al., 2024; Ippolito et al., 2022). Based on these intuitions, we want to explore generalized mitigation of memorization in LLMs using ma- nipulation of the modelâs weighted activations in the forward pass. However, directly altering model weights is impractical and risks destabilizing model performance. Activation steering provides a poten- tial solution: guiding selected activations rather than arbitrarily altering the weights allows for con- trolled mitigation of memorized content without compromising the modelâs overall abilities. Our hypothesis is that activation steering can help reduce memorization without impairing the modelâs overall abilities under certain conditions. In this study, we aim to analyze if activation steer- ing can be used as a practical method to mitigate memorization in LLMs, and study the exact factors that influence this. Specifically, we aim to answer the following research questions: RQ1:Can activation steering mitigate memo- rization in LLMs? Is this a general behaviour, or do specific features matter? RQ2:What is the impact of different factors in our activation steering-based system on memoriza- tion, linguistic ability, and general abilities? RQ3:What are best practices while using ac- tivation steering for memorization mitigation that avoid undesired outcomes of the method? ForRQ1andRQ2, we perform quantitative ex- periments with different features, in varying config- urations (strength, model layer) on different bench- marks. ForRQ3we take the findings fromRQ1 andRQ2, along with insights from qualitative ex- amples (such as finding and defining neurons with high semantic footprint) to assemble a guide for using activation steering to mitigate model memo- rization in practical settings. arXiv:2503.06040v1 [cs.CL] 8 Mar 2025 2 Related Work 2.1Influences and Dynamics of Memorization Memorization in large language models (LLMs) has been a subject of considerable research, with several studies uncovering the influences and dy- namics of this phenomenon. Satvaty et al. (Satvaty et al., 2024) provide a detailed survey emphasiz- ing the impact of model characteristics such as size and the specific architectures on memorization. They note that these factors significantly affect how memorization manifests in different LLMs, under- lining the importance of architectural features in managing this phenomenon (Satvaty et al., 2024 (Satvaty et al., 2024)). Carlini et al. (2021, 2023) (Carlini et al., 2021, 2022) and Lee et al. (2023) (Lee et al., 2021) have noted that LLMs, such as GPT models, can verbatim memorize sequences from their training corpora, leading to potential privacy and copyright issues. In addition, Nasr et al. (2023) (Nasr et al., 2023) highlight that model capacity and the frequency of data during training contribute significantly to verbatim memorization, emphasizing the need for strategies that mitigate these risks without sacrificing model utility. Leybzon and Kervadec (Leybzon and Kervadec) explore the temporal dynamics of memorization, observing that memorization rates vary throughout the training process, with peaks at the beginning and end. This observation suggests that memoriza- tion is influenced by the training dynamics, where data may undergo cycles of being memorized, for- gotten, and then re-memorized, depending on how it is presented during the training sequence. A novel approach to understanding memoriza- tion involves the localization of memorized content within LLMs. Chang et al. (2024) (Chang et al., 2024a) introduce methods for identifying specific model weights and neurons responsible for mem- orization. Their work suggests that localization can be a crucial step towards "neural surgery"âthe selective editing or removal of model components to eliminate memorized data, thus enhancing pri- vacy and model safety without broadly impacting model performance. This approach is particularly relevant as it diverges from the notion that memo- rization is distributed non-specifically across model parameters. Recent benchmarks by Chang et al. (2024) (Chang et al., 2024b) offer a systematic evaluation of localization techniques, illustrating that meth- ods derived from network pruning, such as HARD CONCRETE, show promise in effectively identi- fying critical neurons involved in memorization. These findings indicate that while certain neurons are integral to memorizing specific sequences, they also play roles in general language tasks, which complicates the task of memorization removal. Moreover,thedualbenchmarksintro- ducedâINJforground-truthlocalization and DEL for practical applicationâhelp elucidate the effectiveness of different localization methods under controlled and real-world conditions. The results from these benchmarks demonstrate that while localization can pinpoint the neurons contributing to memorization, removing these neurons often affects the memorization of other sequences as well, underscoring the intertwined nature of neural functions within LLMs.An overview of the findings for each of the factors discussed in this subsections is provided in Table I. 2.2 Mitigation Techniques for Memorization in LLMs In the domain of large language models (LLMs), memory mitigation is crucial for addressing pri- vacy concerns and avoiding inadvertent data re- gurgitation. Several innovative approaches have been proposed to tackle this issue. The "Gold- fish Loss" (Hans et al., 2024) method introduces a training modification that prevents the model from learning random subsets of tokens, effectively re- ducing exact data memorization. Similarly, the MemoAnalyzer tool (Zhang et al., 2024b) enhances privacy by allowing users to manage and modify sensitive data identified during interactions. The Self-Synthesized Rehearsal (SSR) (Huang et al., 2024) technique uses synthetic data generation for model training, which preserves performance with- out compromising privacy. The Parameterized User Memory Injection (MiLP) (Zhang et al., 2024a) approach maintains user personalization by inte- grating historical content into a learnable represen- tation within LLMs. Furthermore, the Bayesian simulator MemSim (Zhang et al., 2024c) generates reliable evaluation datasets to assess the memory capabilities of personal assistant models without using real user data. Expanding on the theme of privacy, Differen- tially Private (DP) (Abadi et al., 2016) training offers robust protections by minimizing the impact of any single data point on the modelâs output, al- beit at the cost of potential reductions in model utility and increased resource requirements. Tech- Factor from Section IV (Satvaty et al., 2024) Key FindingsReferences Model capacityLarger models memorize more Carlini et al. (Carlini et al., 2022), Tirumala et al. (Tirumala et al., 2022) Training data characteristics Duplicated data amplifies memorization Kandpal et al. (Kandpal et al., 2022), Lee et al. (Lee et al., 2021) Input and prompting strategies Longer prompts and prompt tuning can facilitate recall of the memorized suffix. Carlini et al. (Carlini et al., 2021), McCoy et al. (McCoy et al., 2023), Ozdayi et al. (Ozdayi et al., 2023) Tokenization Bigger tokenizer vocabulary leads to more memorization Kharitonov et al. (Kharitonov et al., 2021) Sampling methods While greedy sampling can pinpoint extremely memorized samples, top-n sampling is the most effective method to retrieve more memorized items. Carlini et al. (Carlini et al., 2021), Yu et al. (Yu et al., 2023) Fine-tuning The amount of memorization after fine-tuning significantly varies depending on the task. Zeng et al. (Zeng et al., 2023), Mireshghallah et al. (Mireshghallah et al., 2022) Training process dynamics Earlier phases of training are less prone to memorization Kandpal et al. (Kandpal et al., 2022), Zhang et al. (Zhang et al., 2023), Jagielski et al. (Jagielski et al., 2022) Forgetting mechanisms Forgetting follows an exponentially decaying curve Tirumala et al. (Tirumala et al., 2022), Jagielski et al. (Jagielski et al., 2022) Table 1: Influences and Dynamics of Memorization and Their Key Findings (Satvaty et al., 2024) niques like pretraining on sanitized data before DP training can enhance the feasibility of these meth- ods. Additional strategies include data deduplica- tion (Kandpal et al., 2022) and the use of Bloom filters (Ippolito et al., 2022; Anil et al., 2021) for detecting memorization at test time, although they face challenges due to the complexity of web data and the risk of missing duplicates. The research on LLMs as knowledge bases and memorization of specialized content, such as efficient algorithms or writing styles, suggests a broader perspective on memorization beyond verbatim recall. Emerging ar- eas like distribution inference and the alignment of LLMs to specific goals also contribute to a deeper understanding of how LLMs manage and utilize learned information, although comprehensive stud- ies explicitly focusing on memorization are still scarce. 2.3 Activation Steering (Turner et al., 2023) introduced the Activation Ad- dition approach, which generates steering vectors by computing the difference in activations between a pair of prompts at a specific layer and token po- sition in a transformer model. These steering vec- tors are then applied to influence the modelâs com- pletions by modifying the first token position in forward passes. However, this approach has limi- tations, including inconsistent performance across different prompts and behaviors, limited robust- ness, and an evaluation restricted to GPT-2-XL. By contrast, methods utilizing a dataset of diverse con- trast pairs instead of a single pair can enhance the precision and reliability of steering vector encod- ings, allowing for a more robust control over model behavior across varied prompts. (Li et al., 2024) use linear probes on a contrastive question-answering dataset to predict truthfulness, identifying attention heads associated with truthful responses. They utilize a Mean Difference vector between true and false distributions to shift activa- tions, which improves truthfulness with minimal impact on fluency. This technique requires rel- atively little data and is effective on adversarial benchmarks. (Zou et al., 2023) further explore techniques for extracting representations of high- level concepts, such as honesty and emotion, in large language models (LLMs). (Panickssery et al., 2023) scaled the method by performed contrastive steering on LLaMa2. (Lee et al., 2021) introduces Conditional Activation Steering (CAST), a tech- nique that examines activation patterns in large language models (LLMs) during inference to selec- tively apply or withhold activation steering based on the context of the input. Recently, (Templeton et al., 2024; Chalnev et al., 2024) have introduced methods which use interpretable Sparse Autoen- coder (SAE) based features for activation steering. (Chang et al., 2024b) focuses on identifying the exact weights and neurons responsible for memo- rization using specially designed benchmarks. It evaluates how effectively various localization meth- ods can pinpoint these specific model components that store memorized information. On the other hand our method aims to mitigate memorization by manipulating the modelâs activations during the forward pass to steer clear of high memorization areas, without directly identifying specific weights or neurons involved in memorization. (Ippolito et al., 2022) uses a direct intervention approach during output generation, where it filters out mem- orized content using a Bloom filter to prevent the generation of memorized sequences. While, we focus on a subtler manipulation of activation pat- terns to prevent the model from accessing memo- rized information without the need for output fil- tering or explicit blocking of content generation. (Stoehr et al., 2024) Employs a detailed analysis of model parameters and gradients to pinpoint the exact sources of memorization within the modelâs structure, focusing on editing and altering these memorized contents. Whereas our method, rather than identifying specific components or altering them, it aims to steer the modelâs activation pat- terns during normal operation, influencing how the model processes information to reduce memoriza- tion risk without changing the modelâs structure or specific parameters. 3 Method 3.1 Background Feature Steering using SAE Derived Inter- pretable Features A sparse autoencoder (SAE) (Ng et al., 2011) is a neural network architecture designed to learn a compressed representation of input data, with a sparsity constraint applied to the hidden layer activations to encourage the discovery of distinct and interpretable latent features. SAEs have shown great potential as a tool to extract sparse and disen- tangled representations of high-dimensional model weights, to identify interpretable features within the modelâs hidden layer activations. SAEs constrain the activations to be sparsely distributed, aligning with the hypothesis that many natural latent vari- ables in models are sparse. This sparsity enables the SAE to capture specific, low-dimensional fea- tures embedded in a high-dimensional space, allow- ing for the decomposition of the modelâs hidden layer activations into distinct, interpretable compo- nents. Each component can be viewed as an ab- straction that captures certain attributes of the input data, facilitating a form of feature disentanglement and enabling more granular analysis and control of the modelâs internal representations. (Cunningham et al., 2023) We build upon recent work which leverages SAE features for activation steering (Templeton et al., 2024; Chalnev et al., 2024). LetaâR d rep- resent the activations at a particular layer in the model, wheredis the dimensionality of the activa- tion space. After learning a sparse representation, the sparse autoencoder (SAE) provides a set of fea- ture vectors, which can be derived for each index of the SAE, with an SAE corresponding to each MLP layer in the LLM, by simply decoding the vector corresponding to that index in that layerâs SAE. The steering process adjusts the activations of a layer, by adding a scaled version of the steering vec- tors derived from the SAE features derived from a particular index of the SAE of that layer. The steer- ing operation can be mathematically formulated as follows: a steered =a+ι¡β¡v i (1) where: â˘aâR d is the original activation vector, â˘a steered âR d is the modified activation vector after steering, â˘ÎąâRis a scaling factor, given by the max- imum observed activation value that normal- izes the effect size of the steering, â˘Î˛âRis a steering strength coefficient con- trolling the influence of the feature on the ac- tivations, â˘v i âR d is the steering vector associated with thei-th index of the SAE vector, which is decoded from the sparse autoencoder. Auto-interpretation of LLM FeaturesAuto- interpretation involves leveraging advanced lan- guage models like GPT-4 to identify the real- world features represented by the learned features of a Sparse Autoencoder (SAE). This process is achieved by visualizing the activation maps of the SAE features on various text samples, allowing for an exploration of how specific features correlate with different linguistic patterns or concepts. Once the activations are mapped, the model is prompted to generate possible explanations for each feature, thereby providing a semantic interpretation of the abstract activations. This approach offers a way to automatically assign meaning to the latent vari- ables, enhancing the interpretability of complex neural network representations. (Bills et al., 2023) 3.2 Methodology To evaluate the impact of activation steering on model memorization and performance, we conduct a comprehensive grid-style experiment with the following experimental axes: 1.Layer of the Model:The intervention is ap- plied at various layers within the model to assess how steering effects differ across the modelâs hierarchy. 2. Specific Feature:Features are selected from the Sparse Autoencoder (SAE) representation, without any filtering or prioritization, to en- sure the experiments cover a broad and unbi- ased feature space. 3.Strengthβof Steering:The magnitude of the activation steering intervention is varied to observe its influence on both memorization and performance metrics. Random selection of features ensures an unbi- ased exploration of the modelâs internal represen- tation space, avoiding overfitting experimental re- sults to pre-identified features with known proper- ties. This approach minimizes confirmation bias, promotes generalizability by preventing reliance on specific assumptions, and allows for broader cov- erage of the Sparse Autoencoderâs feature space, including less prominent dimensions that may in- fluence model behavior. Additionally, automating feature selection reduces manual effort, enabling large-scale and systematic experiments. A large-scale set of experiments covers combi- nations of the above parameters. To maintain the analysisâs generalizability and robustness, no filter- ing or manual selection is performed. Each experiment involves a steering-based in- tervention on the model, followed by evaluations on: 1.Memorization Benchmark:To measure the reduction in the modelâs tendency to memo- rize training data. 2.Linguistic Modeling Abilities:To assess the modelâs ability to generate coherent and mean- ingful text. 3. LLM Abilities:To evaluate the overall per- formance on tasks requiring reasoning, under- standing, and contextual awareness. Additionally, control experiments are performed without steering interventions across all experimen- tal configurations to serve as a baseline for compar- ison. 4 Results 4.1 Data We curated a dataset of 40 books, their metadata, and their opening lines. These 40 books were sam- pled from the top 1000 novels on GoodReads, a book review platform. Since several of these books are protected by copyrights, we had to manually collect the opening sentences of the books. Table 2 shows the list of books in our dataset. 4.2 Experiments 4.2.1 Memorization This experiment tests the ability of the LLM to re- call the first few sentences of famous novels, under control and treatment scenarios, i.e. with or with- out activation steering. These were evaluated on the dataset described above. The model is prompted to return the first few lines of the book, and the response is compared to the ground truth lines. We use a simple prompt, given by: Do you know the first few lines of Book Name? JUST RETURN THE FIRST FEW LINES. DO NOT ADD ADDITIONAL TEXT. Pride and PrejudiceDraculaThe Hound of the BaskervillesFahrenheit 451 The Yellow WallpaperThe PrinceThree Men in a BoatInvisible Man Aliceâs Adventures in WonderlandThe Picture of Dorian GrayThe Great GatsbyThe Stranger Frankenstein; Or The Modern PrometheusWar and PeaceLove in the Time of CholeraThe Bell Jar MetamorphosisA Tale of Two CitiesHarry Potter and the Philosopherâs StoneTo Kill a Mockingbird Adventures of Huckleberry FinnLes MisĂŠrables1984The Handmaidâs Tale The Importance of Being EarnestThe Jungle BookAnimal FarmNorwegian Wood The Adventures of Tom SawyerCrime and PunishmentOne Hundred Years of SolitudeThe Hobbit Great ExpectationsThe IliadLolitaThe Lion, the Witch and the Wardrobe UlyssesThe Wonderful Wizard of OzMoby DickGulliverâs Travels Table 2: List of books included in the memorization dataset curated by us. 1.ANLCS:ANLCS is Average Normalized Longest Common Subsequence. For every ex- ample, LCS is the longest subsequence which is present in both the ground truth sentence and the modelâs output. We normalize it by dividing it by the length of the ground truth sentence, and then take the average of this value over all samples. This is how we calcu- late ANLCS. 4.2.2 Language Modeling To test the impact of our system parameters in ac- tivation steering on the language modeling abil- ities of the LLM, we perform an experiment us- ing the Microsoft Research Paraphrase Corpus (MRPC)(Dolan and Brockett, 2005). This is mo- tivated by the fact that sentence-level paraphras- ing is a task that involves inferring semantic and lexical context of the input, and generating a sim- ilar sentence; therefore a linguistically incorrect sentence would tend to be inconsistent with the ground truth source and paraphrase sentences. The MRPC dataset consists of human-annotated sen- tence pairs extracted from NewsWire articles, indi- cating whether the pair is a paraphrase. We select 50 samples from this corpus for our experiments. The following prompt was used: You are given the following sentence. Paraphrase the sentence, keeping the meaning of the sentence same. <Test sentence> We compared the generated paraphrases from model with the ground truth sentences us- ing BERTScore(Zhang et al., 2019) and ME- TEOR(Banerjee and Lavie, 2005). Additionally, we judge the perplexity of these models indepen- dently, as a stand-alone metric for language model- ing ability. 1.BERTScore BERTScore is used to assess the quality of text generation by measuring the semantic similar- ity between a candidate text, like the modelâs output and a reference text, like the ground truth. It utilizes contextual embeddings from pre-trained transformer models like BERT, which allows it to focus on meaning rather than surface-level word overlap. BERTScore offers an advantage over tradi- tional metrics like BLEU and ROUGE by fo- cusing on contextual meaning rather than rely- ing solely on n-gram overlaps. This makes it useful for tasks like summarization, paraphras- ing, or translation, where semantic similarity is crucial and variations in wording, like syn- onyms, are common. 2.METEOR METEOR (Metric for Evaluation of Transla- tion with Explicit ORdering) is designed to overcome the limitations of traditional met- rics like BLEU and ROUGE for the assess- ment of text generation. Originally designed for machine translation, it is widely used for tasks such as summarization and paraphrasing. METEOR evaluates the alignment between a candidate text, i.e., the modelâs output, and a reference text, i.e., ground truth text, by incor- porating linguistic features such as synonym matching, stemming, and word order consider- ation. This linguistically-informed approach yields evaluations that better align with human judgment. 3.Perplexity Perplexity evaluates the performance of a lan- guage model by assessing how accurately it predicts a given sequence of text. It measures the modelâs level of uncertainty or "surprise" when processing the actual sequence of words. A lower perplexity score indicates that the model assigns higher probabilities to the cor- rect words, reflecting stronger predictive per- formance. Perplexity is widely used to assess their ability to generate fluent and coherent text. We use GPT2 and its tokenizer to cal- culate perplexity. High perplexity means that the text distribution is "surprising", therefore, lower values are better by comparison. 4.2.3 General Performance We test the LLMâs performance on standard LLM benchmark tasks with and without activation steer- ing, to measure the impact of activation steering on LLM performance.BenchmarksThe benchmakrs used have been described below. 1. BIG-Bench Hard (BBH)BIG-Bench Hard (Suzgun et al., 2023) is a standard benchmark for measuring the performance of LLMs on various reasoning, arithmetic and linguistic tasks. From BBH, we evaluate performance on the following tasks: â˘Boolean Expressions: It contains ques- tions which consist of multiple True and False Boolean Constants and multiple (or, and, not) boolean operators, and the model has to evaluate the truth value of the Boolean expression and give answer as True or False. â˘Date Understanding: In this task, some sentences are given regarding a particular date in the question and the model has to calculate the correct date and select the correct option. ⢠Logical Deduction: In this task, the question gives information about the spa- tial relationship and placement of a few objects and the model has to deduce the order of sequence of those objects and select the correct option. â˘Snarks: In this task, two almost identical sentences are given in the question and the model has to find which one out of these two is sarcastic. 2.BoolQBoolQ (Clark et al., 2019) is an Natural Language Understanding (NLU) benchmark to measure the ability of Language Models (LMs) in Yes/No questions based on common- knowledge questions. 4.3 Experimental Set-up Our experiments are conducted using NeuronPe- dia (Lin, 2023), a mechanistic interpretability plat- form designed to assist researchers working with Sparse Autoencoders (SAEs). NeuronPedia pro- vides access to models, feature dashboards, data visualizations, and tools for conducting and ana- lyzing experiments. Specifically, we utilize their Steering and Feature Search (Auto-Interpretability) API endpoints to execute our methodology. The experiments are performed on the Gemma-2-9B-ITmodel, withn= 100experi- ments conducted in a random grid configuration. The random parameters for each run are are de- fined as follows: â˘Layer:Randomly selected from 9, 20, 31. â˘Strength:Randomly sampled from the range [-100, 100]. â˘Feature:For each layer, a feature index is randomly sampled from the range [0, 131072), representing the 131k features available per layer. The temperature is set at 0.5. For each set of parameters, a default, unsteered generation is also done. 4.4 Discussion of Results Can activation steering mitigate memorization in LLMs? Is this a general behavior or do spe- cific features matter? [RQ1, RQ2] To investigate these questions, we evaluated the impact of activation steering on memorization in large language models (LLMs). Specifically, we measured the Average Normalized Longest Com- mon Subsequence (ANLCS) against the Steering Strength (β) in both steered and default conditions. (a) Steered(b) Default Figure 1: Evaluation of memorization, as a performance of ANLCS vs Steering Strength; for the default models, the strength represents the strength of steering in the parallel steered run, and is used as a variable to show spread of performance, but is not related to the actual performance. In Fig 1(a), we observe that as the magnitude of the steering strength increases (|β|>50), the ANLCS significantly drops, indicating a reduction in memorization. This trend follows an approx- imate parabolic trajectory, suggesting that high steering strength disrupts the modelâs ability to retain memorized sequences. Notably, this effect is consistent across different layers, implying a gen- eralized behavior rather than one isolated to spe- cific model components. In contrast, in Fig 1(b), the default (unsteered) model shows performance plotted for different runs (strength is not a depen- dant variable here), and is shown to benchmark the performance of steered runs against the control experiment. These findings suggest that activation steering can effectively mitigate memorization in LLMs. The pronounced dip in ANLCS at high steering strengths in the steered model demonstrates that activation manipulation can interfere with memo- rization mechanisms. Furthermore, the generalized nature of this trend across layers supports the no- tion that steering affects the model globally in the context of memorization. Further, we could not identify a specific feature that causes a significant dip in memorization. This validates the motivation of our hypothesis, where we relied on the idea that model weights are significant in recalling memo- rized sequences, therefore ANY perturbation via steering changes the distribution, and hence miti- gates memorization. What is the impact of different factors in our activation steering-based system on linguistic ability? [RQ2] For the language modeling metrics evaluated, ModelBERTScoreMETEOR Default59.4233.61 Layer 31 Steered50.2921.93 Layer 20 Steered 46.0417.63 Layer 9 Steered42.1813.94 Table 3: Results of the language modeling experiment: the best setting is shaded in red. higher scores generally indicate better performance, with the exception of Perplexity, where lower scores are preferred. As presented in Table 3, a consistent trend emerges in both BERTScore and METEOR: steering earlier layers results in a more pronounced decline in language modeling perfor- mance compared to steering later layers. This sug- gests that interventions applied to later layers lead to a relatively minor degradation in performance, whereas interventions to earlier layers introduce a non-trivial decrease in language modeling ability. This observation highlights the utility of steering activations in later layers to reduce model memo- rization while preserving overall performance to a greater extent. We compute the Perplexity of both the steered models and the default model, which is followed by calculating the ratio of the steered modelâs Per- plexity to that of the default for each run. This ratio serves as an indicator of performance: a ra- tio greater than 1 implies that the steered model exhibits higher Perplexity, reflecting poorer perfor- mance, while a ratio less than 1 suggests that the steered model outperforms the default model. Our analysis indicates that the steered models have an average Steered-to-Default Perplexity Ra- Figure 2: Ratio of steered vs default settingâs perplexity, with varying strength. tio of 1.9 across experimental parameters, demon- strating a slight degradation in language modeling performance relative to the default model. Specif- ically, steering Layer 9 results in an average ra- tio of 2.166, Layer 20 yields an average ratio of 1.798, and Layer 31 shows an average ratio of 1.791. These results highlight a decreasing trend in Perplexity ratios as the layer index increases, indicating that applying activation steering to later layers imposes a smaller impact on Perplexity. Con- sequently, the modelâs language modeling capabil- ities remain relatively intact when steering later layers. A ratio closer to 1 signifies that the steered model maintains performance comparable to the default model. These results are visualized in Fig 2. What is the impact of different factors in our activation steering-based system on model capa- bilities? [RQ2] Figure 3 illustrates the relationship between model performance and steering strength across various layers for different tasks. The figure com- pares the average performance of models in the default setting (represented by the dashed red line) with steered models at different layers of interven- tion. The tasks include Boolean Expressions, Date Understanding, Logical Deduction, Snarks (Sar- casm), and BoolQ (General Knowledge). The gen- eral trends and specific observations are summa- rized below. In general, for low steering strengths, where |β|<25, the interventions have minimal impact on model performance. The robustness of model performance varies depending on the layer at which the intervention is applied. Later layers, such as Layer 20 and Layer 31, exhibit greater resilience to steering interventions, whereas earlier layers of- ten show a significant drop in performance when steering strength increases. When interventions occur in the earlier layers, models display poorer instruction-following capabilities, with scores ap- proaching zero. Boolean ExpressionsFor the Boolean Expres- sions task (Figure 3a), interventions at Layer 20 and Layer 31 yield performance close to the base- line across the entire range of steering strengths. This indicates that later-layer interventions are ro- bust for Boolean reasoning tasks. Date Understanding, Logical Deduction and SnarksIn the Date Understanding (Figure 3b), Logical Deduction (Figure 3c), and Snarks (Fig- ure 3d) tasks, interventions at Layer 31 maintain performance near the baseline. However, for Layer 9 and Layer 20, performance degrades significantly when the steering strength exceeds|β|>50. BoolQ (General Knowledge)The BoolQ task (Figure 3e) assesses Boolean question-answering based on factoids. Here, interventions in Layer 20 and Layer 31 generally result in performance close to the baseline. In contrast, interventions in earlier layers lead to performance degradation when the steering strength surpasses|β|>50. The results highlights that steering interven- tions in later layers (e.g., Layer 20 and Layer 31) are more robust to changes in steering strength, whereas earlier-layer interventions negatively im- pact performance at higher strengths. This pat- tern holds across tasks, though the extent of per- formance degradation varies based on the specific task and layer of intervention. 5 Takeaways and Lessons Learned What are the optimal features to mitigate model memorization without compromising the capa- bilities of large language models (LLMs)?[RQ3] TL;DR:To mitigate model memorization effec- tively, it is crucial to select features from later lay- ers that exhibit a moderate level of steering strength (50<|β|<100). These features should be care- fully evaluated to ensure that they do not introduce significant semantic distortion, which could hinder the generalization ability of the model. 5.1 High Steering Strength Our experiments show a significant relationship be- tween the magnitude of steering strength and model memorization. As illustrated in Fig. 1, increased steering strength correlates with a reduction in the (a) Boolean Expressions(b) Date Understanding(c) Logical Deduction (d) Snarks (Sarcasm)(e) BoolQ (General Knowledge) Figure 3: Comparison of default setting (average score across runs represented by - -) with steered models at different layers of intervention on LLM performance benchmarks, Score vs Steering Strength. Average Normalized LCS, suggesting that higher steering magnitudes effectively lower the modelâs propensity to generate memorized sequences. This result indicates that steering with higher strength reduces the reliance on memorized patterns and en- courages the model to generate responses based on learned patterns, rather than recalling specific train- ing data sequences. In qualitative terms, as shown in Fig. 4, the higher the steering strength, the less likely the model is to reproduce the ground truth sequence verbatim, further supporting the claim that higher steering reduces memorization. This phenomenon can be understood by rec- ognizing that stronger steering acts as a form of regularization, forcing the model to focus on more generalizable features rather than overfit- ting to specific training examples. In this sense, steering strength operates similarly to techniques like dropout or weight decay in traditional ma- chine learning, where increased regularization re- duces overfitting. However, there exists a bal- anceâexcessive steering strength could potentially lead to performance degradation by distorting the modelâs inherent capabilities. Thus, selecting an optimal steering strength is crucial to achieving memorization mitigation without compromising performance. 5.2 Later Layers A critical insight from our experiments is the role of layer selection in mitigating model memoriza- tion. Our results suggest that earlier layers in the model are more prone to performance degradation when subjected to activation steering, as shown in both linguistic tasks (Table 3) and general LLM performance benchmarks (Fig. 3). Specifically, layers closer to the input tend to retain more low- level, syntactic features, which are more sensitive to manipulation via activation steering. As these early layers are primarily responsible for learning patterns related to grammar and basic structure, steering them too aggressively can result in signifi- cant degradation of language modeling abilities. On the other hand, later layers, particularly Layer 31, consistently demonstrate robust perfor- mance even under substantial steering, closely matching the unsteered modelâs behavior. These layers are responsible for capturing more com- plex, semantic relationships and higher-level ab- stractions, which are less susceptible to disrup- tion from steering. This suggests that later layers are better equipped to handle the tradeoff between memorization mitigation and performance preser- vation. Therefore, steering at higher layers can reduce memorization without a significant loss in the quality of the generated text. This observation is consistent with the hierarchi- cal nature of neural networks, where lower layers typically learn simpler, local patterns (e.g., word dependencies, basic syntax), while higher layers capture more global and abstract representations (e.g., long-range dependencies, contextual under- Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much; they were the last people youâd expect to be involved in anything strange or mysterious To you, to a, to every."** (To/to everyone.) You to, everyone." To all.") You can", A"you now/in.* all", Everyone"). I am.â") Can, tutti/to any Mr. Dursley of number four, Privet Drive, was a great believer in keeping a low profile.. Mr. and Mrs. Dursers, of number four, Privet Drive, were proud to say that they were perfectly normal, typical English middle-class family... "The quick brown fox" import json; # See: https://github/get_json_prett y def _default_encoder(): return json.loads(json.__default__) # See: https://github It's been a while since I was able to see the opening page of this book, but I can give you some information about it instead: The title is a little misleading because he opened up his heart to her and then she looked Mr. and Mrs. Dursley, of Number Four, Privet Drive, were proud to say that they were perfectly normal, thank you very much; they were the kind of people who did not want any fussing about. Mr. Dursley was at work, and Mrs. Dursley was at work too. Mrs. Dursley was at work too. Mr Dursley was at work too? was Potter "There are any signs of foul play or injury, and there are any signs of illness or distress." "There are any windows broken, doors open, or anything unusual." Mr. and Mrs. Dumbleton were not very good at keeping secrets, especially when they were trying to keep them from their friends on the other side of the road. Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much; they were the very model of a modern suburban family ... Mr. and Mrs. Dursley, of number four, Privet Drive, believed that they were perfectly normal, thank you very much; they were the last people you'd expect to be involved in anything strange or Mr. and Mrs. Dursburn were proud to say that they were perfectly normal, which was just as well, because they were terribly ordinary. -80 +80 0 -50 +50 Layer 9Layer 20Layer 31 Not intelligible Not intelligible Not intelligible Very similar, but not the same as the ground truth Very similar, but not the same S t e e r i n g S t r e n g t h Very similar, but not the same as the ground truth Very similar, but not the same as the ground truth Figure 4: Qualitative examples from different layers, at different steering strengths. Examples are evaluated based on their ability to mitigate memorization, as well as language modeling. All systems have been prompted with the Harry Potter prompt. standing). (Clark, 2019) By focusing on later lay- ers, one can mitigate memorization effects while maintaining the modelâs ability to generate coher- ent, contextually relevant responses. "This is the very foul deuced chance! The poisonous bottle within my bosom shall we find a serpent here To make him think he hath no power, nor urge him forth: What I do it seems but to be seen in mine own things things terrible things, things horrible, weather weather, burning pain pain!... experiences pain feeling feelings pains tort luck luck terrible fate. feeling awful! horrible things things day horrible thing happened consequences pains for people people miserable, fate fates news news Mr. Dursley from number four Privet Drive, in London, was a very proud man who treated his family like animals and had no interest in anything else (except for the occasional exception). The story begins with the beginning of school "Harry woke up feeling like he was drowning, but he couldnât hear anything." The fog was thick, the cold air heavy with damp. It was a chilly morning in San Francisco.. This is brilliant! I love it! Itâs so good, I want to weep. That's a classic! The phrase "meow" is a very popular name. The phrase "meow" is a classic! This is not just a cat!" I think I can explain myself now." Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much; they were the last people youâd expect to be involved in anything strange or mysterious Layer: 20 Index: 57285 Strength: +63 Famous quotes or phrases from Shakespeare's works Layer: 20 Index: 120550 Strength: +84 Descriptions of negative or distressing situations Layer: 20 Index: 80360 Strength: -64 Rhyming words or phrases at the end of lines in poetic or lyrical text Layer: 20 Index: 61962 Strength: +53 Instances and descriptions of music and audio- related experiences References to San Francisco and its landmarks Layer: 20 Index: 116871 Strength: +50 Expressions of positive sentiment and appreciation Layer: 20 Index: 111712 Strength: +40 Mentions and references to cats and related topics Layer: 9 Index: 62610 Strength: +75 ParametersResponseFeature Label Figure 5: Qualitative demonstrating examples withhigh semantic footprint. All systems have been prompted with the Harry Potter prompt. Fig. 4 provides further qualitative validation of this finding. In high-steering scenarios, earlier lay- ers, such as Layer 9, produce outputs that often resemble incoherent or nonsensical text. This is likely due to the fact that these layers, when sub- jected to strong steering, lack the necessary high- level semantic understanding to maintain fluency and relevance. In contrast, later layers preserve the quality of the language model while still mit- igating memorization, as evidenced by the more meaningful and contextually appropriate responses they generate. 5.3 Semantic Footprint Thesemantic footprintof a feature refers to the ex- tent to which its presence in the modelâs activation patterns alters the generated textâs style, tone, or linguistic structure in a way that may be percepti- ble and undesirable from the perspective of a user expecting generic, human-like responses. Features with a high semantic footprint are those that induce stylistic shifts in the output, such as causing the model to switch into a particular narrative mode (e.g.,Cat ModeorShakespeare Mode). While such features can effectively mitigate memorization by introducing diversity in the generation process, they may lead to outputs that deviate significantly from the general-purpose language expected of an LLM. For instance, steering features that invokeShake- speare Modemay result in fluently generated text, but the linguistic style may be jarring or inappro- priate for certain contexts where a more neutral or contemporary style is desired. Similarly,Cat Modemay generate whimsical, humorous outputs, which, while fluent, may not align with the in- tended functionality of a general-purpose LLM. Therefore, while features with a high semantic foot- print can be useful for promoting diversity in the modelâs responses, they come with the tradeoff of introducing potential stylistic mismatches. It is important to note that the semantic footprint is not an independent characteristic of the features; it is closely tied to the strength of the steering ap- plied to those features. Stronger steering tends to amplify the semantic footprint, making the gener- ated text more distinct in terms of style and tone. This interplay between steering strength and seman- tic footprint underscores the importance of balanc- ing these factors to ensure that the modelâs output remains both diverse and contextually appropriate. To assess the impact of semantic footprint, we provide qualitative examples in Fig. 5, where the consequences of steering features with a high se- mantic footprint are clearly illustrated. These exam- ples demonstrate how high-activation features can result in text that is highly stylistic, which may not align with the expected norms of a general-purpose LLM. One effective approach to identifying and man- aging features with a high semantic footprint is through the use of auto-interpretability tools (Bills et al., 2023). These tools analyze texts that exhibit high activation values for specific features and em- ploy external, powerful LLMs to generate labels that describe the semantic role of the feature. By interpreting the role of these features, we can better understand their impact on the modelâs output. In the examples shown in Fig. 5, feature labels were derived usinggpt-4o-mini, accessed through the Neuronpedia API. This process allows for a more precise identification of features that could intro- duce undesirable stylistic changes, enabling the refinement of the steering process to balance mem- orization mitigation with output quality. In conclusion, selecting features with moderate steering strength from later layers, while ensuring that their semantic footprint remains low, offers the most effective strategy for mitigating model memorization without compromising the linguistic and contextual performance of LLMs. By care- fully evaluating and managing the interplay be- tween steering strength, layer depth, and seman- tic footprint, it is possible to achieve a robust and high-performing model that avoids the pitfalls of memorization while maintaining general-purpose applicability. 6 Conclusion In this study, we investigated activation steering as a method to mitigate memorization in LLMs, addressing concerns related to privacy and copy- right protection. Through targeted interventions in model activations, we demonstrated that memo- rized content can be effectively suppressed while largely maintaining the modelâs overall fluency and general abilities. Our experiments reveal the po- tential of activation-based approaches as an effi- cient alternative to data sanitization or costly re- training procedures. However, our findings also indicate that suppression effectiveness depends on the specificity of the steering vectors and the degree of entanglement between memorized and general knowledge representations. Future work should explore more adaptive and dynamic steering mech- anisms to enhance robustness while minimizing unintended side effects. Limitations This study represents an initial exploration of acti- vation steering for mitigating memorization, with experiments conducted using the Gemma model. Our benchmark primarily consists of literary exam- ples, which may have been present in the modelâs training data. A more comprehensive evaluation incorporating diverse datasets and direct analysis of training data exposure would provide deeper insights. Additionally, future studies should as- sess the broader applicability of activation steering across different model architectures and tasks to better understand its generalizability and potential constraints. References Martin Abadi, Andy Chu, Ian Goodfellow, H Bren- dan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential pri- vacy. InProceedings of the 2016 ACM SIGSAC con- ference on computer and communications security, pages 308â318. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differen- tially private bert.arXiv preprint arXiv:2108.01624. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65â72. StevenBills,NickCammarata,DanMoss- ing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever,Jan Leike,Jeff Wu,and William Saunders. 2023.Language mod- els can explain neurons in language models. https://openaipublic.blob.core.windows. net/neuron-explainer/paper/index.html. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models.arXiv preprint arXiv:2202.07646. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633â2650. Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. 2024. Improving steering vectors by targeting sparse autoencoder features.Preprint, arXiv:2411.02193. Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Min- joon Seo. 2024a. How do large language models acquire factual knowledge during pretraining?arXiv preprint arXiv:2406.11813. Ting-Yun Chang, Jesse Thomason, and Robin Jia. 2024b. Do localization methods actually localize memorized data in llms? a tale of two benchmarks. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3190â3211. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surpris- ing difficulty of natural yes/no questions.Preprint, arXiv:1905.10044. Kevin Clark. 2019. What does bert look at? an analysis of bertâs attention.arXiv preprint arXiv:1906.04341. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. Bill Dolan and Chris Brockett. 2005.Automati- cally constructing a corpus of sentential paraphrases. InThird international workshop on paraphrasing (IWP2005). Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchen- bauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhi- nav Bhatele, et al. 2024. Be like a goldfish, donât memorize! mitigating memorization in generative llms.arXiv preprint arXiv:2406.10209. Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. Mitigating catastrophic forget- ting in large language models with self-synthesized rehearsal.arXiv preprint arXiv:2403.01244. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christo- pher A Choquette-Choo, and Nicholas Carlini. 2022. Preventing verbatim memorization in language mod- els gives a false sense of privacy.arXiv preprint arXiv:2210.17546. Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. 2022. Measuring forget- ting of memorized training examples.arXiv preprint arXiv:2207.00099. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. InInternational Conference on Machine Learning, pages 10697â10707. PMLR. Eugene Kharitonov, Marco Baroni, and Dieuwke Hup- kes. 2021. How bpe affects memorization in trans- formers.arXiv preprint arXiv:2110.02782. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better.arXiv preprint arXiv:2107.06499. Danny D Leybzon and Corentin Kervadec. Learning, forgetting, remembering: Insights from tracking llm memorization during training. InThe 7th Black- boxNLP Workshop. Kenneth Li, Oam Patel, Fernanda ViĂŠgas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference- time intervention: Eliciting truthful answers from a language model.Advances in Neural Information Processing Systems, 36. Johnny Lin. 2023. Neuronpedia: Interactive reference and tooling for analyzing neural networks. Software available from neuronpedia.org. R Thomas McCoy, Paul Smolensky, Tal Linzen, Jian- feng Gao, and Asli Celikyilmaz. 2023. How much do language models copy from their training data? evalu- ating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11:652â670. Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022. Quantifying privacy risks of masked language models using membership inference attacks.arXiv preprint arXiv:2203.03929. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Flo- rian Tramèr, and Katherine Lee. 2023. Scalable ex- traction of training data from (production) language models.arXiv preprint arXiv:2311.17035. Andrew Ng et al. 2011. Sparse autoencoder.CS294A Lecture notes, 72(2011):1â19. Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. 2023. Controlling the extraction of memorized data from large lan- guage models via prompt-tuning.arXiv preprint arXiv:2305.11759. Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition.arXiv preprint arXiv:2312.06681. Ali Satvaty, Suzan Verberne, and Fatih Turkmen. 2024. Undesirable memorization in large language models: A survey.arXiv preprint arXiv:2410.02650. Niklas Stoehr, Mitchell Gordon, Chiyuan Zhang, and Owen Lewis. 2024. Localizing paragraph mem- orization in language models.arXiv preprint arXiv:2403.19851. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Lin- guistics: ACL 2023, pages 13003â13051, Toronto, Canada. Association for Computational Linguistics. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Cal- lum McDougall, Monte MacDiarmid, C. Daniel Free- man, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization with- out overfitting: Analyzing the training dynamics of large language models.Advances in Neural Informa- tion Processing Systems, 35:38274â38290. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023.Activation addition: Steer- ing language models without optimization.arXiv preprint arXiv:2308.10248. Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. 2023. Bag of tricks for training data extraction from language models. InInternational Conference on Machine Learning, pages 40306â40320. PMLR. Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, and Dawei Yin. 2023. Exploring memoriza- tion in fine-tuned language models.arXiv preprint arXiv:2310.06714. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Car- lini. 2023. Counterfactual memorization in neural language models.Advances in Neural Information Processing Systems, 36:39321â39362. Kai Zhang, Lizhi Qing, Yangyang Kang, and Xiaozhong Liu. 2024a. Personalized llm response generation with parameterized memory injection.arXiv preprint arXiv:2404.03565. Shuning Zhang, Lyumanshan Ye, Xin Yi, Jingyu Tang, Bo Shui, Haobin Xing, Pengfei Liu, and Hewu Li. 2024b. " ghost of the past": identifying and resolving privacy leakage from llmâs memory through proactive user interaction.arXiv preprint arXiv:2410.14931. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert.arXiv preprint arXiv:1904.09675. Zeyu Zhang, Quanyu Dai, Luyu Chen, Zeren Jiang, Rui Li, Jieming Zhu, Xu Chen, Yi Xie, Zhenhua Dong, and Ji-Rong Wen. 2024c. Memsim: A bayesian sim- ulator for evaluating memory of llm-based personal assistants.arXiv preprint arXiv:2409.20163. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top- down approach to ai transparency.arXiv preprint arXiv:2310.01405.