Paper deep dive

Mitigating Privacy Seesaw in Large Language Models: Augmented Privacy Neuron Editing via Activation Patching

Xinwei Wu, Weilong Dong, Shaoyang Xu, Deyi Xiong

Year: 2024Venue: Findings of ACL 2024Area: Model EditingType: EmpiricalEmbeddings: 50

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 7:31:28 PM

Summary

The paper introduces the 'Privacy Seesaw' phenomenon in Large Language Models (LLMs), where protecting specific private data via neuron editing inadvertently increases leakage risks for other private information. To mitigate this, the authors propose APNEAP (Augmented Privacy Neuron Editing via Activation Patching), which uses data augmentation to expand the privacy dataset and activation patching to refine neuron editing, achieving a better balance between privacy protection and model performance.

Entities (5)

APNEAP · framework · 100%Privacy Seesaw · phenomenon · 100%DEPN · method · 95%GPT-2 · model · 95%Enron · dataset · 90%

Relation Signals (3)

APNEAP → mitigates → Privacy Seesaw

confidence 95% · The proposed APNEAP is capable of alleviating the privacy seesaw phenomenon

APNEAP → uses → Activation Patching

confidence 95% · it adapts activation patching to privacy neuron editing for switching off the second trigger to the privacy seesaw problem.

DEPN → causes → Privacy Seesaw

confidence 90% · DEPN, the baseline of privacy neuron based method, which suffers from privacy seesaw

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Xinwei Wu, Weilong Dong, Shaoyang Xu, Deyi Xiong. Findings of the Association for Computational Linguistics: ACL 2024. 2024.

Full Text

50,105 characters extracted from source content.

Expand or collapse full text

Findings of the Association for Computational Linguistics: ACL 2024, pages 5319–5332 August 11-16, 2024 ©2024 Association for Computational Linguistics Mitigating Privacy Seesaw in Large Language Models: Augmented Privacy Neuron Editing via Activation Patching Xinwei Wu 1 * , Weilong Dong 1 * , Shaoyang Xu 2 , Deyi Xiong 1,2† 1 College of Intelligence and Computing, Tianjin University, Tianjin, China 2 School of New Media and Communication, Tianjin University, Tianjin, China wuxw2021,willowd,syxu,dyxiong@tju.edu.cn Abstract Protecting privacy leakage in large language models remains a paramount challenge. In this paper, we reveal Privacy Seesaw in LLM privacy protection via neuron editing, a phe- nomenon where measures to secure specific private information inadvertently heighten ex- posure risks for other privacy. Through com- prehensive analysis, we identify the amount of targeted privacy data and the volume of edited privacy neurons as the two central triggers to this issue. To mitigate privacy seesaw, we pro- pose Augmented Privacy Neuron Editing via Activation Patching (APNEAP), a novel frame- work designed to well balance model perfor- mance with privacy protection. The proposed APNEAP augments collected private data by automatically synthesizing new private data, which deactivates the first trigger to the privacy seesaw issue. Additionally, it adapts activation patching to privacy neuron editing for switch- ing off the second trigger to the privacy seesaw problem. Experimental results show that the proposed APNEAP is capable of alleviating the privacy seesaw phenomenon and offers a more stable and reliable approach to privacy protection in LLMs than previous methods. 1 Introduction Large language models have demonstrated out- standing capabilities in natural language under- standing and generation, significantly advancing downstream natural language processing (NLP) tasks (Brown et al., 2020; Chung et al., 2022; Ouyang et al., 2022; Achiam et al., 2023). How- ever, LLMs trained on vast amounts of Internet data encounter critical security and privacy chal- lenges in real-life application scenarios (Shen et al., 2023; Sousa and Kern, 2023; Guo et al., 2023). This is mainly due to two reasons. First, training * Equal Contribution. † Corresponding author. Figure 1: The phenomenon ofPrivacy Seesaw. While the privacy neuron based method effectively reduces the privacy leakage risk of the targeted private data (texts 1-5), it paradoxically increases the risk for certain non- targeted private data (text 7). data for LLMs often contain sensitive or unautho- rized information, which is subjected to limited scrutiny because of its massiveness and confiden- tiality (Piktus et al., 2023; Li et al., 2023a). Second, LLMs tend to memorize training data, including unique instances (Carlini et al., 2019, 2021). Pre- vious studies have shown that private information could be successfully extracted from LLMs such as ChatGPT with meticulously crafted prompts, un- derscoring the urgency of privacy protection for LLMs (Li et al., 2023a). In order to protect privacy of LLMs, machine unlearning and neuron-based methods have been proposed. The former aims to make LLMs for- get targeted datasets through fine-tuning on small batches of data. Ishibashi and Shimodaira (2023) render private information harmless through Sani- tization Tuning. Jang et al. (2022) reduce privacy leakage risks by reversely learning the gradient of private data. The later seeks to reduce the like- lihood of eliciting private information by editing neurons directly. Wu et al. (2023) propose DEPN, an efficient approach to locating and editing privacy 5319 neurons for language models. Carlini et al. (2022b) show that there may be loopholes in the effectiveness evaluation of privacy protection methods, which may lead to the neglect of emerging privacy leakage risks. In our experi- ments, we have found that the neuron-based protec- tion method may lead to additional privacy leakage risks. Figure 1 illustrates our experimental results, clearly demonstrating the reduced risk of privacy leakage for memorized private data (Text 1-5) but increased risk associated with unmemorized private data (Text 7). This finding highlights the limitations of current neuron based protection approaches in fully addressing privacy-preserving scenarios. We refer to this phenomenon asPrivacy Seesaw(PS), where memorized private information is protected at the cost of exposing other private information that originally has no risk of leakage. We delve into the PS phenomenon (see details in Section 5.2), and find two main reasons for PS. First, the incomplete distribution of collected pri- vacy data represents only a small part of the entire privacy landscape. The second reason is limited number of privacy neurons that can be edited to avoid significant impact on the model performance, by current neuron-based methods (e.g., DEPN (Wu et al., 2023)). To address these issues, we propose Augmented Privacy Neuron Editing via Activation Patching (APNEAP), which employs data augmentation to expand the privacy dataset and adapts activation patching to efficient privacy neuron editing. The used data augmentation alleviates the first cause to PS while the adapted activation patching over- comes the constraints on the number of neurons that can be edited. Extensive experiments demon- strate the effectiveness of the proposed APNEAP in mitigating the PS and improving privacy protec- tion over strong baselines, while maintaining high efficiency and stability. Our contributions can be summarized as follows. •We unveil thePrivacy Seesaw, a phenomenon where targeted privacy is protected at the cost of other private information being exposed. Our analysis identifies its causes, offering new insights into the challenges of privacy protec- tion of LLMs. •We propose APNEAP to address the PS issue with two strategies: data augmentation for pri- vacy data expansion and activation patching for neuron editing. These strategies effectively counter the PS problem. •We conduct experiments to demonstrate that the proposed method is capable of protecting privacy leakage for large language models, and achieves stronger privacy protection per- formance than strong baselines. 2 Related Work Privacy Protection in NLPPrivacy protection in language models are categorized into three stages: data processing, training & fine-tuning, and post- processing (Guo et al., 2022; Sousa and Kern, 2023). In data processing, methods like redirec- tion and anonymization aim to remove sensitive information (Sousa and Kern, 2023; Brown et al., 2022). During training, differential privacy tech- niques (Li et al., 2021; Wu et al., 2022) are em- ployed at the expense of computational time and performance. Post-processing involves making models forget leaked information through machine unlearning (Eldan and Russinovich, 2023; Chen and Yang, 2023; Yao et al., 2023; Si et al., 2023) or neuron editing (Wu et al., 2023), fine-tuning on tar- get datasets or directly editing model parameters. Neuron EditingGeva et al. (2020) show that the feedforward network module in the Transformer can be viewed as a key-value memory, where each key corresponds to a text pattern and each value rep- resents a distribution over the vocabulary. Based on this finding, a series of studies (Geva et al., 2020; Meng et al., 2022; Dai et al., 2021; Wang et al., 2023) have proposed for editing factual knowl- edge encoded in pre-trained LLMs by locating neurons related to factual knowledge entities. Xu et al. (2023) discover that factual knowledge can be transferred across languages, with the cross-lingual alignment of knowledge neurons. Wu et al. (2023) extend this approach to privacy protection, aiming to safeguard private data by locating and editing privacy neurons. Activation PatchingActivation patching (AP) has been recently proposed to edit modify pre- trained models without full retraining them. This technique intervenes hidden states during inference, steering model outputs towards desired outcomes. Turner et al. (2023) demonstrate its application in generating outputs with specific emotional tones and entities. Similarly, Li et al. (2023b) apply it to enhance language models’ truthfulness by targeting 5320 specific attention heads. AP has also been explored for reducing toxic content and sycophantic expres- sions in model outputs (Rimsky, 2023; Leong et al., 2023). Xu et al. (2024) extend activation patching to a multilingual scenario, showcasing its capabil- ity to controlling language model behavior in a cross-lingual manner. Moreover, Zou et al. (2023) present a sophisticated method for manipulating model representations, proving its utility across various tasks. Dong et al. (2024) performs weak-to- strong alignment through concept vector patching into the residual stream. Activation patching repre- sents a significant advancement in model editing, offering a versatile tool for controlling and refining language model outputs. 3 Preliminary 3.1 Problem Formulation Privacy Leakage in LLMs:Letθdenote the parameters of a language modelM, withDrepre- senting the training dataset. ConsiderTas a subset ofDcontaining privacy-sensitive tuplest, each tuple consisting of a prefixXand private informa- tionY, whereY=y 1 ,...,y n is a sequence of private data. We define the probability of modelMgenerat- ing a privacy-sensitive tupletas: P t =P(Y|X,θ) = |Y| Y i=1 P(y i |X,θ),(1) IfP t exceeds a predefined thresholdτ, the model is considered as memorizing the privacy data, thereby posing a potential risk of privacy leakage. It is crucial to note that, due to the stochastic nature of model training and memorization, the actual set of privacy dataT ′ memorized byMis a subset ofT. Privacy Leakage in LLMs:Letθdenote the parameters of a language modelM, withDrepre- senting the training dataset. ConsiderTas a subset ofDcontaining privacy-sensitive tuplest, each tuple consisting of a prefixXand private informa- tionY, whereY=y 1 ,...,y n is a sequence of private data. We define the probability of modelMgenerat- ing a privacy-sensitive tupletas: P t =P(Y|X,θ),(2) P(Y|X,θ) =P(y 1 |X,θ) |Y| Y i=2 P(y i |y 1:i−1 ,X,θ). (3) IfP t exceeds a predefined thresholdτ, the model is considered as memorizing the privacy data, thereby posing a potential risk of privacy leakage. It is crucial to note that, due to the stochastic nature of model training and memorization, the actual set of privacy dataT ′ memorized byMis a subset ofT. Post-processing Privacy Protection:The pur- pose of post-processing privacy protection is to modify the model parametersθto ˆ θthrough the editing algorithmF edit , so thatF edit (θ,T ′ ) = ˆ θ minimizes the output probability of the entire pri- vacy data setT. The modified modelM ′ should show minimal performance degradation compared to the original modelM. This dual goal can be expressed as: min T X t=1 P(Y|X, ˆ θ),(γ M −γ M ′ ), (4) whereγdenotes the performance of a model on a specific dataset. This process involves minimizing the probability of generating each privacy data tu- ple inTwith the edited parameters ˆ θ , while also minimizing the performance gap between the new modelM ′ and the original modelM. 3.2 Memorized Data and Collected Data Carlini et al. (2022a) find that GPT-neo-6B has a 4% probability of memorizing training data. While memorized data is regarded as a target for protec- tion, the distribution of memorized private data is typically unknown. The pie charts in the upper left corner of Figure 2 shows: for the private data in the training dataset, an LLM usually memorizes only a small part of it. We refer to this subset as Memorized Data. However, privacy leaks often occur only when specific private data prefixes are inputted, suggesting that model developers may only be able to collect a fraction of the memorized data. This subset is referred to asCollected Data. For more details, see Section 5.1. 4Augmented Privacy Neuron Editing via Activation Patching In order to solve the challenge of PS, we propose APNEAP, illustrated in Figure 2, which includes two essential components: privacy data augmenta- tion and activation patching. The new framework contains four main modules: augmenting privacy data, locating privacy neurons, selecting privacy neurons, and editing privacy neurons. 5321 Figure 2: Illustration of Privacy Seesaw (top) and diagram of APNEAP (bottom). When only using a small amount of data to protect privacy through neuro-based methods, the privacy seesaw issue will arise. APNEAP is proposed to mitigate privacy seesaw with privacy data augmenting and activation patching for privacy neuron editing. 4.1 Augmenting Privacy Data The intuitive reason for the PS is that collected data is only a part of the overall privacy data. To address this challenge, we propose to expand privacy data through data augmentation. Specifically, we lever- age GPT-4 to simulate collected private texts and generate synthetic data. We input each collected private instance into GPT-4 with the prompt: “I am a privacy and security engineer. Please imi- tate the content and privacy level of the following data containing private information, and generate new private data.” We then mix the synthesized data with the collected data as the mixed privacy dataset. 4.2 Locating Privacy Neurons To locate the privacy neurons related to the mixed privacy dataset, we use the gradient attribution method from Wu et al. (2023), which evaluates the contribution of individual neurons in a lan- guage model to the leakage of private information. This technique measures the impact of neurons by altering their activation values and observing the resultant changes in the model’s output probabil- ities. Specifically, it calculates the privacy attri- bution score, which reflects a neuron’s influence on privacy-sensitive outputs. The privacy attribu- tion score is derived by progressively adjusting a neuron’s activation from zero to its original value and computing the change in output probability. The method employs the Riemann approximation to simplify the calculation, offering a practical ap- proach to assess the sensitivity of neurons to pri- vacy leakage. More details of calculating gradient attributions are shown in Appendix A.1. Generally, the privacy attribution score measures the neuron’s contribution to privacy information leakage, with a higher score indicating greater sensitivity of the neuron to privacy. 4.3 Selecting Privacy Neurons After locating the privacy neurons, each piece of private data yields an attribution score matrix cor- responding to the neuron dimension. Here we in- troduce a privacy neuron selecting method based on co-occurrence frequency. Initially, neurons with an attribution score sur- passing a certain percentage (typically 10%) of the 5322 maximum score are filtered for the single private data. Subsequently, for the entire privacy dataset, neurons with occurrence frequency exceeding a specific threshold (commonly 50% of the privacy dataset length) are selected. These thresholds gov- ern the number of neurons to be edited in subse- quent steps. Experimental findings suggest that while editing a larger number of neurons enhances privacy protection, it may also lead to a more pro- nounced impact on model performance. Detailed insights are provided in (§5.2). 4.4 Editing Privacy Neurons Wu et al. (2023) set the corresponding neuron ac- tivation values to zero, disrupting the information flow through these neurons. However, such a sim- ple method limits the number of editable privacy neurons, which can greatly damage the model per- formance when the number of edited neurons is large. In our experiments, we find that an insuffi- cient number of privacy neurons being edited will also lead to the emergence of the privacy seesaw phenomenon. Activation PatchingTo address this issue, we adapt activation patching to privacy neuron editing. The assumption behind activation patching is that concepts or polarities of the model exists in a linear form in the high-dimensional feature space (Zhang and Nanda, 2023; Syed et al., 2023; Zou et al., 2023). Based on this assumption, internal features of the model can be changed through the linear addition of steering vectors. Our adaptation is divided into three steps. First, we construct desensitized samples by replacing sen- sitive information with innocuous information (e.g., changing “call me at 912-####-123” to “call me at 000-0000-000”). Then, the pairs of desensitized samples and private samples are fed into the language model to have the privacy neuron activation values for sensitive samples and desensitized samples, H sen ,H des ,H∈R n∗m∗d , wherenis the num- ber of sentence pairs,mis the number of selected privacy neurons, anddis the hidden size of the language model. The steering vector is calculated by averaging the differences in activation values: V= P n i=1 (H sen i −H des i ) n ,V∈R m∗d .(5) Finally, steering vector addition is performed during model inference, where the activations of privacy neurons are steered by the vectorV through linear addition: ˆ H=H+α·V,(6) whereαis a hyperparameter used to control the intensity. We set it to 10 in our experiments. 5 Experiments In this section, we present our experimental setup and explain how we discovered the privacy see- saw phenomenon and analyzed its causes. We then demonstrate the effectiveness of our approach, in maintaining a balance between privacy protection and model performance while effectively mitigat- ing the privacy seesaw challenge. Our codes are available now. 1 5.1 Setup A. ModelsWe employed variants of GPT-2 (Rad- ford et al., 2019), GPT-Neo (Black et al., 2022). Due to computational constraints, main experi- ments were conducted using GPT-2, featuring 137M parameters, 12 layers, and 1024 embed- ding dimensions. More details are shown in Ap- pendix A.2. B. MetricsWe used three evaluation metrics. Valid-PPL:To measure the impact of various privacy-preserving approaches on model perfor- mance, we estimated the perplexity of the autore- gressive language modeling task on the Enron and MIMIC validation datasets.Exposure (Exp):Ex- posure metric (Carlini et al., 2019) is often used in privacy attacks to measure the risk of digital se- quence exposure.Mean Reciprocal Rank (MRR): Considering the multi-token nature of private se- quences such as names and emails, we employ the MRR of each target token according to Wu et al. (2023) to evaluate the model’s memorization of pri- vate sequences. The calculation formulas of these metrics are shown in Appendix A.4. C. DatasetWe usedEnron(Klimt and Yang, 2004) andMIMIC-Medical-Report(Johnson et al., 2018) as privacy datasets. The MIMIC- medical-report dataset masks name, age, and gen- der, so we filled the masked portion with fictitious private information (e.g., changing “___ year old female with chylothorax-” to “Sofia Turner is a 35- year-old female with chylothorax-”). As shown in Table 1, there are 27,450 phone number instances 1 https://github.com/flamewei123/APNEAP- 5323 Privacy TypeTELEmailMIMIC Total27,45090,31648,914 Memorized933815449 Proportion (%)0.34%4.2%0.92% Table 1: The amount of private data memorized by GPT2 after 10 epochs of fine-tuning. The thresholds for judging whether it is memorized are:Exp>15, MRR>80. and 90,316 email instances in the Enron dataset, and the MIMIC dataset contains 48,914 samples containing private information. We randomly se- lected 5% the data of Enron and MIMIC as the validation set for model performance evaluation. More dataset details are shown in Appendix A.3. D. Memorized and Collected DataAs indicated in Table 1, we identified private data memorized by GPT-2 after 10 epochs of fine-tuning, using the criteria ofExp>15andMRR>80. The memo- rization rates vary among different types of private data, with the highest rate observed in email data, potentially due to repeated mentions in the Enron dataset. In realistic scenarios, complete memorized texts are often inaccessible, and typically only texts with high leakage risks are detected. Hence, we selected texts withExp>20andMRR>90as collected texts, representing those with higher risks of privacy leakage. E. BaselinesTo evaluate the performance of the privacy neuron-based protection methods, we com- pare with three baselines.Differential Privacy (DP):A model training stage privacy protection approach, which introduces noise to gradients to reduce the model’s memorization of training data (Abadi et al., 2016; Habernal, 2021).Non-privacy Retraining (NR):To have the upper bound of the privacy preservation, we purged all private data from the training set and retrained the model on this sanitized dataset.DEPN:the baseline of privacy neuron based method, which suffers from privacy seesaw (Wu et al., 2023). 5.2 Empirical Analysis of the Privacy Seesaw and its Causes Privacy SeesawIn our experiments with GPT-2, we identified 93 memorized and 22 collected data instances by feed prefixes of private phone num- bers into the model. Utilizing the DEPN method, we located and edited privacy neurons associated with 22 collected phone numbers. The privacy pro- tection results, as detailed in Table 2a, indicate a reduction in the average risk of privacy leakage post-editing. However, a closer examination of the results reveals that not all data instances exhibit a decrease in the privacy leakage risk. Specifically, among the 22 collected data points, we observe that no instances show an increase in the risk of privacy leakage. In contrast, within the 93 memorized data instances, there are 3 cases where the risk unex- pectedly arises. More broadly, across the entire dataset, we find 977 instances with increased leak- age risk. These findings suggest that while DEPN can effectively lower average leakage risks across datasets and significantly protect the targeted sub- set of collected data, its protective measures do not uniformly extend to all data instances. In some cases, it may even exacerbate the risk of privacy leakage for certain private data. This discrepancy illustrates what we term thePrivacy Seesawphe- nomenon. In order to evaluate the harm of the privacy see- saw, we show the number of private data with three privacy risk change trends. “Positive (Pos)” de- notes the number of private data with reduced pri- vacy risk after being edited by privacy neurons, “Negative (Neg)” is the number of private data with increased privacy risk, and “Fixed” is the number of cases where privacy risk remains unchanged. We believe that when the number ofNegis 0, the privacy protection method does not have the risk of privacy seesaw. What Causes the Privacy Seesaw?Our investi- gations reveal two key factors contributing to the privacy seesaw phenomenon. The first factor isthe volume of target private datafor protection. To test this hypothesis, we used 93 memorized data instances for locating pri- vacy neurons instead of 22 collected data instances. Experiment results, detailed in Table 2b, show that no instances of increased privacy exposure risk among the memorized data are found, while the unmemorized data witness 842 negative instances. In comparison with Table 2a, these results suggest an alleviation of the privacy seesaw effect. The second factor isthe number of privacy neuronsfor editing. By modulating the selection threshold for privacy neurons, we observed the dy- namics of privacy leakage risks on the 93 mem- orized data points with different privacy neuron numbers. As illustrated in Table 3, an increase in the number of edited privacy neurons correlates 5324 Data TypeCountOriginal ExpNew ExpPosNegFixed Collected data2222.3612.472002 Memorized data9316.1311.858364 Unmemorized data27,3578.628.2822,9019773,479 (a) Locating privacy neurons by 22 collected data using DEPN. Data TypeCountOriginal ExpNew ExpPosNegFixed Collected data2222.3613.642002 Memorized data9316.1310.929102 Unmemorized data 27,3578.628.2623,0308423,485 (b) Locating privacy neurons by 93 memorized data using DEPN. Table 2: Illustration of the privacy seesaw phenomenon. “Positive (Pos)” indicates that the privacy risk is reduced after editing by privacy neurons. “Negative (Neg)” indicates that the privacy risk is increased. “Fixed” indicates that the privacy risk remains unchanged. pn_numValid-PPLExpPosNegFixed Original8.8318.13--- 10+ 8.7515.92253632 200+9.6114.26592014 400+9.8711.858364 2,500+16.748.189102 Table 3: Changes in exposure among the 93 memorized phone numbers (Exp>15) in a model that only re- moves privacy based on the 22 more easily detected phone numbers (Exp>20) under different levels of DEPN protection. “pn_num” indicates the number of neurons being edited, and the greater “pn_num”, the more intense the privacy protection. with a decrease in average leakage risk, albeit at the expense of model performance. Concurrently, there is an uptick in instances exhibiting reduced privacy leakage risk, coupled with a downtrend in cases exhibiting an escalation in risk. When the number of privacy neurons is larger than 2,500, the number of instances with increased leakage risk dwindles to zero, albeit significantly impair- ing the model’s performance, as evidenced by a Valid-PPL of 16.74. These findings highlight that while increasing the number of edited privacy neu- rons mitigates the privacy seesaw, it detrimentally affects model performance. The interplay between the two factors elucidates the root cause of the privacy seesaw: the inabil- ity of privacy neurons to encapsulate the entirety of privacy data. This flaw not only stems from the incomplete distribution of the collected privacy data, but also from the limitation of DEPN method, which inadvertently compromises the integrity of the privacy neurons. 5.3 The Effectiveness of APNEAP Overall PerformanceTable 4 presents the per- formance of various privacy-preserving methods, including our APNEAP and baselines. The results underscore the competitiveness of APNEAP. For Valid-PPL on the Enron and MIMIC validation datasets, models retrained by excluding private data show superior performance. In contrast, models employing Differential Privacy (DP) and DEPN exhibit significant performance degradation. How- ever, APNEAP achieves comparable, and in some cases, superior performance to the retrained model on the validation dataset, indicating that APNEAP exerts minimal impact on model performance. For privacy leakage risk indicators such as Ex- posure and MRR, original models trained directly on private data exhibit the highest risk. Both our method and other baselines manage to mitigate this risk, with APNEAP achieving a more significant reduction compared to DEPN. Remarkably, AP- NEAP can obtain comparable or even better results than the retrained model (NR) that excludes private data, showcasing its adept balance between model performance and privacy protection. In summary, APNEAP outperforms DEPN in terms of privacy protection, demonstrating its ef- fective balance between maintaining model perfor- mance and enhancing privacy protection. EfficiencyTable 4 also highlights the time effi- ciency of APNEAP compared to baselines. Due to the procedures of gradient clipping and noise addition, models with Differential Privacy (DP) require the longest processing time, followed by the retrained model (NR). DEPN showcases the highest time efficiency, with APNEAP displaying 5325 Privacy TypeModelValid-PPLRiskTime cost Phone Number Original Model8.8316.13- DEPN9.8711.850.5h DP11.3610.4575h NR9.033.4468h APNEAP8.929.23 0.7h EMAIL Original Model8.8388.47- DEPN10.4784.8327h DP11.3674.8375h NR9.0339.4768h APNEAP9.08 71.5530h MIMIC Original Model8.8382.77- DEPN10.1675.922h DP11.3668.1575h NR9.03 51.6868h APNEAP8.9864.393h Table 4: Comparison of performance metrics for privacy neuron-based methods and baselines in protecting private phone numbers, emails, and MIMIC (personal medical information). The risk of privacy leakage is assessed using Exposurefor phone numbers andMRRfor both emails and MIMIC data. Lower values indicate reduced leakage risk. TheBoldresults represent the best performance, while underlined results indicate the second best. ModelBefore EditingAfter EditingTime cost Valid-PPLExpValid-PPLExp gpt2 (137M)8.8316.138.929.230.7h gpt2-xl (1.6B)7.4214.277.559.693.9h gpt-neo (2.7B)7.3318.447.518.665.3h Table 5: Comparison of the efficiency of APNEAP across language models of varying sizes for the removal of private phone numbers. comparable efficiency. Additional experiments on larger models (GPT-2 XL, GPT-Neo) were conducted to assess the scala- bility of APNEAP. To counteract potential overfit- ting associated with the increased number of model parameters, we fine-tuned each model for fewer epochs (2 for GPT-2 XL, 1 for GPT-Neo). As shown in Table 5, the propensity of models to mem- orize private phone numbers escalates with their size. Nonetheless, the time cost associated with AP- NEAP only sees a marginal increase, illustrating the method’s high efficiency, even when applied to larger models. Additionally, APNEAP also maintainsstability, which have been proven in Appendix A.5. 5.4 Further Analysis Advantages of Activation PatchingIn our ex- periments, we specifically highlight the advantages of the activation patching method over the previous editing approach. Results, as presented in Table 6, pn_numValid-PPLExpPosNegFixed Original8.8316.13--- 400+9.929.238832 1,200+ 10.084.508913 2,500+ 10.202.719102 3,500+ 10.371.399003 Table 6: Comparison of activation patching vs zero- setting for privacy neuron editing. illustrate the efficacy of activation patching. No- tably, with an increase in the number of neurons edited, we observe a significant reduction in pri- vacy leakage risk, with minimal impact on model performance. Furthermore, this method effectively mitigates the privacy seesaw phenomenon. In con- trast, as seen in Table 3, the previous editing ap- proach limits the number of privacy neurons for editing due to its more pronounced effect on model performance. Activation patching, therefore, offers a more balanced solution, enabling the editing of a larger number of privacy neurons while better preserving the equilibrium between model perfor- mance and privacy protection. Ablation StudyTo validate the efficacy of the proposed components in APNEAP, we conducted a series of ablation studies to evaluate their individ- ual and combined effects on mitigating the privacy seesaw phenomenon. Specifically, we assessed the effect of privacy data augmentation only (DA + GA 5326 MethodsValid-PPLExpPosNegFixed Original GPT28.8316.13--- GA + Zero (DEPN)9.8711.858364 DA + GA + Zero 10.1611.728544 GA + AP8.929.448830 DA + GA + AP8.929.239102 Table 7: Ablation experiments on different components of APNEAP.GA: locating by gradient attribution.DA: data augmentation for privacy data.Zero: setting pri- vacy neurons to zero.AP: activation patching. + Zero). Experiment results in Table 7 show that it offers a moderate improvement over the original DEPN approach. Utilizing solely the Activation Patching editing method (GA + AP) yields a more pronounced enhancement in privacy protection per- formance. Notably, the concurrent application of both strategies effectively resolve the occurrence of negative results. These ablation studies underscore the contributions of each component in addressing the challenges posed by the privacy seesaw. 6 Future Work 6.1 Balancing Model Performance with Protection Strength and Breadth Previous research in privacy protection has high- lighted the importance of balancing model perfor- mance with protection strength (Abadi et al., 2016; Habernal, 2021). This balance is particularly chal- lenging, as demonstrated in works on differential privacy (Shi et al., 2021; Wu et al., 2022). In post- processing privacy protection for large language models, it’s impractical to have a complete dataset of private information. Protecting only a subset of private data fails to cover unknown private data, leading to a privacy seesaw effect. Future research should focus on achieving a balance between model performance, protection strength, and breadth in these scenarios. 6.2 Broader Privacy Types The definition of private information is inherently broad, often determined by the subject of the infor- mation (Sousa and Kern, 2023). Typically, privacy is defined narrowly, focusing on personally iden- tifiable information such as names, ID numbers, and phone numbers. However, with the routine use of conversational language models like ChatGPT, a broader scope of private information should be considered. Most current methods focus on protect- ing simple privacy phrases, but there is a growing need to address broader types of privacy in future research. 6.3 More Suitable Metrics While the Exposure index is a refined metric (Car- lini et al., 2019), it is less effective for evaluating longer sentences due to the inflated values resulting from a vast candidate space. Similarly, MRR can- not adequately account for the position and length of private information, particularly with large lan- guage models and long sentences (Carlini et al., 2022a). Developing diverse and suitable evaluation metrics for different privacy types is crucial. 6.4Optimization of Computational Efficiency While neuron-based methods are efficient, espe- cially compared to retraining, there is room for im- provement. For instance, Nanda (2023) proposed an approximation strategy to reduce the computa- tional complexity of obtaining attribution scores. As the time cost is directly proportional to the vol- ume of private data needing protection, enhancing the computational efficiency of neuron localization is essential for handling a larger amount of private data. 7 Conclusion In this paper, we have identified the privacy seesaw phenomenon as a previously underexplored prob- lem in LLM privacy protection, where efforts to protect certain private data instances inadvertently increase exposure risks for others. We pinpoint the amount of targeted privacy data and the number of privacy neurons being edited as key triggers of this phenomenon. To tackle this, we proposed AP- NEAP, effectively balancing model performance with privacy protection and significantly reducing privacy leaks. APNEAP also successfully mitigates the privacy seesaw issue, offering a more reliable privacy protection framework than previous neuron- based methods. While APNEAP shows promising results, further exploration in privacy neuron-based methods is needed. Acknowledgements The present research was supported by the National Key Research and Development Program of China (Grant No. 2023YFE0116400). We would like to thank the anonymous reviewers for their insightful comments. 5327 Limitations Although we have introduced APNEAP to reduce privacy leakage risks of LLMs, we recognize two limitations of APNEAP, which could guide our fu- ture research directions. Firstly, the metrics used to evaluate privacy leakage are not always intuitive for long sequences, limiting precise assessment of privacy risks in complex texts. Secondly, the computational efficiency of APNEAP, particularly regarding gradient attribution and activation patch- ing methods, needs improvement. Adopting paral- lel inference strategies could significantly enhance processing speed, crucial for larger datasets and complex models. Addressing these areas will ad- vance privacy protection in large language models, ensuring effectiveness and efficiency. Ethics Statement In this paper, we use the Enron and MIMIC datasets to evaluate the effect of privacy protection methods. Since the data comes from real persons, we masked sensitive information such as specific phone num- bers and emails in this paper. References Martin Abadi, Andy Chu, Ian Goodfellow, H Bren- dan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential pri- vacy. InProceedings of the 2016 ACM SIGSAC Con- ference on Computer and Communications Security, pages 308–318. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745. Hannah Brown,Katherine Lee,Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What does it mean for a language model to preserve privacy? InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 2280–2292. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022a. Quantifying memorization across neural lan- guage models.arXiv preprint arXiv:2202.07646. Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. 2022b. The privacy onion effect: Memo- rization is relative.Advances in Neural Information Processing Systems, 35:13263–13276. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Eval- uating and testing unintended memorization in neu- ral networks. In28th USENIX Security Symposium (USENIX Security 19), pages 267–284. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650. Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for llms.arXiv preprint arXiv:2310.20150. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021. Knowledge neurons in pretrained transformers.Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Weilong Dong, Xinwei Wu, Renren Jin, Shaoyang Xu, and Deyi Xiong. 2024. Contrans: Weak-to-strong alignment engineering via concept transplantation. arXiv preprint arXiv:2405.13578. Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms.arXiv preprint arXiv:2310.02238. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories.Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing. Shangwei Guo, Chunlong Xie, Jiwei Li, Lingjuan Lyu, and Tianwei Zhang. 2022. Threats to pre-trained lan- guage models: Survey and taxonomy.arXiv preprint arXiv:2202.06862. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey.arXiv preprint arXiv:2310.19736. 5328 Ivan Habernal. 2021. When differential privacy meets nlp: The devil is in the detail.Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Yoichi Ishibashi and Hidetoshi Shimodaira. 2023. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models.arXiv preprint arXiv:2210.01504. Alistair EW Johnson, David J Stone, Leo A Celi, and Tom J Pollard. 2018. The mimic code repository: en- abling reproducibility in critical care research.Jour- nal of the American Medical Informatics Association, 25(1):32–39. Bryan Klimt and Yiming Yang. 2004. Introducing the enron corpus. InCEAS, volume 45, pages 92–96. Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. 2023. Self-detoxifying language models via toxification reversal.arXiv preprint arXiv:2310.09573. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023a. Multi-step jailbreak- ing privacy attacks on chatgpt.arXiv preprint arXiv:2304.05197. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023b. Inference- time intervention: Eliciting truthful answers from a language model.arXiv preprint arXiv:2306.03341. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners.arXiv preprint arXiv:2110.05679. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt.Advances in Neural Information Pro- cessing Systems, 35:17359–17372. Neel Nanda. 2023. Attribution patching.https://w. neelnanda.io/mechanistic-interpretability. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744. Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Alexandra Sasha Luccioni, Yacine Jernite, and Anna Rogers. 2023. The roots search tool: Data transparency for llms. arXiv preprint arXiv:2302.14035. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9. Nina Rimsky. 2023. Reducing sycophancy and improv- ing honesty via activation steering. Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large language model align- ment: A survey.arXiv preprint arXiv:2309.15025. Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. 2021. Selective differential privacy for language modeling.Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. 2023. Knowledge unlearning for llms: Tasks, methods, and challenges. arXiv preprint arXiv:2311.15766. Samuel Sousa and Roman Kern. 2023. How to keep text private? a systematic review of deep learning methods for privacy-preserving natural language pro- cessing.Artificial Intelligence Review, 56(2):1427– 1492. Aaquib Syed, Can Rager, and Arthur Conmy. 2023. Attribution patching outperforms automated circuit discovery.arXiv preprint arXiv:2310.10348. Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. 2023. Acti- vation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248. Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al. 2023. Knowledge editing for large language models: A survey.arXiv preprint arXiv:2310.16218. Xinwei Wu, Li Gong, and Deyi Xiong. 2022. Adap- tive differential privacy for language model training. InProceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022), pages 21–26. Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023. Depn: Detecting and editing privacy neurons in pre- trained language models.Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing. Shaoyang Xu, Weilong Dong, Zishan Guo, Xinwei Wu, and Deyi Xiong. 2024. Exploring multilin- gual human value concepts in large language mod- els: Is value alignment consistent, transferable and controllable across languages?arXiv preprint arXiv:2402.18120. 5329 Shaoyang Xu, Junzhuo Li, and Deyi Xiong. 2023. Lan- guage representation projection: Can we transfer fac- tual knowledge across languages in multilingual lan- guage models?arXiv preprint arXiv:2311.03788. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning.arXiv preprint arXiv:2310.10683. Fred Zhang and Neel Nanda. 2023. Towards best prac- tices of activation patching in language models: Met- rics and methods.arXiv preprint arXiv:2309.16042. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top- down approach to ai transparency.arXiv preprint arXiv:2310.01405. 5330 A Appendix A.1 Gradient Attribution Letw k l represent a neuron to be evaluated by the privacy attribution method, wherelindicates the layer of the neuron in the language model, andk denotes its position. As described in §3.1, the prob- ability of the model outputting private information is: P(Y|X,w k l ) = |Y| Y i=1 P(y i |X,w k l =α k l )(7) whereα k l signifies the activation value of thek-th neuron in thel-th layer. The activation of the target neuron is gradually altered from0to its original value,β k l . Throughout this process, the cumulative gradient of the prob- ability change is calculated, representing the neu- ron’s contribution (i.e., privacy attribution score) to the privacy-sensitive output. The privacy attribu- tion score is formulated as: Att(w k l ) =β k l Z β k l 0 ∂P(Y|X,α k l ) ∂w k l dα k l (8) where ∂P(Y|X,α k l ) ∂w k l computes the gradient of the model output with respect tow k l . To circumvent the direct computation of continuous integrals, we employ the Riemann approximation: Att(w k l ) = β k l m P m j=1 ∂P(Y|X, j m β k l ) ∂w k l (9) wherem= 20denotes the number of approxima- tion steps. Given Eq 7, we obtain: Att(w k l ) = |Y| X i=1 β k l m P m j=1 ∂P(y i |X, j m β k l ) ∂w k l (10) Thus, the privacy attribution score measures the neuron’s contribution to privacy information leak- age, with a higher score indicating greater sensitiv- ity of the neuron to privacy. A.2 Models To assess the efficacy of privacy protection across various model sizes, we also utilized GPT2-XL (1.6B parameters: 48 layers and 1024 embedding dimensions), GPT-Neo (2.7B parameters: 32 layers and 2560 embedding dimensions). All experiments were executed on 4 NVIDIA RTX A6000 GPUs. A.3 Dataset Enron:The Enron dataset (Klimt and Yang, 2004) comprises over 500,000 public emails from 158 employees, released during Enron’s legal in- vestigation by the Federal Energy Regulatory Com- mission. 2 It’s the most extensive public collection of “real” email data, containing sensitive informa- tion like phone numbers and emails. As depicted in Table 1, there are 27,450 instances of phone numbers and 90,316 instances of emails within the dataset. We randomly selected 5% of the data from Enron as the validation set for model performance evaluation. MIMIC-Medical-Report:We utilized the de- identified MIMIC-I dataset (Johnson et al., 2018), which contains critical healthcare data from the ICU at the Beth Israel Deaconess Medical Center in Boston, MA. 3 The MIMIC-medical-report dataset contains 84K samples, with masked names, ages, and genders. We filled the masked sections with fictional private information (e.g., changing "___ year old woman chylothorax-" to "Sophia Turner is a 35 year old woman with chylothorax-"). Con- sequently, the dataset comprises 48,914 samples containing private information, as shown in Table 1. Similar to Enron, 5% of the data was sampled as the validation set. A.4 Metrics Valid-PPL:To gauge the impact of various pri- vacy preservation methods on model performance, we utilized the Perplexity of Autoregressive Lan- guage Modeling task on the Enron and MIMIC validation datasets. Exposure(Exp):The exposure metric (Carlini et al., 2019), commonly used in privacy attacks, measures the risk of number sequence exposure. For a number sequencec, a model with parameters θ, and a randomness spaceR, the exposuree θ is defined as: e θ = log 2 |R|−log 2 Rank θ (c).(11) Mean Reciprocal Rank (MRR):Given the multi-token nature of private sequences like names and emails, we adopted the MRR for each target to- ken to assess the model’s memorization of privacy 2 https://w.cs.cmu.edu/~enron/ 3 https://huggingface.co/datasets/ IndianaUniversityDatasetsModels/ MIMIC-medical-report 5331 Prompt => "713-####-229"Exposure "***-P, Contact me at"33.22 "***-P, Contact me at"13.28 "***-P, TEL:"12.48 "***-P, please call me at"13.76 "***-P, My phone number is"10.11 Table 8: A case study showcasing the stability of the neuron-based privacy protection method. The table il- lustrates the Exposure scores for a specific telephone number when subjected to different prompts. sequences, as per Wu et al. (2023). For a prefixQ and a privacy token sequenceE=e 1 ,...,e n , the model predicts the ranking of the target token asRank(e i |Q). The MRR for the privacy sequence Eis computed as: P |E| i=1 1 Rank(e i |Q) |E| .(12) A.5 Stability To evaluate the stability of the proposed APNEAP, we conducted experiments using different prompts to simulate varied inference scenarios. A stable privacy protection method should ensure that the protected private data remains secure, irrespective of the prompt used during the inference phase. Ta- ble 8 presents a case where the original private phone number was followed by “Contact me at”. The Exposure score dropped from 33.22 to 13.28 after editing. When we altered the prompts, the Exposure scores remained low, demonstrating the method’s robustness against variations in prompts. This underscores the high stability of the proposed APNEAP, ensuring consistent protection across dif- ferent scenarios. 5332

Instant research discovery