Paper deep dive
Extracting Training Data from Large Language Models
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel
Models: GPT-2 Medium (345M), GPT-2 Small (124M), GPT-2 XL (1.5B)
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/12/2026, 8:31:01 PM
Summary
This paper demonstrates that large language models (LLMs) like GPT-2 are vulnerable to training data extraction attacks, where an adversary can recover verbatim training examplesâincluding personally identifiable informationâusing only black-box query access. The authors define 'k-eidetic memorization' to quantify this phenomenon and show that larger models are more susceptible to such privacy leaks, highlighting the need for better safeguards in training large-scale models.
Entities (5)
Relation Signals (3)
Nicholas Carlini â authored â Extracting Training Data from Large Language Models
confidence 100% ¡ Extracting Training Data from Large Language Models Nicholas Carlini
OpenAI â developed â GPT-2
confidence 100% ¡ We use the GPT-2 model released by OpenAI
GPT-2 â vulnerableto â Training Data Extraction Attack
confidence 95% ¡ We demonstrate our attack on GPT-2... and are able to extract hundreds of verbatim text sequences
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.
Tags
Links
Trouble viewing inline? Open PDF directly â
Full Text
88,679 characters extracted from source content.
Expand or collapse full text
Extracting Training Data from Large Language Models Nicholas Carlini 1 Florian Tramèr 2 Eric Wallace 3 Matthew Jagielski 4 Ariel Herbert-Voss 5,6 Katherine Lee 1 Adam Roberts 1 Tom Brown 5 Dawn Song 3 Ălfar Erlingsson 7 Alina Oprea 4 Colin Raffel 1 1 Google 2 Stanford 3 UC Berkeley 4 Northeastern University 5 OpenAI 6 Harvard 7 Apple Abstract It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform atraining data extraction attackto recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the modelâs training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in justonedocument in the training data. We comprehensively evaluate our extraction attack to un- derstand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing pos- sible safeguards for training large language models. 1 Introduction Language models (LMs)âstatistical models which assign a probability to a sequence of wordsâare fundamental to many natural language processing tasks. Modern neural-network- based LMs use very large model architectures (e.g., 175 bil- lion parameters [7]) and train on massive datasets (e.g., nearly a terabyte of English text [55]). This scaling increases the ability of LMs to generate fluent natural language [53, 74, 76], and also allows them to be applied to a plethora of other tasks [29, 39, 55], even without updating their parameters [7]. At the same time, machine learning models are notorious for exposing information about their (potentially private) train- ing dataâboth in general [47, 65] and in the specific case of language models [8, 45]. For instance, for certain models it is known that adversaries can applymembership inference attacks[65] to predict whether or not any particular example was in the training data. GPT-2 East Stroudsburg Stroudsburg... Prefix --- Corporation Seabank Centre ------ Marine Parade Southport Peter W--------- -----------@---.------------.com +-- 7 5--- 40-- Fax: +-- 7 5--- 0--0 Memorized text Figure 1:Our extraction attack.Given query access to a neural network language model, we extract an individual per- sonâs name, email address, phone number, fax number, and physical address. The example in this figure shows informa- tion that is all accurate so we redact it to protect privacy. Such privacy leakage is typically associated withoverfitting [75]âwhen a modelâs training error is significantly lower than its test errorâbecause overfitting often indicates that a model has memorized examples from its training set. Indeed, overfitting is a sufficient condition for privacy leakage [72] and many attacks work by exploiting overfitting [65]. The association between overfitting and memorization hasâ erroneouslyâled many to assume that state-of-the-art LMs willnotleak information about their training data. Because these models are often trained on massive de-duplicated datasets only for a single epoch [7, 55], they exhibit little to no overfitting [53]. Accordingly, the prevailing wisdom has been that âthe degree of copying with respect to any given work is likely to be, at most,de minimisâ [71] and that models do not significantly memorize any particular training example. 1 arXiv:2012.07805v2 [cs.CR] 15 Jun 2021 Contributions.In this work, we demonstrate that large lan- guage models memorize and leak individual training exam- ples. In particular, we propose a simple and efficient method for extracting verbatim sequences from a language modelâs training set using only black-box query access. Our key in- sight is that, although training examples do not have notice- ably lower losses than test examples onaverage, certainworst- casetraining examples are indeed memorized. In our attack, we first generate a large, diverse set of high- likelihood samples from the model, using one of three general- purpose sampling strategies. We then sort each sample using one of six different metrics that estimate the likelihood of each sample using a separate reference model (e.g., another LM), and rank highest the samples with an abnormally high likelihood ratio between the two models. Our attacks directly apply to any language model, including those trained on sensitive and non-public data [10,16]. We use the GPT-2 model [54] released by OpenAI as a representative language model in our experiments. We choose to attack GPT-2 to minimize real-world harmâthe GPT-2 model and original training data source are already public. To make our results quantitative, we define a testable def- inition of memorization. We then generate1,800candidate memorized samples,100under each of the3Ă6attack config- urations, and find that over600of them are verbatim samples from the GPT-2 training data (confirmed in collaboration with the creators of GPT-2). In the best attack configuration, 67% of candidate samples are verbatim training examples. Our most obviously-sensitive attack extracts the full name, phys- ical address, email address, phone number, and fax number of an individual (see Figure 1). We comprehensively analyze our attack, including studying how model size and string fre- quency affects memorization, as well as how different attack configurations change the types of extracted data. We conclude by discussing numerous practical strategies to mitigate privacy leakage. For example, differentially-private training [1] is theoretically well-founded and guaranteed to produce private models if applied at an appropriate record level, but it can result in longer training times and typically degrades utility. We also make recommendations, such as carefully de-duplicating documents, that empirically will help to mitigate memorization but cannot prevent all attacks. 2 Background & Related Work To begin, we introduce the relevant background on large (billion-parameter) neural network-based language models (LMs) as well as data privacy attacks. 2.1 Language Modeling Language models are a fundamental building block of cur- rent state-of-the-art natural language processing pipelines [12, 31, 50, 52, 55]. While the unsupervised objectives used to train these models vary, one popular choice is a ânext-step predictionâ objective [5, 31, 44, 52]. This approach constructs a generative model of the distribution Pr(x 1 ,x 2 ,...,x n ), wherex 1 ,x 2 ,...,x n is a sequence of tokens from a vocabulary Vby applying the chain rule of probability Pr(x 1 ,x 2 ,...,x n ) =Î n i=1 Pr(x i |x 1 ,...,x iâ1 ). State-of-the-art LMs use neural networks to estimate this probability distribution. We letf θ (x i |x 1 ,...,x iâ1 )denote the likelihood of tokenx i when evaluating the neural net- workfwith parametersθ. While recurrent neural networks (RNNs) [26, 44] used to be a common choice for the neu- ral network architecture of LMs,attention-based models [4] have recently replaced RNNs in state-of-the-art models. In particular,TransformerLMs [70] consist of a sequence of at- tention layers and are the current model architecture of choice. Because we believe our results are independent of the exact architecture used, we will not describe the Transformer archi- tecture in detail here and instead refer to existing work [3]. Training Objective.A language model is trained to max- imize the probability of the data in a training setX. In this paper, each training example is a text documentâfor example, a specific news article or webpage from the internet. Formally, training involves minimizing the loss function L(θ) =âlogÎ n i=1 f θ (x i |x 1 ,...,x iâ1 ) over each training example in the training datasetX. Because of this training setup, the âoptimalâ solution to the task of language modeling is tomemorizethe answer to the ques- tion âwhat token follows the sequencex 1 ,...,x iâ1 ?â for ev- ery prefix in the training set. However, state-of-the-art LMs are trained with massive datasets, which causes them to not exhibit significant forms of memorization: empirically, the training loss and the test loss are nearly identical [7, 53, 55]. Generating Text.A language model can generate new text (potentially conditioned on some prefixx 1 ,...,x i ) by iteratively samplingËx i+1 âźf θ (x i+1 |x 1 ,...,x i )and then feedingËx i+1 back into the model to sampleËx i+2 âź f θ (x i+2 |x 1 ,...,Ëx i+1 ). This process is repeated until a desired stopping criterion is reached. Variations of this text generation method include deterministically choosing the most-probable token rather than sampling (i.e., âgreedyâ sampling) or setting all but the top-nprobabilities to zero and renormalizing the probabilities before sampling (i.e., top-nsampling 1 [18]). GPT-2. Our paper focuses on the GPT variant of Trans- former LMs [7, 52, 54]. Specifically, we demonstrate our train- ing data extraction attacks on GPT-2, a family of LMs that 1 For notational clarity, we write top-ninstead of the more common top-k because we will use the constantkfor a separate purpose. 2 were all trained using the same dataset and training algorithm, but with varying model sizes. GPT-2 uses a word-pieces [61] vocabulary with a byte pair encoder [22]. GPT-2 XL is the largest model with 1.5 billion parameters. For the remainder of this paper, the âGPT-2â model refers to this 1.5 billion parameter model or, when we specifically indicate this, its Small and Medium variants with 124 million and 334 million parameters, respectively. The GPT-2 model family was trained on data scraped from the public Internet. The authors collected a dataset by follow- ing outbound links from the social media website Reddit. The webpages were cleaned of HTML, with only the document text retained, and thende-duplicatedat the document level. This resulted in a final dataset of40GB of text data, over which the model was trained for approximately 12 epochs. 2 As a result, GPT-2 does not overfit: the training loss is only roughly10%smaller than the test loss across all model sizes. 2.2 Training Data Privacy It is undesirable for models to remember any details that are specific to their (potentially private) training data. The field of training data privacy develops attacks (to leak training data details) and defenses (to prevent leaks). Privacy Attacks.When models are not trained with privacy-preserving algorithms, they are vulnerable to numer- ous privacy attacks. The least revealing form of attack is the membership inference attack[28, 47, 65, 67]: given a trained model, an adversary can predict whether or not aparticular example was used to train the model. Separately, model inver- sion attacks [21] reconstruct representative views of a subset of examples (e.g., a model inversion attack on a face recog- nition classifier might recover a fuzzy image of a particular person that the classifier can recognize). Training data extraction attacks, like model inversion at- tacks, reconstruct training datapoints. However, training data extraction attacks aim to reconstructverbatimtraining exam- ples and not just representative âfuzzyâ examples. This makes them more dangerous, e.g., they can extract secrets such as verbatim social security numbers or passwords. Training data extraction attacks have until now been limited to small LMs trained on academic datasets under artificial training setups (e.g., for more epochs than typical) [8, 66, 68, 73], or settings where the adversary has a priori knowledge of the secret they want to extract (e.g., a social security number) [8, 27]. Protecting Privacy.An approach to minimizing memoriza- tion of training data is to apply differentially-private training techniques [1, 9, 43, 60, 64]. Unfortunately, training models with differentially-private mechanisms often reduces accu- racy [34] because it causes models to fail to capture the long 2 Personal communication with the GPT-2 authors. tails of the data distribution [19,20,67]. Moreover, it increases training time, which can further reduce accuracy because cur- rent LMs are limited by the cost of training [35, 38, 55]. As a result, state-of-the-art LMs such as GPT-2 [53], GPT-3 [7], and T5 [55] do not apply these privacy-preserving techniques. 3 Threat Model & Ethics Training data extraction attacks are often seen as theoretical or academic and are thus unlikely to be exploitable in practice [71]. This is justified by the prevailing intuition that privacy leakage is correlated with overfitting [72], and because state- of-the-art LMs are trained on large (near terabyte-sized [7]) datasets for a few epochs, they tend to not overfit [53]. Our paper demonstrates that training data extraction attacks arepractical. To accomplish this, we first precisely define what we mean by âmemorizationâ. We then state our threat model and our attack objectives. Finally, we discuss the ethical considerations behind these attacks and explain why they are likely to be a serious threat in the future. 3.1 Defining Language Model Memorization There are many ways to define memorization in language modeling. As mentioned earlier, memorization is in many ways anessentialcomponent of language models because the training objective is to assign high overall likelihood to the training dataset. LMs must, for example, âmemorizeâ the correct spelling of individual words. Indeed, there is a research direction that analyzes neural networks as repositories of (memorized) knowledge [51, 59]. For example, when GPT-2 is prompted to complete the sen- tence âMy address is 1 Main Street, San Francisco CAâ, it generates â94107â: a correct zip code for San Francisco, CA. While this is clearly memorization in some abstract form,we aim to formalize our definition of memorization in order to restrict it to cases that we might consider âunintendedâ [8]. 3.1.1 Eidetic Memorization of Text We define eidetic memorization as a particular type of mem- orization. 3 Informally, eidetic memorization is data that has been memorized by a model despite only appearing in a small set of training instances. The fewer training samples that con- tain the data, the stronger the eidetic memorization is. To formalize this notion, we first define what it means for a model to have knowledge of a strings. Our definition is loosely inspired by knowledge definitions in interactive proof systems [24]: a modelf θ knows a stringsifscan be extracted by interacting with the model. More precisely, we focus on black-boxinteractions where the model generatessas the most likely continuation when prompted with someprefix c: 3 Eidetic memory(more commonly calledphotographic memory) is the ability to recall information after seeing it only once. 3 Definition 1 (Model Knowledge Extraction)A stringsis extractable 4 from an LMf θ if there exists a prefixcsuch that: sâarg max s Ⲡ:|s Ⲡ|=N f θ (s Ⲡ|c) We abuse notation slightly here to denote byf θ (s Ⲡ|c)the likelihood of an entire sequences Ⲡ. Since computing the most likely sequencesis intractable for largeN, thearg maxin Definition 1 can be replaced by an appropriatesampling strat- egy(e.g., greedy sampling) that reflects the way in which the modelf θ generates text in practical applications. We then define eidetic memorization as follows: Definition 2 (k-Eidetic Memorization)A stringsisk- eidetic memorized (forkâĽ1) by an LMf θ ifsis extractable fromf θ andsappears in at mostkexamples in the training data X : | xâX:sâx | â¤k. Key to this definition is what âexamplesâ means. For GPT- 2, each webpage is used (in its entirety) as one training exam- ple. Since this definition counts the number of distinct training examples containing a given string, and not the total number of times the string occurs, a string may appear multiple times on one page while still counting ask=1 memorization. This definition allows us to define memorization as a spec- trum. While there is no definitive value ofkat which we might say that memorization is unintentional and potentially harm- ful, smaller values are more likely to be so. For any givenk, memorizing longer strings is also âworseâ than shorter strings, although our definition omits this distinction for simplicity. For example, under this definition, memorizing the correct spellings of one particular word is not severe if the word oc- curs in many training examples (i.e.,kis large). Memorizing the zip code of a particular city might be eidetic memorization, depending on whether the city was mentioned in many train- ing examples (e.g., webpages) or just a few. Referring back to Figure 1, memorizing an individual personâs name and phone number clearly (informally) violates privacy expectations, and also satisfies our formal definition: it is contained in just a few documents on the Internetâand hence the training data. 3.2 Threat Model Adversaryâs Capabilities.We consider an adversary who has black-box input-output access to a language model. This allows the adversary to compute the probability of arbitrary sequencesf θ (x 1 ,...,x n ), and as a result allows the adversary to obtain next-word predictions, but it does not allow the adversary to inspect individual weights or hidden states (e.g., attention vectors) of the language model. 4 This definition admits pathological corner cases. For example, many LMs when when prompted withâRepeat the following sentence: _____.âwill do so correctly. This allowsanystring to be âknownâ under our definition. Simple refinements of this definition do not solve the issue, as LMs can also be asked to, for example, down-case a particular sentence. We avoid these pathological cases by prompting LMs only with short prefixes. This threat model is highly realistic as many LMs are available through black-box APIs. For example, the GPT- 3 model [7] created by OpenAI is available through black-box API access. Auto-complete models trained on actual user data have also been made public, although they reportedly use privacy-protection measures during training [10]. Adversaryâs Objective.The adversaryâs objective is to ex- tract memorized training data from the model. The strength of an attack is measured by how private (formalized as being k-eidetic memorized) a particular example is. Stronger attacks extract more examples in total (both more total sequences, and longer sequences) and examples with lower values ofk. We do not aim to extracttargetedpieces of training data, but ratherindiscriminatelyextract training data. While targeted attacks have the potential to be more adversarially harmful, our goal is to study the ability of LMs to memorize data generally, not to create an attack that can be operationalized by real adversaries to target specific users. Attack Target.We select GPT-2 [54] as a representative LM to study for our attacks. GPT-2 is nearly a perfect target. First, from an ethical standpoint, the model and data are public, and so any memorized data that we extract isalreadypublic. 5 Second, from a research standpoint, the dataset (despite being collected from public sources) was never actually released by OpenAI. Thus, it is not possible for us to unintentionally âcheatâ and develop attacks that make use of knowledge of the GPT-2 training dataset. 3.3 Risks of Training Data Extraction Training data extraction attacks present numerous privacy risks. From an ethical standpoint, most of these risks are miti- gated in our paper because we attack GPT-2, whose training data is public. However, since our attacks would apply toany LM, we also discuss potential consequences of future attacks on models that may be trained on private data. Data Secrecy. The most direct form of privacy leakage oc- curs when data is extracted from a model that was trained on confidential or private data. For example, GMailâs auto- complete model [10] is trained on private text communica- tions between users, so the extraction of unique snippets of training data would break data secrecy. Contextual Integrity of Data.The above privacy threat corresponds to a narrow view of data privacy asdata secrecy. 5 Since the training data is sourced from the public Web, all the outputs of our extraction attacks can also be found via Internet searches. Indeed, to evaluate whether we have found memorized content, we search for the content on the Internet and are able to find these examples relatively easily. 4 A broader view of the privacy risks posed by data extrac- tion stems from the framework of data privacy ascontextual integrity[48]. That is, data memorization is a privacy in- fringement if it causes data to be used outside of its intended context. An example violation of contextual integrity is shown in Figure 1. This individualâs name, address, email, and phone number are notsecretâthey were shared online in a specific context of intended use (as contact information for a software project)âbut are reproduced by the LM in a separate context. Due to failures such as these, user-facing applications that use LMs may inadvertently emit data in inappropriate contexts, e.g., a dialogue system may emit a userâs phone number in response to another userâs query. Small-kEidetic Risks.We nevertheless focus onk-eidetic memorization with a smallkvalue because it makes extraction attacks more impactful.While there are cases where large-k memorization may still matter (for example, a company may refer to the name of an upcoming product multiple times in privateâand even though it is discussed often the name itself may still be sensitive) we study the small-kcase. Moreover, note that although we frame our paper as an âat- tackâ, LMs will output memorized dataeven in the absence of an explicit adversary. We treat LMs as black-box generative functions, and the memorized content that we extract can be generated through honest interaction with the LM. Indeed, we have even discovered at least one memorized training exam- ple among the1,000GPT-3 samples that OpenAI originally released in its official repository [49]. 3.4 Ethical Considerations In this paper, we will discuss and carefully examinespecific memorized content that we find in our extraction attacks. This raises ethical considerations as some of the data that we ex- tract contains information about individual users. As previously mentioned, we minimize ethical concerns by using data that is already public. We attack the GPT-2 model, which is available online. Moreover, the GPT-2 training data was collected from the public Internet [54], and is in principle available to anyone who performs the same (documented) collection process as OpenAI, e.g., see [23]. However, there are still ethical concerns even though the model and data are public. It is possibleâand indeed we find it is the caseâthat we might extract personal informa- tion for individuals from the training data. For example, as shown in Figure 1, we recovered a personâs full name, ad- dress, and phone number. In this paper, whenever we succeed in extracting personally-identifying information (usernames, phone numbers, etc.) we partially mask out this content with the token. We are aware of the fact that this does not provide complete mediation: disclosing that the vulnerability exists allows a malicious actor to perform these attacks on their own to recover this personal information. Just as responsible disclosure still causes some (limited) harm, we believe that the benefits of publicizing these attacks outweigh the potential harms. Further, to make our attacks public, we must necessarily reveal some sensitive information. We contacted the individual whose information is partially shown in Figure 1 to disclose this fact to them in advance and received permission to use this example. Our research findings have also been disclosed to OpenAI. Unfortunately, we cannot hope to contact all researchers who train large LMs in advance of our publication. We thus hope that this publication will spark further discussions on the ethics of memorization and extraction among other companies and research teams that train large LMs [2, 36, 55, 63]. 4 Initial Training Data Extraction Attack We begin with a simple strawman baseline for extracting training data from a language model in a two-step procedure. â˘Generate text.We generate a large quantity of data by unconditionally sampling from the model (Section 4.1). â˘Predict which outputs contain memorized text. We next remove the generated samples that are unlikely to contain memorized text using a membership inference attack (Section 4.2). These two steps correspond directly to extracting model knowledge (Definition 1), and then predicting which strings might bek-eidetic memorization (Definition 2). 4.1 Initial Text Generation Scheme To generate text, we initialize the language model with a one- token prompt containing a special start-of-sentence token and then repeatedly sample tokens in an autoregressive fashion from the model (see Section 2.1 for background). We hope that by sampling according to the modelâs assigned likelihood, we will sample sequences that the model considers âhighly likelyâ, and that likely sequences correspond to memorized text. Concretely, we sample exactly256tokens for each trial using the top-nstrategy from Section 2.1 withn=40. 4.2 Initial Membership Inference Given a set of samples from the model, the problem of training data extraction reduces to one of membership inference: pre- dict whether each sample was present in the training data [65]. In their most basic form, past membership inference attacks rely on the observation that models tend to assign higher con- fidence to examples that are present in the training data [46]. Therefore, a potentially high-precision membership inference classifier is to simply choose examples that are assigned the highest likelihood by the model. Since LMs areprobabilisticgenerative models, we follow prior work [8] and use a natural likelihood measure: theper- 5 plexityof a sequence measures how well the LM âpredictsâ the tokens in that sequence. Concretely, given a sequence of tokensx 1 ,...,x n , the perplexity is defined as P=exp ( â 1 n n â i=1 logf θ (x i |x 1 ,...,x iâ1 ) ) That is, if the perplexity is low, then the model is not very âsurprisedâ by the sequence and has assigned on average a high probability to each subsequent token in the sequence. 4.3 Initial Extraction Results We generate 200,000 samples using the largest version of the GPT-2 model (XL, 1558M parameters) following the text generation scheme described in Section 4.1. We then sort these samples according to the modelâs perplexity measure and investigate those with the lowest perplexity. This simple baseline extraction attack can find a wide va- riety of memorized content. For example, GPT-2 memorizes the entire text of the MIT public license, as well as the user guidelines of Vaughn Live, an online streaming site. While this is âmemorizationâ, it is onlyk-eidetic memorization for a large value ofkâthese licenses occur thousands of times. The most interesting (but still not eidetic memorization for low values ofk) examples include the memorization of popu- lar individualsâ Twitter handles or email addresses (omitted to preserve user privacy). In fact, all memorized content we identify in this baseline setting is likely to have appeared in the training dataset many times. This initial approach has two key weaknesses that we can identify. First, our sampling scheme tends to produce a low diversity of outputs. For example, out of the200,000samples we generated, several hundred are duplicates of the memo- rized user guidelines of Vaughn Live. Second, our baseline membership inference strategy suffers from a large number of false positives, i.e., content that is assigned high likelihood but is not memorized. The majority of these false positive samples contain ârepeatedâ strings (e.g., the same phrase repeated multiple times). Despite such text being highly unlikely, large LMs often incorrectly assign high likelihood to such repetitive sequences [30]. 5 Improved Training Data Extraction Attack The proof-of-concept attack presented in the previous section has low precision (high-likelihood samples are not always in the training data) and low recall (it identifies nok-memorized content for lowk). Here, we improve the attack by incorporat- ing better methods for sampling from the model (Section 5.1) and membership inference (Section 5.2). 5.1 Improved Text Generation Schemes The first step in our attack is to randomly sample from the lan- guage model. Above, we used top-nsampling and conditioned the LM on the start-of-sequence token as input. This strategy has clear limitations [32]: it will only generate sequences that are likely from beginning to end. As a result, top-nsampling from the model will cause it to generate the same (or similar) examples several times. Below we describe two alternative techniques for generating more diverse samples from the LM. 5.1.1 Sampling With A Decaying Temperature As described in Section 2.1, an LM outputs the probability of the next token given the prior tokensPr(x i |x 1 ,...,x iâ1 ). In practice, this is achieved by evaluating the neural networkz= f θ (x 1 ,...,x iâ1 )to obtain the âlogitâ vectorz, and then com- puting the output probability distribution asy=softmax(z) defined by softmax(z) i =exp(z i )/ â n j=1 exp(z j ). One can artificially âflattenâ this probability distribution to make the model less confident by replacing the output softmax(z)withsoftmax(z/t), fort>1. Here,tis called the temperature. A higher temperature causes the model to be less confident and more diverse in its output. However, maintaining a high temperature throughout the generation process would mean that even if the sampling process began to emit a memorized example, it would likely randomly step off the path of the memorized output. Thus, we use a softmax temperature that decays over time, starting att=10and decaying down tot=1over a period of the first20tokens (â10% of the length of the sequence). This gives a sufficient amount of time for the model to âexploreâ a diverse set of prefixes while also allowing it to follow a high-confidence paths that it finds. 5.1.2 Conditioning on Internet Text Even when applying temperature sampling, there are still some prefixes that are unlikely to be sampled but nevertheless occur in actual data. As a final strategy, our third sampling strategy seeds the model with prefixes from our own scrapes of the Internet. This sampling strategy ensures that we will generate samples with a diverse set of prefixes that are similar in nature to the type of data GPT-2 was trained on. We follow a different data collection process as used in GPT-2 (which follows Reddit links) in order to reduce the like- lihood that our dataset has any intersection with the modelâs training data. In particular, we select samples from a subset of Common Crawl 6 to feed as context to the model. 7 6 http://commoncrawl.org/ 7 It is possible there is some intersection between these two datasets, effec- tively allowing this strategy to âcheatâ. We believe this does not considerably affect results. First, any overlap between the two datasets is rare on average. Second, because we only use between the first5to10tokens of each sample, any possible overlap will be small in absolute terms. 6 200,000 LM Generations LM (GPT-2) Sorted Generations (using one of 6 metrics) Deduplicate Training Data Extraction Attack Prefixes Evaluation Internet Search Choose Top-100 Check Memorization Match No Match Figure 2:Workflow of our extraction attack and evaluation. 1) Attack.We begin by generating many samples from GPT-2 when the model is conditioned on (potentially empty) prefixes. We then sort each generation according to one of six metrics and remove the duplicates. This gives us a set of potentially memorized training examples.2) Evaluation.We manually inspect 100 of the top-1000 generations for each metric. We mark each generation as either memorized or not-memorized by manually searching online, and we confirm these findings by working with OpenAI to query the original training data. An open-source implementation of our attack process is available athttps://github.com/ftramer/LM_Memorization. As in prior work [55], we perform basic data-sanitization by removing HTML and JavaScript from webpages, and we de-duplicate data on a line-by-line basis. This gives us a dataset of50MB of text. We randomly sample between5and 10tokens of context from this scraped data and then continue LM generation with top-nsampling as in Section 4.1. 5.2 Improved Membership Inference Performing membership inference by filtering out samples with low likelihood has poor precision due to failures in the underlying language model: there are many samples that are assigned spuriously high likelihood. There are predominantly two categories of such samples: â˘Trivial memorization.We identify many cases where GPT-2 outputs content that is uninteresting because of how common the text is. For example, it repeats the num- bers from 1 to 100 with high probability. â˘Repeated substrings.One common failure mode of LMs is their propensity to repeatedly emit the same string over and over [30, 37]. We found many of the high-likelihood samples that are not memorized are indeed repeated texts (e.g., âI love you. I love you. . . â). Our insight is that we can filter out these uninteresting (yet still high-likelihood samples) by comparing to a second LM. Given a second model that accurately captures text likelihood, we should expect it willalsoassign high likelihood to these forms of memorized content. Therefore, a natural strategy for finding more diverse and rare forms of memorization is to filter samples where the original modelâs likelihood is âunexpectedly highâ compared to a second model. Below we discuss four methods for achieving this. Comparing to Other Neural Language Models.Assume that we have access to a second LM that memorizes a different set of examples than GPT-2. One way to achieve this would be to train a model on a disjoint set of training data, in which case it is unlikely that the two models will memorize the same data for smallk. An alternate strategy is to take a much smaller model trained on the same underlying dataset: because smaller models have less capacity for memorization, we conjecture that there are samples that arek-eidetic memorized (for small k) by the largest GPT-2 model, but which are not memorized by smaller GPT-2 models. Specifically, we use the Small (117M parameters) and Medium (345M parameters) models. Comparing to zlib Compression.It is not necessary that we compare to anotherneuralLM; any technique that quan- tifies some notion of âsurpriseâ for a given sequence can be useful. As a simple baseline method, we compute the zlib [41] entropy of the text: the number of bits of entropy when the sequence is compressed with zlib compression. We then use the ratio of the GPT-2 perplexity and the zlib entropy as our membership inference metric. Although text compressors are simple, they can identify many of the examples of trivial mem- orization and repeated patterns described above (e.g., they are excellent at modeling repeated substrings). Comparing to Lowercased Text.Instead of detecting memorization by comparing one model to another model, another option detects memorization by comparing the per- plexity of the model to the perplexity of thesamemodel on a âcanonicalizedâ version of that sequence. Specifically, we mea- sure the ratio of the perplexity on the sample before and after lowercasingit, which can dramatically alter the perplexity of memorized content that expects a particular casing. 7 Perplexity on a Sliding Window.Sometimes a model is not confident when the sample contains one memorized sub- string surrounded by a block of non-memorized (and high perplexity) text. To handle this, we use the minimum perplex- ity when averaged over a sliding window of 50 tokens. 8 6 Evaluating Memorization We now evaluate the various data extraction methods and study common themes in the resulting memorized content. 6.1 Methodology An overview of our experimental setup is shown in Figure 2. We first build three datasets of200,000generated samples (each of which is256tokens long) using one of our strategies: â˘Top-n(§4.1) samples naively from the empty sequence. â˘Temperature(§5.1.1) increases diversity during sampling. â˘Internet(§5.1.2) conditions the LM on Internet text. We order each of these three datasets according to each of our six membership inference metrics: â˘Perplexity: the perplexity of the largest GPT-2 model. â˘Small: the ratio of log-perplexities of the largest GPT-2 model and the Small GPT-2 model. â˘Medium: the ratio as above, but for the Medium GPT-2. ⢠zlib: the ratio of the (log) of the GPT-2 perplexity and the zlib entropy (as computed by compressing the text). â˘Lowercase: the ratio of perplexities of the GPT-2 model on the original sample and on the lowercased sample. â˘Window: the minimum perplexity of the largest GPT-2 model across any sliding window of 50 tokens. For each of these3Ă6=18configurations, we select100 samples from among the top-1000samples according to the chosen metric. 9 This gives us1,800total samples of poten- tially memorized content. In real-world attacks, adversaries will look to uncover large amounts of memorized content and thus may generate many more samples. We focus on a smaller set as a proof-of-concept attack. Data De-Duplication.To avoid âdouble-countingâ memo- rized content, we apply an automated fuzzy de-duplication step when we select the 100 samples for each configuration. Given a samples, we define thetrigram-multisetofs, de- notedtri(s)as a multiset of all word-level trigrams ins(with words split on whitespace and punctuation characters). For example, the sentence âmy name my name my nameâ has two trigrams (âmy name myâ and âname my nameâ) each of 8 Chosen after a cursory hyper-parameter sweep and manual analysis. 9 To favor low-ranked samples, while also exploring some of the higher- ranked samples, we select the100samples so that the fraction of selected samples with rank belowkis â k/1000. multiplicity2. We mark a samples 1 as a duplicate of another samples 2 , if their trigram multisets are similar, specifically if |tri(s 1 )âŠtri(s 2 )|âĽ|tri(s 1 )|/2. Evaluating Memorization Using Manual Inspection. For each of the1,800selected samples, one of four authors manually determined whether the sample contains memo- rized text. Since the training data for GPT-2 was sourced from the public Web, our main tool is Internet searches. We mark a sample as memorized if we can identify a non-trivial substring that returns anexact matchon a page found by a Google search. Validating Results on the Original Training Data. Fi- nally, given the samples that we believe to be memorized, we work with the original authors of GPT-2 to obtain lim- ited query access to their training dataset. To do this we sent them all1,800sequences we selected for analysis. For effi- ciency, they then performed a fuzzy3-gram match to account for memorization with different possible tokenizations. We marked samples as memorized if all3-grams in the mem- orized sequence occurred in close proximity in the training dataset. This approach eliminates false negatives, but has false positives. It can confirm that our samples are memorized but cannot detect cases where we missed memorized samples. In some experiments below, we report exact counts for how often a particular sequence occurs in the training data. We obtained these counts by asking the GPT-2 authors to perform a separategrepover the entire dataset to get an exact count. 6.2 Results In total across all strategies, we identify604unique memo- rized training examples from among the1,800possible can- didates, for an aggregate true positive rate of33.5%(our best variant has a true positive rate of67%). Below, we categorize what types of content is memorized by the model, and also study which attack methods are most effective. Categories of Memorized Content. We manually grouped the memorized samples into different categories (a descrip- tion of these categories is in Appendix A). The results are shown in Table 1. Most memorized content is fairly canonical text from news headlines, log files, entries from forums or wikis, or religious text. However, we also identify a significant amount of unique data, containing 128-bit UUIDs, (correctly- resolving) URLs containing random substrings, and contact information of individual people and corporations. In Sec- tion 6.3, we study these cases in more detail. Efficacy of Different Attack Strategies.Table 2 shows the number of memorized samples broken down by the dif- ferent text generation and membership inference strategies. 8 CategoryCount US and international news109 Log files and error reports79 License, terms of use, copyright notices54 Lists of named items (games, countries, etc.)54 Forum or Wiki entry53 Valid URLs50 Named individuals (non-news samples only)46 Promotional content (products, subscriptions, etc.)45 High entropy (UUIDs, base64 data)35 Contact info (address, email, phone, twitter, etc.)32 Code31 Configuration files30 Religious texts25 Pseudonyms15 Donald Trump tweets and quotes12 Web forms (menu items, instructions, etc.)11 Tech news11 Lists of numbers (dates, sequences, etc.)10 Table 1: Manual categorization of the 604 memorized training examples that we extract from GPT-2, along with a descrip- tion of each category. Some samples correspond to multiple categories (e.g., a URL may contain base-64 data). Categories inboldcorrespond to personally identifiable information. Sampling conditioned on Internet text is the most effective way to identify memorized content, however, all generation schemes reveal a significant amount of memorized content. For example, the baseline strategy of generating with top-n sampling yields191unique memorized samples, whereas conditioning on Internet text increases this to 273. As discussed earlier, looking directly at the LM perplexity is a poor membership inference metric when classifying data generated with top-nor temperature sampling: just 9% and 3% of inspected samples are memorized, respectively. The comparison-based metrics are significantly more effective at predicting if content was memorized. For example,67%of Internetsamples marked by zlib are memorized. Figure 3 compares the zlib entropy and the GPT-2 XL perplexity for each sample, with memorized examples high- lighted. Plots for the other strategies are shown in Figure 4 in Appendix B. Observe that most samples fall along a diagonal, i.e., samples with higher likelihood under one model also have higher likelihood under another model. However, there are numerous outliers in the top left: these samples correspond to those that GPT-2 assigns a low perplexity (a high likelihood) but zlib is surprised by. These points, especially those which are extreme outliers, are more likely to be memorized than those close to the diagonal. The different extraction methods differ in thetypeof mem- orized content they find. A complete breakdown of the data is given in Appendix A; however, to briefly summarize: 123456789 GPT-2 Perplexity 100 200 300 400 500 600 700 800 zlib Entropy All Samples Selected Memorized Figure 3: The zlib entropy and the perplexity of GPT-2 XL for 200,000samples generated with top-nsampling. In red, we show the 100 samples that were selected for manual inspec- tion. In blue, we show the 59 samples that were confirmed as memorized text. Additional plots for other text generation and detection strategies are in Figure 4. 1.The zlib strategy often finds non-rare text (i.e., has a high k-memorization). It often finds news headlines, license files, or repeated strings from forums or wikis, and there is only one âhigh entropyâ sequence this strategy finds. 2. Lower-casing finds content that is likely to have irregular capitalization, such as news headlines (where words are capitalized) or error logs (with many uppercase words). 3.The Small and Medium strategies often find rare content. There are 13 and 10 high entropy examples found by us- ing the Small and Medium GPT-2 variants, respectively (compared to just one with zlib). 6.3 Examples of Memorized Content We next manually analyze categories of memorized content that we find particularly compelling. (Additional examples are presented in Appendix C.) Recall that since GPT-2 is trained on public data, our attacks are not particularly severe. Nevertheless, we find it useful to analyze what we are able to extract to understand the categories of memorized contentâ with the understanding that attacking a model trained on a sensitive dataset would give stronger results. Personally Identifiable Information.We identify numer- ous examples of individual peoplesâ names, phone numbers, addresses, and social media accounts. 9 Inference Strategy Text Generation Strategy Top-nTemperatureInternet Perplexity9339 Small414258 Medium383345 zlib594667 Window332858 Lowercase532260 Total Unique191140273 Table 2: The number of memorized examples (out of 100 candidates) that we identify using each of the three text gen- eration strategies and six membership inference techniques. Some samples are found by multiple strategies; we identify 604 unique memorized examples in total. We find 46 examples that contain individual peoplesâ names. When counting occurrences of named individuals, we omit memorized samples that relate to national and in- ternational news (e.g., if GPT-2 emits the name of a famous politician, we do not count this as a named individual here). We further find 32 examples that contain some form of contact information (e.g., a phone number or social media handle). Of these, 16 contain contact information for businesses, and 16 contain private individualsâ contact details. Some of this memorized content is exclusive to just a few documents. For example, we extract the usernames of six users participating in an IRC conversation that appeared in exactly one training document. URLs.We identify50examples of memorized URLs that correctly resolve to live webpages. Many of these URLs con- tain uncommon pieces of text, such as random numbers or base-64 encoded strings. We also identify several URLs that resolve correctly but we cannot identify their source (and we thus do not count them as âmemorizedâ in our evaluation). Code. We identify31generated samples that contain snip- pets of memorized source code. Despite our ability to recover the source code verbatim, we are almost alwaysunableto recover the original authorship notices or terms of use. Often, this information is given either before the code itself or in a LICENSE file that appears separately. For many of these sam- ples, we can alsoextendtheir length and recover thousands of lines of (near verbatim) source code (see Section 6.4). Unnatural Text.Memorization is not limited to natural- looking text. We find21instances of random number se- quences with at least 50 bits of entropy. 10 For example, we 10 We estimate the entropy through manual analysis by guessing the entropy space given the format of the string. Memorized String Sequence Length Occurrences in Data DocsTotal Y2......y587110 7C......1840122 XM... ...WA54136 ab......2c64149 f......af32164 C7......ow43183 0x... ...C010196 76......84171122 a7... ...4b401311 Table 3:Examples ofk=1eidetic memorized, high- entropy content that we extractfrom the training data. Each is contained injust onedocument. In the best case, we extract a 87-characters-long sequence that is contained in the training dataset just 10 times in total, all in the same document. extract the following UUID: 1e4bd2a8-e8c8-4a62-adcd-40a936480059 from the model; a Google search for this string identifies just 3 documents containing this UUID, and it is contained in just one GPT-2 training document (i.e., it is1-eidetic memorized). Other memorized random number sequences include UUIDs contained in only a few documents (not listed to preserve privacy), git commit hashes, random IDs used for ad tracking, and product model numbers. Table 3 gives nine examples ofk=1eidetic memorized content, each of which is a random sequences between10 and87characters long. In each of these cases, the memorized example is contained in exactlyonetraining document, and the total number of occurrences within that single document varies betweenjust 10and 311. Data From Two Sources.We find samples that contain two or more snippets of memorized text that are unrelated to one another. In one example, GPT-2 generates a news article about the (real) murder of a woman in 2013, but then attributes the murder to one of the victims of a nightclub shooting in Orlando in 2016. Another sample starts with the memorized Instagram biography of a pornography producer, but then goes on to incorrectly describe an American fashion model as a pornography actress. This type of generation is notk-eidetic memorization (these independent pieces of information never appear in the same training documents), but it is an example of a contextual integrity violation. Removed Content. Finally, GPT-2 memorizes content that has since been removed from the Internet, and is thus now primarilyaccessible through GPT-2. We are aware of this content as it is still cached by Google search, but is no longer 10 present on the linked webpage. Some of this data is not par- ticularly interesting in its own right, e.g., error logs due to a misconfigured webserver that has since been fixed. However, the fact that this type of memorization occurs highlights that LMs that are trained entirely on (at-the-time) public data may end up serving as an unintentional archive for removed data. 6.4 Extracting Longer Verbatim Sequences In our previous experiments, we extract strings of 256 tokens in length. Here, we briefly investigate if we can extract longer sequences. In particular, we extend the length of some of the memorized sequences by seeding the model with each sample and continuing to generate. To do this, we apply a beam- search-like decoding method introduced in prior work [8] instead of greedy decoding which often fails to generate long verbatim sequences. We can extend many of the memorized samples. For exam- ple, we identify a piece of source code taken from a repository on GitHub. We can extend this snippet to extract an entire file, namely1450 linesof verbatim source code. We can also extract the entirety of the MIT, Creative Commons, and Project Gutenberg licenses. This indicates that while we have extracted604memorized examples, we could likely extend many of these to much longer snippets of memorized content. 6.5 Memorization is Context-Dependent Consistent with recent work on constructing effective âpromptsâ for generative LMs [7, 62], we find that the memo- rized content is highly dependent on the modelâs context. For example, GPT-2 will complete the prompt â3.14159â with the first25digits ofĎcorrectly using greedy sampling. However, we find that GPT-2 âknowsâ (under Definition 2) more digits ofĎbecause using the beam-search-like strategy introduced above extracts 500 digits correctly. Interestingly, by providing the more descriptive prompt âpi is 3.14159â, straight greedy decoding gives the first799 digits ofĎâmore than with the sophisticated beam search. Further providing the context âe begins 2.7182818, pi begins 3.14159â, GPT-2 greedily completes the first824digits ofĎ. This example demonstrates the importance of the context: in the right setting, orders of magnitude more extraction is feasible than when the context is just slightly suboptimal. We find that this holds true for our memorized examples as well. None of the273extracted samples found using Internet conditioning can be reliably reproduced when using the same prefix initially provided to GPT-2 that produced this sample. However, nearly all can be reproduced with high probability if we provided the entire sequence of data up to (but not including) the beginning of the memorized content. The important lesson here is that our work vastlyunder- estimatesthe true amount of content that GPT-2 memorized. There are likely prompts that would identify much more mem- orized content, but because we stick to simple prompts we do not find this memorized content. 7 Correlating Memorization with Model Size & Insertion Frequency Thus far, we have shown that language models can memorize verbatim training strings, even when they are trained for few epochs and achieve small train-test accuracy gaps. A natural question is how many times a string must appear for it to be memorized (i.e.,kin Definition 2). Prior work has investigated LM memorization by varying the number of times particular âcanaryâ tokens were inserted into a training dataset [8]. The main limitation of this approach is that it is synthetic: canaries are inserted artificially after the dataset has been collected and may not be representative of natural data. Here, we study how well GPT-2 memorizesnaturally oc- curringcanaries in the training data. In particular, we consider a piece of memorized content with the following prefix: "color":"fuchsia","link":"https://w. reddit.com/r/The_Donald/comments/ Thereddit.comURL above is completed by a specific 6-character article ID and a title. We located URLs in this specific format in a single document onpastebin.com. Each URL appears a varying number of times in this document, and hence in the GPT-2 training dataset. 11 Table 4 shows a subset of the URLs that appear more than once, and their respective counts in the document. 12 This allows us to ask the question: how many times must an example appear in the training dataset for us to extract it? Methods.We attempt two approaches to extract URLs of this format, and run three variants of GPT-2 (XL, Medium, and Small). The two approaches vary the âdifficultyâ of the attack, so even if the more difficult fails the easier may succeed. First, we directly prompt each variant of GPT-2 with the prefix above, and use top-nsampling to generate10,000pos- sible extensions. Then, we test whether any of the URLs in the training document were among those that were emitted by GPT-2. We count a URL as emitted if it matches verbatim with one of the 10,000 generations. Some URLs are not extractable with this technique, and so we make the problem easier for GPT-2 by additionally providing GPT-2 the 6-character random token that begins each URL. Given this additional prefix, we then sample from 11 The purpose of this text dump was to tag users of Reddit who posted frequently on specific topics. In doing so, this page repeats some of the same links many times because many users comment on the same links. 12 We confirmed with OpenAI that the counts here are within 5% of the true counts of these URLs in the training data. 11 OccurrencesMemorized? URL (trimmed)DocsTotalXL MS /r/51y/milo_evacua...1359X 1 /2 /r/zin/hi_my_name...1113X /r/ 7ne/for_all_yo...176X 1 /2 /r/5mj/fake_news_...172X /r/5wn/reddit_admi...164X /r/lp8/26_evening...156X /r/ jla/so_pizzagat...151X 1 /2 /r/ubf/late_night...151X 1 /2 /r/ eta/make_christ...135X 1 /2 /r/6ev/its_officia...133X /r/3c7/scott_adams...117 /r/k2o/because_his...117 /r/ tu3/armynavy_ga...18 Table 4: We show snippets of Reddit URLs that appear a varying number of times in asingletraining document. We condition GPT-2 XL, Medium, or Small on a prompt that contains the beginning of a Reddit URL and report aXif the corresponding URL was generated verbatim in the first 10,000generations. We report a 1 /2if the URL is generated by providing GPT-2 with the first 6 characters of the URL and then running beam search. the model using the beam search procedure. This task is eas- ier in two ways: we have first provided more context and additionally use a higher recall sampling strategy. Results.Table 4 summarizes the key results. Under the more difficult of the two approaches, the full-sized 1.5 billion parameter GPT-2 model emits all examples that are inserted 33 times or more, the medium-sized 345 million parameter memorizes half of the URLs, and the smallest 117 million parameter model memorizesnoneof these URLs. When given the additional context and using beam search, the medium model can emit four more URLs, and the small model only emits the one URL that was inserted 359 times. These results illustrate two fundamental lessons in LM memorization. First,largermodels memorize significantly more training data: even hundreds of millions of parameters are not enough to memorize some of the training points. The ability of LMs to improve with model size has been exten- sively studied [35, 38]; we show a negative trend where these improvements come at the cost of decreased privacy. Second, for the largest LM, complete memorization occurs after just 33insertions. This implies that any potentially sensitive infor- mation that is repeated a non-trivial amount of times is at risk for memorization, even if it was only repeated multiple times in a single training document. 8 Mitigating Privacy Leakage in LMs Now that we have shown that memorized training data can be extracted from LMs, a natural question is how to mitigate these threats. Here we describe several possible strategies. Training With Differential Privacy.Differential privacy (DP) [13, 14] is a well-established notion of privacy that of- fers strong guarantees on the privacy of individual records in the training dataset. Private machine learning models can be trained with variants of the differentially private stochastic gra- dient descent (DP-SGD) algorithm [1] which is widely imple- mented [17, 25]. Large companies have even used DP in pro- duction machine learning models to protect usersâ sensitive information [15,69]. The tradeoffs between privacy and utility of models have been studied extensively: differentially-private training typically prevents models from capturing the long tails of the data distribution and thus hurts utility [19, 20, 67]. In the content of language modeling, recent work demon- strates the privacy benefits of user-level DP models [56]. Un- fortunately, this work requires labels for which users con- tributed each document; such labels are unavailable for data scraped from the open Web. It may instead seem natural to aim for DP guarantees at the granularity of individual web- pages, but rare snippets of text (e.g., an individualâs name and contact information as in Figure 1) might appear in more than one webpage. It is thus unclear how to apply DP in a principled and effective way on Web data. Curating the Training Data.One cannot manually vet the extremely large training datasets used for training LMs. How- ever, there are methods to limit the amount of sensitive con- tent that is present, e.g., by identifying and filtering personal information or content with restrictive terms of use [11, 58]. Aside from attempting to remove sensitive content, it is also important to carefully de-duplicate the data. Many lan- guage modeling datasets are de-duplicated at the document- or paragraph-level, which means that a single document can still contain many repeated occurrences of a sensitive piece of content. We envision more sophisticated strategies to de- duplicate the training data, or limit the contribution of any single source of training data. It is also vital to carefully source the training data. Many of the potentially-sensitive training examples that we extracted (e.g., individualsâ personal information) came from websites that are known to host sensitive content, e.g.,pastebinis the 12th most popular domain in GPT-2âs training set. Overall, sanitizing data is imperfectâsome private data will always slip throughâand thus it serves as a first line of defense and not an outright prevention against privacy leaks. Limiting Impact of Memorization on Downstream Appli- cations. In many downstream applications, e.g., dialogue 12 systems [76] and summarization models [29], LMs arefine- tunedon task-specific data. On the positive side, this finetun- ing process may cause the LM to âforgetâ [42, 57] some of the data that is memorized during the pre-training stage. On the negative side, fine-tuning may introduce its own privacy leakages if the task-specific data also contains private infor- mation. An interesting direction for future work is to explore how memorization is inherited by fine-tuned models. Downstream applications built on top of language models could also attempt tofilter outgenerated text that contains memorized content, if such content can be reliably detected (e.g., using various membership inference strategies). Auditing ML Models for Memorization.Finally, after mitigating privacy leaks, it is vital to audit models to empiri- cally determine the privacy level they offer in practice [33]. Auditing is important even when using differential privacy, as it can complement theoretical upper bounds on privacy leakage [1]. We envision using our proposed methods, as well as existing attacks [8, 33, 65, 72], to audit LMs. 9 Lessons and Future Work Extraction Attacks Are a Practical Threat. Prior work shows that (100Ăto1000Ăsmaller) language models poten- tially memorize training data in semi-realistic settings [8, 73]. Our results show that state-of-the-art LMsdomemorize their training data in practice, and that adversaries can extract this data with simple techniques. Our attacks are practical even when the data contains a given sequence only a few times. As our attacks interact with a language model as a black- box, our results approximate theworst-casebehavior of lan- guage models when interacting with benign users. In particu- lar, among600,000(honestly) generated samples, our attacks find that at least 604 (or 0.1%) contain memorized text. Note that this is likely an extremely loose lower bound. We only manually inspected1,800potential candidate memorized samples; if we had started with more candidates we would likely have identified significantly more memorized content. Developing improved techniques for extracting memorized data, including attacks that are targeted towards specific con- tent, is an interesting area for future work. Memorization Does Not Require Overfitting.It is often believed that preventing overfitting (i.e., reducing the train- test generalization gap) will prevent models from memorizing training data. However, large LMs have no significant train- test gap, and yet we still extract numerous examples verbatim from the training set. The key reason is that even though onaveragethe training loss is only slightly lower than the validation loss, there are still some training examples that have anomalously low losses. Understanding why this happens is an important problem for future work [6, 40]. Larger Models Memorize More Data.Throughout our experiments, larger language models consistently memorized more training data than smaller LMs. For example, in one setting the1.5billion parameter GPT-2 model memorizes over18Ăas much content as the124million parameter model (Section 7). Worryingly, it is likely that as LMs become bigger (in fact they already are100Ălarger than the GPT-2 model we study [7]), privacy leakage will become even more prevalent. Memorization Can Be Hard to Discover.Much of the training data that we extract is only discovered when prompt- ing the LM with a particular prefix. Currently, we simply attempt to use high-quality prefixes and hope that they might elicit memorization. Better prefix selection strategies [62] might identify more memorized data. Adopt and Develop Mitigation Strategies.We discuss several directions for mitigating memorization in LMs, in- cluding training with differential privacy, vetting the training data for sensitive content, limiting the impact on downstream applications, and auditing LMs to test for memorization. All of these are interesting and promising avenues of future work, but each has weaknesses and are incomplete solutions to the full problem. Memorization in modern LMs must be ad- dressed as new generations of LMs are emerging and becom- ing building blocks for a range of real-world applications. 10 Conclusion For large language models to be widely adopted, they must address the training data memorization problems that we have identified. Our extraction attacks are practical and efficient, and can recover hundreds of training examples from a model, even when they are contained in just one training document. Our analysis is best viewed as a cautionary tale of what could happen when training large LMs on sensitive data. Even though our attacks target GPT-2 (which allows us to ensure that our work is not harmful), the same techniques apply to any LM. Moreover, because memorization gets worse as LMs become larger, we expect that these vulnerabilities will become significantly more important in the future. There will therefore need to be techniques developed to specifically address our attacks. Training with differentially- private techniques is one method for mitigating privacy leak- age, however, we believe that it will be necessary to develop new methods that can train models at this extreme scale (e.g., billions of parameters) without sacrificing model accuracy or training time. More generally, there are many open ques- tions that we hope will be investigated further, including why models memorize, the dangers of memorization, and how to prevent memorization. 13 Acknowledgements We are grateful for comments on early versions of this paper by Dan Boneh, Andreas Terzis, Carey Radebaugh, Daphne Ip- polito, Christine Robson, Kelly Cooke, Janel Thamkul, Austin Tarango, Jack Clark, Ilya Mironov, and Om Thakkar. Florian Tramèr is supported by NSF award CNS-1804222. Summary of Contributions ⢠Nicholas, Dawn, Ariel, Tom, Colin and Ălfar proposed the research question of extracting training data from GPT-2 and framed the threat model. â˘Colin, Florian, Matthew, and Nicholas stated the memoriza- tion definitions. ⢠Florian, Ariel, and Nicholas wrote code to generate candi- date memorized samples from GPT-2 and verify the ground truth memorization. â˘Florian, Nicholas, Matthew, and Eric manually reviewed and categorized the candidate memorized content. ⢠Katherine, Florian, Eric, and Colin generated the figures. â˘Adam, Matthew, and Eric ran preliminary investigations in language model memorization. â˘Nicholas, Florian, Eric, Colin, Katherine, Matthew, Ariel, Alina, Ălfar, Dawn, and Adam wrote and edited the paper. â˘Tom, Adam, and Colin gave advice on language models and machine learning background. ⢠Alina, Ălfar, and Dawn gave advice on the security goals. References [1] MartĂn Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InACM CCS, 2016. [2]Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open-domain chatbot.arXiv preprint arXiv:2001.09977, 2020. [3]Jay Alammar. The illustrated transformer.Visualizing Machine Learning One Concept at a Time, 2018. [4]Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. Neural machine translation by jointly learning to align and translate. InICLR, 2015. [5]Yoshua Bengio, RĂŠjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. JMLR, 2003. [6]Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrele- vant training data necessary for high-accuracy learning? arXiv preprint arXiv:2012.06421, 2020. [7]Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020. [8]Nicholas Carlini, Chang Liu, Ălfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2019. [9]Kamalika Chaudhuri and Claire Monteleoni. Privacy- preserving logistic regression. InNIPS, 2009. [10]Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. Gmail smart compose: Real-Time assisted writing. InKDD, 2019. [11]Andrea Continella, Yanick Fratantonio, Martina Lindor- fer, Alessandro Puccetti, Ali Zand, Christopher Kruegel, and Giovanni Vigna. Obfuscation-Resilient Privacy Leak Detection for Mobile Apps Through Differential Analysis. InNDSS, 2017. [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In NAACL, 2019. [13]C Dwork, F McSherry, K Nissim, and A Smith. Cali- brating noise to sensitivity in private data analysis. In TCC, 2006. [14] Cynthia Dwork. Differential privacy: A survey of results. InTAMC, 2008. [15]Ălfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. RAPPOR: Randomized aggregatable privacy-preserving ordinal response. InACM CCS, 2014. [16] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks.Nature, 2017. [17]Facebook. Opacus.https://github.com/pytorch/ opacus. [18] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchi- cal neural story generation. InACL, 2018. 14 [19]Vitaly Feldman. Does learning require memorization? A short tale about a long tail. InSTOC, 2020. [20]Vitaly Feldman and Chiyuan Zhang. What neural net- works memorize and why: Discovering the long tail via influence estimation. InNeurIPS, 2020. [21] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence informa- tion and basic countermeasures. InACM CCS, 2015. [22]Philip Gage. A new algorithm for data compression.C Users Journal, 12(2):23â38, 1994. [23]Aaron Gokaslan and Vanya Cohen.OpenWeb- Text corpus.http://Skylion007.github.io/ OpenWebTextCorpus, 2019. [24]Shafi Goldwasser, Silvio Micali, and Charles Rackoff. The knowledge complexity of interactive proof systems. SICOMP, 1989. [25]Google. Tensorflow Privacy.https://github.com/ tensorflow/privacy. [26] Alex Graves. Generating sequences with recurrent neu- ral networks.arXiv preprint arXiv:1308.0850, 2013. [27] Peter Henderson, Koustuv Sinha, Nicolas Angelard- Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data- driven dialogue systems. InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123â129, 2018. [28]Sorami Hisamoto, Matt Post, and Kevin Duh. Member- ship inference attacks on sequence-to-sequence models: Is my data in your machine translation system? InTACL, 2020. [29]Andrew Hoang, Antoine Bosselut, Asli Celikyilmaz, and Yejin Choi. Efficient adaptation of pretrained trans- formers for abstractive summarization.arXiv preprint arXiv:1906.00138, 2019. [30]Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020. [31]Jeremy Howard and Sebastian Ruder. Universal lan- guage model fine-tuning for text classification. InACL, 2018. [32]Daphne Ippolito, Daniel Duckworth, Chris Callison- Burch, and Douglas Eck. Automatic detection of gener- ated text is easiest when humans are fooled. InACL. [33]Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? InNeurIPS, 2020. [34]Bargav Jayaraman and David Evans. Evaluating differ- entially private machine learning in practice. InUSENIX Security Symposium, 2019. [35] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scal- ing laws for neural language models.arXiv preprint arXiv:2001.08361, 2020. [36]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denois- ing sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension.arXiv preprint arXiv:1910.13461, 2019. [37]Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective func- tion for neural conversation models. InNAACL, 2016. [38]Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gonzalez. Train large, then compress: Rethinking model size for efficient train- ing and inference of transformers. InICML, 2020. [39] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A ro- bustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019. [40]Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen.Understanding membership inferences on well-generalized learning models.arXiv preprint arXiv:1802.04889, 2018. [41]Jean loup Gailly and Mark Adler. zlib compression library. [42] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. InPsychology of learning and moti- vation. 1989. [43]H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. InICLR, 2018. [44]Tomas Mikolov, Martin KarafiĂĄt, Lukas Burget, Jan Cer- nock ` y, and Sanjeev Khudanpur. Recurrent neural net- work based language model. InInterspeech, 2010. [45]Randall Munroe. Predictive models.https://xkcd. com/2169/, 2019. 15 [46]Milad Nasr, Reza Shokri, and Amir Houmansadr. Ma- chine learning with membership privacy using adversar- ial regularization. InACM SIGSAC, 2018. [47]Milad Nasr, Reza Shokri, and Amir Houmansadr. Com- prehensive privacy analysis of deep learning: Passive and active white-box inference attacks against central- ized and federated learning. InIEEE S&P, 2019. [48]Helen Nissenbaum. Privacy as contextual integrity. Washington Law Review, 2004. [49]OpenAI.Language models are few-shot learners. https://github.com/openai/gpt-3, 2020. [50]Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. InNAACL, 2018. [51]Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebas- tian Riedel. Language models as knowledge bases? In EMNLP, 2019. [52] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. [53]Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever. Better language models and their implications.OpenAI Blog, 2019. [54]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. [55]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. InJMLR, 2020. [56]Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data.arXiv preprint arXiv:2009.10031, 2020. [57]Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.Psychological review, 1990. [58] Jingjing Ren, Ashwin Rao, Martina Lindorfer, Arnaud Legout, and David Choffnes. ReCon: Revealing and con- trolling PII leaks in mobile network traffic. InMobiSys, 2016. [59]Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? InEMNLP, 2020. [60]Benjamin IP Rubinstein, Peter L Bartlett, Ling Huang, and Nina Taft. Learning in a large function space: Privacy-preserving mechanisms for SVM learning.Pri- vacy and Confidentiality, 2012. [61]Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. InACL, 2016. [62]Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. InEMNLP, 2020. [63]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter lan- guage models using model parallelism.arXiv preprint arXiv:1909.08053, 2019. [64]Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. InACM CCS, 2015. [65]Reza Shokri, Marco Stronati, Congzheng Song, and Vi- taly Shmatikov. Membership inference attacks against machine learning models. InIEEE S&P, 2017. [66]Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. InACM CCS, 2020. [67]Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. InKDD, 2018. [68]Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning.arXiv preprint arXiv:2006.07490, 2020. [69] Abhradeep Guha Thakurta, Andrew H. Vyrros, Umesh S. Vaishampayan, Gaurav Kapoor, Julien Freudi- ger, Vivek Rangarajan Sridhar, and Doug Davidson. Learning new words, 2017. US Patent 9,594,741. [70]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ĺukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNIPS, 2017. [71] Kit Walsh. USPTO request for comments on intellectual property protection for artificial intelligence innovation â public comment by the electronic frontier founda- tion.https://w.uspto.gov/sites/default/ files/documents/Electronic%20Frontier% 20Foundation_RFC-84-FR-58141.PDF, 2020. 16 [72]Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyz- ing the connection to overfitting. InIEEE CSF, 2018. [73] Santiago Zanella-BĂŠguelin, Lukas Wutschitz, Shruti Tople, Victor RĂźhle, Andrew Paverd, Olga Ohrimenko, Boris KĂśpf, and Marc Brockschmidt. Analyzing infor- mation leakage of updates to natural language models. InACM CCS, 2020. [74] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. InNeurIPS, 2019. [75]Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization.ICLR, 2017. [76] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DialoGPT: Large-scale generative pre- training for conversational response generation. InACL Demo Track, 2020. A Categorization of Memorized Data Table 5 describes the high-level categories that we assigned to the 604 memorized samples extracted from GPT-2. Note that a single sample can belong to multiple categories. Tables 6 and 7 (omitted for space) show the categorization broken down by attack strategy. B Distribution of Model Perplexities Figure 4 shows the distribution of the perplexities of samples generated with each of our three text generation strategies and ordered based on our six membership inference strategies. C Additional Case Studies of Memorization Here we present additional results from our manual analysis of the memorized content. Memorized Leaked Podesta Emails from WikiLeaks. We identify several memorized URLs that originated from the leaked Podesta Emails available on WikiLeaks 13 . There is only one training document that contains these memorized URLs. Due to the nature of email, the text of one message is often included in subsequent replies to this email. As a result, a URL that is used (intentionally) only once can be included in the dataset tens of times due to the replies. 13 https://en.wikipedia.org/wiki/Podesta_emails Memorized Donald Trump Quotes and Tweets.The GPT-2 training dataset was collected when the 2016 US Pres- idential election was often in the news. As a result, we find several instances of memorized quotes from Donald Trump, both in the form of official remarks made as President (found in the official government records), as well as statements made on Twitter. Memorized Promotional Content. We extract memorized samples of promotional content, such as advertisements for books, beauty products, software products. One of these sam- ples includes a link to an authorâs valid Patreon account, along with a list of named and pseudonymous prior donors. Memorized Number Sequences. We identify many ex- amples where GPT-2 emits common number sequences. Nearly ten examples contain the integers counting up from some specific value. We also find exam- ples of GPT-2 counting the squares1, 2, 4, 8, 16, 25, 36, Fibonacci numbers1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, or digits ofĎ, 3.14159265358979323846264. None of these examples should be unexpected, but the quantity of memorized number sequences was surprising to us. Memorized News Headlines. Numerous memorized text snippets are verbatim copies of news articles and headlines. A large number of these memorized samples are attributed to a single source:thehill.com, an American news website. Interestingly, most of these samples follow the exact same template: (1) they contain a list of different news headlines separated by a âpipeâ symbol (|), (2) the sample begins with twomergedwords, e.g., âTrumpJesuitâ, (3) the headline list ends with the all-caps word âMOREâ, and (4) the sample contains the all-caps word âADVERTISEMENTâ. We indeed find pages on the Web that contain copies of headlines fromthehill.comunder this exact template. The peculiarities of these snippets likely contributed to their mem- orization. For example, the token TrumpJesuit does not appear in any other context on the entire Web. Memorized Base-64 Content.One particularly interesting form of memorization that we identify is the ability of GPT-2 to emit base-64 encoded content. For example, we extract out of the model the following sequence: bWFzdGVyfGltYWdlc3w3OTkxOXxpbWFnZS9wbmd 8aW1hZ2VzL2hkZS9oMDQvODg0NTY3MjYxMTg3MC 5wbmd8ZmFkMTMlNmFiYWJhZjFiMjJlYTAyNzU0Z which decodes to the sequence âmaster|images|79919|image /png|images/hde/h04/8845672611870.png|...â. Despite our at- tempts, we are unable to identify where this content originates. 17 (a) Top-n(2.6% duplicates) (b) Internet (7.1% duplicates) (c) Temperature (0.6% duplicates) Figure 4: For each of our three text generation strategies (Top-n, Internet and Temperature), we generate200,000samples using GPT-2 and apply a de-duplication procedure. The two left-most plots show the distribution of perplexities for the full sample, and the most likely window of 50 tokens. The remaining plots compare the distribution of perplexities of GPT-2 to other measure of sample likelihood: zlib entropy, perplexity under GPT-2 Small and GPT-2 Medium, and perplexity of lower-cased samples. Each plot highlights the 100 samples we selected for manual inspection (red) and the subset that was confirmed as memorized (blue). CategoryCountDescription US and international news 109General news articles or headlines, mostly about US politics Log files and error reports 79Logs produced by software or hardware License, terms of use, copyright notices 54 Software licenses or website terms of use, copyright for code, books, etc. Lists of named items54Ordered lists, typically alphabetically, of games, books, countries, etc. Forum or Wiki entry53User posts on online forums or entries in specific wikis Valid URLs50A URL that resolves to a live page Named individuals46Samples that contain names of real individu- als. We limit this category tonon-news sam- ples. E.g., we do not count names of politi- cians or journalists within news articles Promotional content45Descriptions of products, subscriptions, newsletters, etc. High entropy35Random content with high entropy, e.g., UUIDs Base64 data, etc. CategoryCountDescription Contact info32Physical addresses, email addresses, phone numbers, twitter handles, etc. Code31 Snippetsofsourcecode, including JavaScript Configuration files30 Structured configuration data, mainly for software products Religious texts25Extracts from the Bible, the Quran, etc. Pseudonyms15 Valid usernames that do not appear to be tied to a physical name Donald Trump tweets and quotes 12Quotes and tweets from Donald Trump, of- ten from news articles Web forms11Lists of user menu items, Website instruc- tions, navigation prompts (e.g., âplease enter your email to continueâ) Tech news11News related to technology Lists of numbers10Lists of dates, number sequences,Ď, etc. Sports news9News related to sports Movie synopsis, cast5List of actors, writers, producers. Plot syn- opsis. Pornography5Content of pornographic nature, often lists of adult film actors. Table 5: Descriptions for the categories of memorized text. Categories inboldcorrespond to personally identifiable information. 18 CategoryCount US and international news88 Forum or Wiki entry34 License, terms of use, copyright notice28 Named individuals25 Promotional content18 Lists of named items15 Contact info20 Donald Trump tweets and quotes12 Pseudonyms7 Valid URLs7 Sports news6 Movie synopsis or cast6 (a) Top-n(191 samples) CategoryCount Log files and error reports86 Lists of named items53 Valid URLs40 License, terms of use, copyright notice36 High entropy33 Configuration files32 Code29 Named individuals18 Promotional content14 Contact info12 Pseudonyms11 Forum or Wiki entry9 US and international news7 Tech news7 Pornography5 Web forms5 Lists of numbers5 (b) Internet (273 samples) CategoryCount US and international news31 Religious texts28 License, terms of use, copyright notice24 Promotional content20 Forum or Wiki entry17 Named individuals12 Lists of named items12 Valid URLs12 Tech news8 Contact info8 High entropy6 Lists of numbers6 (c) Temperature (140 samples) Table 6: Memorized content found in samples produced by each of the our three text generation strategies. We show categories with at least 5 samples. CategoryCount License, terms of use, copyright notice11 Lists of named items8 Log files and error reports7 Valid URLs6 Lists of numbers5 (a) Perplexity (51 samples) CategoryCount US and international news21 Lists of named items18 License, terms of use, copyright notice16 Promotional content11 Valid URLs11 Log files and error reports10 Named individuals8 High entropy8 Forum or Wiki entry7 Configuration files6 Code6 (b) Window (119 samples) CategoryCount US and international news40 License, terms of use, copyright notice31 Lists of named items17 Forum or Wiki entry14 Named individuals13 Promotional content13 Contact info12 Log files and error reports11 Valid URLs10 Code10 Tech news6 Configuration files6 Pseudonyms5 (c) zlib (172 samples) CategoryCount US and international news39 Log files and error reports29 Lists of named items17 Forum or Wiki entry12 Named individuals11 License, terms of use, copyright notice10 High entropy9 Configuration files6 Promotional content5 Tech news5 (d) Lowercase (135 samples) CategoryCount Log files and error reports17 Forum or Wiki entry15 Religious texts14 Valid URLs13 High entropy13 Lists of named items12 License, terms of use, copyright notice12 Promotional content11 Configuration files11 Named individuals11 other9 US and international news9 Contact info8 Donald Trump tweets and quotes7 Code6 (e) Small (141 samples) CategoryCount Valid URLs17 Log files and error reports14 US and international news13 Contact info12 Religious texts12 Named individuals11 Promotional content11 High entropy10 Forum or Wiki entry9 Lists of named items8 License, terms of use, copyright notice8 Code5 Donald Trump tweets and quotes5 (f) Medium (116 samples) Table 7: Memorized content found using our six membership inference strategies. We show categories with at least 5 samples. 19