Paper deep dive

TokenSwap: A Lightweight Method to Disrupt Memorized Sequences in LLMs

Parjanya Prajakta Prashant, Kaustubh Ponkshe, Babak Salimi

Year: 2025Venue: NeurIPS 2025 (Spotlight)Area: Model EditingType: EmpiricalEmbeddings: 76

Models: DistilGPT-2, Llama-3.2-3B, Llama-3-8B, Pythia-6.9B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/11/2026, 12:52:52 AM

Summary

TokenSwap is a lightweight, post-hoc inference-time defense mechanism for Large Language Models (LLMs) that mitigates verbatim memorization of training data. By selectively swapping token probabilities of high-frequency grammar tokens with those from a smaller, less-memorizing auxiliary model (e.g., DistilGPT-2), TokenSwap disrupts the chain of conditional probabilities required for verbatim reproduction without requiring access to model weights or training corpora.

Entities (6)

Llama-3-8B · language-model · 100%Pythia-6.9B · language-model · 100%TokenSwap · method · 100%DistilGPT-2 · language-model · 95%CP-Fuse · method · 90%MemFree · method · 90%

Relation Signals (4)

TokenSwap → evaluatedon → Pythia-6.9B

confidence 100% · Evaluations on Pythia-6.9B and Llama-3-8B show up to a 10x drop in exact memorization

TokenSwap → comparedto → MemFree

confidence 95% · We compare against the two inference-time baselines: CP-Fuse... and MemFree

TokenSwap → mitigates → Memorized Generation

confidence 95% · Our method offers a practical, accessible solution for mitigating memorized generation in deployed LLMs.

TokenSwap → uses → DistilGPT-2

confidence 90% · For TokenSwap, we employ DistilGPT-2 (80M) as p_aux.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:As language models scale, their performance improves dramatically across a wide range of tasks, but so does their tendency to memorize and regurgitate parts of their training data verbatim. This tradeoff poses serious legal, ethical, and safety concerns, especially in real-world deployments. Existing mitigation techniques, such as differential privacy or model unlearning, often require retraining or access to internal weights making them impractical for most users. In this work, we introduce TokenSwap, a lightweight, post-hoc defense designed for realistic settings where the user can only access token-level outputs. Our key insight is that while large models are necessary for high task performance, small models (e.g., DistilGPT-2) are often sufficient to assign fluent, grammatically plausible probabilities to common function words - and crucially, they memorize far less. By selectively swapping token probabilities between models, TokenSwap preserves the capabilities of large models while reducing their propensity for verbatim reproduction. Evaluations on Pythia-6.9B and Llama-3-8B show up to a 10$\times$ drop in exact memorization with negligible task degradation. Our method offers a practical, accessible solution for mitigating memorized generation in deployed LLMs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

75,766 characters extracted from source content.

Expand or collapse full text

TOKENSWAP: A Lightweight Method to Disrupt Memorized Sequences in LLMs Parjanya Prajakta Prashant ∗ UC San Diego Kaustubh Ponkshe ∗ MBZUAI Babak Salimi UC San Diego Abstract As language models scale, their performance improves dramatically across a wide range of tasks, but so does their tendency to memorize and regurgitate parts of their training data verbatim. This tradeoff poses serious legal, ethical, and safety concerns, especially in real-world deployments. Existing mitigation techniques, such as differential privacy or model unlearning, often require retraining or access to internal weights making them impractical for most users. In this work, we introduceTOKENSWAP, a lightweight, post-hoc defense designed for realistic settings where the user can only access token-level outputs. Our key insight is that while large models are necessary for high task performance, small models (e.g., DistilGPT-2) are often sufficient to assign fluent, grammatically plausible probabilities to common function words - and crucially, they memorize far less. By selectively swapping token probabilities between models,TOKENSWAPpreserves the capabilities of large models while reducing their propensity for verbatim repro- duction. Evaluations on Pythia-6.9B and Llama-3-8B show up to a 10×drop in exact memorization with negligible task degradation. Our method offers a practical, accessible solution for mitigating memorized generation in deployed LLMs. 1 Introduction Large language models (LLMs) such asGPT–4, GEMINI, andLLAMAhave demonstrated strong performance across a wide range of tasks, from natural language understanding to complex rea- soning [Achiam et al., 2023, Team et al., 2023, Dubey et al., 2024]. These capabilities are driven by their massive parameter counts and extensive training corpora, enabling human-level fluency and impressive reasoning across domains. Often referred to as emergent properties, such abilities arise directly from scale, with well-established scaling laws predicting performance gains. However, increased scale also introduces a critical drawback: the tendency of LLMs to memorize and reproduce parts of their training data. [Carlini et al., 2021, 2022, Biderman et al., 2024, Nasr et al., 2025] One of the most pressing consequences of memorization is the verbatim or near-verbatim generation of training data [Karamolegkou et al., 2023, Tirumala et al., 2022, Allen-Zhu and Li, 2025]. Although memorization is an inherent property and not necessarily harmful, its consequence of verbatim generation leads to plagiarism and copyright violation. This behavior poses serious risks to both model providers and end-users. Providers may face legal challenges, including copyright infringe- ment lawsuits [Karamolegkou et al., 2023, Grynbaum and Mac, 2023, Panwar, 2025], while users unknowingly risk legal liability by reproducing protected content. Crucially, the threat is not limited to exact substring matches: even approximate or near-verbatim outputs can constitute infringement, as evidenced by lawsuits like theNew York Timescase against OpenAI for near-verbatim content generation [Freeman et al., 2024]. ∗ Equal Contribution; Correspondence to Parjanya Prajakta Prashant <pprashant@ucsd.edu>, Kaustubh Ponkshe <ponkshekaustubh11@gmail.com> Preprint. Under review. arXiv:2502.05159v2 [cs.LG] 27 May 2025 Figure 1: Overview ofTOKENSWAP. Our approach replaces token probabilities of high-frequency "grammar-based" tokens with those from a small auxiliary language model. This mitigates memorized generation while maintaining fluency and model performance. The top path shows standard LLM generation, while the bottom path demonstrates howTOKENSWAPalters token selection to disrupt memorization and produce novel text. We consider the perspective of a typical user of commerical LLMs such as GPT-4 [Achiam et al., 2023], Gemini [Team et al., 2023], Llama3 [Dubey et al., 2024], and Deepseek [Liu et al., 2024]. These models do not share their training data and many do not make their weights publicly available. Even in cases where weights are openly shared, hosting a production-grade LLM requires substantial memory resources, rendering it impractical for the average user. Consequently, it is reasonable to assume that most users can only interact with these models through APIs hosted on external servers, with access limited to model outputs such as token-level logits. Despite these practical constraints, to our knowledge, none of the existing methods, whether designed to prevent memorization or mitigate verbatim output, can effectively operate under such limited access conditions. Existing methods require access to training data and/or model weightsApproaches to address memorization are broadly categorized into pre-training and post-training interventions. Pre-training methods include deduplication [Kandpal et al., 2022], differential privacy (DP) [Abadi et al., 2016], and selective token exclusion during training [Hans et al., 2024]. While these approaches can reduce memorization, they often incur substantial computational costs and degrade model performance [Anil et al., 2021]. Post-training interventions focus on unlearning techniques that attempt to modify specific neurons and weights or utilize finetuning methods to prevent models from generating memorized content [Maini et al., 2023, Sakarvadia et al., 2024, Chen et al., 2025, Russinovich and Salem, 2025]. However, these methods remain susceptible to training data extraction [Shumailov et al., 2024], often impair general model capabilities [Huang et al., 2024], and can lead to unintended forgetting of critical aspects such as safety guardrails [Wang et al., 2025]. This challenge is further complicated by theoretical findings suggesting that some degree of memorization may be inherent to achieving generalization in learning algorithms [Attias et al., 2024]. In contrast, another line of work focuses on preventing the generation of memorized content at inference time without modifying model weights. These approaches include blocking exact matches to training data [Ippolito et al., 2022] or combining logits from multiple models trained on disjoint datasets [Abad et al., 2024]. However, these methods too require access to training data or mul- tiple LLMs trained on strictly disjoint datasets. Table 1 summarizes the various approaches and assumptions under which they operate (see Appendix A for a comprehensive review). Memorization scales with sizeThe propensity to reproduce training data consistently increases with the size of the language model [Carlini et al., 2022, Biderman et al., 2024]. Since model performance generally scales positively with size, users are forced into a trade-off between obtaining high performance and mitigating memorized generations. Figure 2 demonstrates this relationship using a series of Pythia models, showing the trade-off between memorization and cross-entropy loss. 2 2.533.544.5 0 20 40 60 2.533.544.5 0 20 40 60 6.9B 2.8B 1.4B 1B 410M 160M 70M 6.9B 2.8B 1.4B 1B 410M 160M Cross Entropy Loss (CE Loss)↓ Exact Match Rate (%) (EMR) ↓ Standard TOKENSWAP Figure 2:Memorization (EMR) vs Performance (CE Loss) across different model sizes. Larger, more capa- ble models exhibit higher memorization.TOKENSWAP, with Pythia-70M as the auxiliary model, achieves low memorization rates while maintaining competitive per- formance. Details in Section 4.2 and Section 5. In this work, we presentTOKENSWAP, an inference-time method that significantly allevi- ates this tradeoff by combining large model per- formance with small model memorization (Fig- ure 2).TOKENSWAPselectively replaces the probabilities of a subset of common grammar to- kens (e.g., “the”, “of”, “and”) of the large main model with those from a small auxiliary model. This technique disrupts the verbatim generation by breaking the high-probability paths that lead to verbatim reproduction. This disruption has a cascading effect: once one token deviates from the memorized sequence, all subsequent predic- tions are conditioned on this altered context, fur- ther preventing reproduction. Importantly, since small models reliably approximate probabilities for common grammatical tokens,TOKENSWAP preserves the large model’s performance. For auxiliary models of size much smaller than the main model, this provides a verbatim memorization mitigation method which requires accessneither to the training data nor the model weights. Since we treat the effect of memorization, and not the cause itself, our method is able to reduce verbatim generation at inference time. We extensively evaluateTOKENSWAPthrough both controlled experiments and real-world deploy- ments. In controlled fine-tuning experiments (Section 4.1),TOKENSWAPachieves a 50-800× reduction in verbatim generation compared to undefended models. Evaluations on commercial-grade models such as Pythia-6.9b and Llama-3-8b (Section 4.2) demonstrate reductions in verbatim genera- tion by upto 10×, without compromising downstream task performance. Furthermore, comparisons with Goldfish [Hans et al., 2024] show thatTOKENSWAPmatches or surpasses the effectiveness of state-of-the-art pre-training methods (Section 4.3). Table 1: Comparison ofTOKENSWAPwith existing methods based on their assumptions.TO- KENSWAPuniquely avoids requiring access to model weights or the copyrighted training corpus. While it employs an auxiliary model, the memory overhead is small≈1%due to the small size of the auxiliary model. PT: pre-training, UL: unlearning, FT: fine-tuning, Inf: inference-time. Model AccessCopyrighted Corpus AccessInference Overhead Deduplication [36] PTweights✓✗ Goldfish [30] PTweights✓✗ Balanced subnet [53] ULweights✓✗ Obliviate [51] FTweights✓✗ MemFree [34] Inf logits✓querying entire dataset CP-Fuse [1] Inflogits✓twice of standard generation TOKENSWAPInf logits✗small auxiliary model 2 Preliminaries 2.1 Language Models: Notation and Setup We consider auto-regressive language models that model the log-probability of a token conditioned on all previous tokens in a sequence. They operate over a vocabularyV=v 1 ,...,v |V| of typically |V|≈10 5 −10 6 tokens. Given an input prompt(x −l p ,...,x −1 )∈V l p of lengthl p followed by a response sequence(x 0 ,...,x l−1 )∈V l of lengthl, an auto-regressive language model parametrizes the joint probability by factorizing over conditional probabilities: p(x 0 ,...,x l |x −l p ,...,x −1 ) = l Y i=0 p(x i |x <i ),(1) 3 For each positioni, the model outputs a distributionp i [v]overV, wherep i [v] =p(x i =v|x <i ). Since language models are trained to maximize the likelihood of observed sequences, they tend to assign high probabilities to tokens that frequently follow specific prefixes during training. This increases the risk of memorization andverbatim reproductionof training data. 2.2 Extractable Memorization Memorization in language models can manifest in various ways, but a practically relevant and widely adopted framework isextractable memorization[Carlini et al., 2021, 2022]. Carlini et al. [2021] demonstrate that models can be induced to regurgitate training sequences when prompted with prefixes from their training data. The following definition formalizes this concept: Definition 1(Extractable Memorization).A sequencex= (x 0 ,...,x l−1 )of lengthlis con- sideredextractable withl p tokens of contextfrom a language modelpif there exists a prefix x − = (x −l p ,...,x −1 )of lengthl p such that[x − ∥x]appears in the training data ofp, andp reproducesxvia greedy decoding. Formally, for eachi∈0,...,l−1: x i = arg max x ′ ∈V p x ′ |x <i ,x − . This definition is practically useful because: (1) it aligns with real-world risks of copyright and memorized generation [Nasr et al., Karamolegkou et al., 2023], (2) it provides a concrete, testable condition that can be evaluated on real models, and (3) it extends to models of different sizes, capturing the well-documented trend that larger models memorize more data [Carlini et al., 2022, Biderman et al., 2024]. This scaling behavior is important in motivating our methodology in Section 3. 3 Methodology As discussed earlier, small language models (e.g., DistilGPT-2, Pythia-70M) have lower propensity to reproduce training data compared to large models (e.g., Llama3, GPT-4). We introduceTOKENSWAP, a lightweight, post-hoc method that combines the strengths of both model scales: large-model perfor- mance with small-model memorization. During inference,TOKENSWAPreplaces the probabilities for selected tokens of a large model with those a small model. AlgorithmLetp main andp aux denote the probability distributions of the main and auxiliary models respectively, wherep main (x t |x <t )andp aux (x t |x <t )represent their token probabilities conditioned on previous tokens. We assume the parameter count of the main model significantly exceeds that of the auxiliary model. Given these models,TOKENSWAPselectively replaces probabilities for a fixed subset of tokensG ⊂V. The complete procedure is formalized in Algorithm 1. At each positioni, TOKENSWAPqueries bothp main andp aux to obtain probability distributions conditioned on the current contextx <i . For tokens in subsetG ⊂ V, probabilities from the main model are replaced with scaled probabilities from the auxiliary model, with scaling factorαensuring the final distributionp final remains a valid distribution. This prevents reproduction of memorized sequences: if any tokenx i in a memorized sequence belongs toG, its probability underp final is determined by the auxiliary model. Since the auxiliary model memorizes less, this disrupts the chain of conditional probabilities required for verbatim generation of most sequences. Importantly, for tokensv /∈G, their probabilities remain unchanged, i.e.,p final i [v] =p main i [v]. SelectingGfor Effective Memorization DisruptionThe choice ofGaffects both memorization and model performance. By modifying token probabilities,TOKENSWAPdisrupts memorized sequences while preserving fluency. However, not all tokens are equally effective for this purpose.G should consist of tokens that frequently appear in memorized text, as replacing their probabilities reduces the likelihood of exact reproduction. At the same time, modifying inappropriate tokens can degrade model performance, especially for specialized tasks. For instance, ifGincludes numeric tokens, mathematical reasoning may degrade. Therefore,Gshould satisfy two key criteria. First, it must contain frequently occurring tokens. Second, it should avoid tokens where probability replacement impacts the model’s capabilities. 4 Algorithm 1TOKENSWAP Require:Main modelp main , auxiliary modelp aux , token subsetG, promptx <0 1:fori= 0,1,...do 2:p main i ←p main (·|x <i )Get main model probabilities 3:p aux i ←p aux (·|x <i )Get auxiliary model probabilities 4:α← P v∈G p main i [v] P v∈G p aux i [v] Compute normalization 5:forv∈Vdo 6:p final i [v]← p main i [v],ifv /∈G α·p aux i [v],ifv∈G 7:end for 8:x i ∼p final i Sample next token 9:end for Empirical studies suggest that small models correctly approximate the probabilities of high-frequency function words while diverging more on rare or domain-specific terms [Pinto et al., 2024a]. Addition- ally, small language models (≈100M) can generate coherent and grammatically correct text [Eldan and Russinovich, 2023]. Based on these insights, we constructGfrom grammar-based high-frequency tokens (e.g. - ’the’, ’in’). Further, sinceGconsists of high-frequency words, there exists a natural one-to-one mapping between tokens even whenp main andp aux use different tokenizers and vocab- ularies. While this approach is well-suited for natural language, structured domains such as code may require domain-specific adaptations. Additional details on the construction ofGare provided in Appendix C.2 and C.3. 4 Experiments In this section, we demonstrate the effectiveness ofTOKENSWAP, in both controlled and real-world settings. Our experiments evaluate TOKENSWAPalong two dimensions: • The method’s efficacy in preventing exactand approximatereproduction of training data. • The impact on model performance across common-sense reasoning, language and fluency. We evaluateTOKENSWAPacross three settings to demonstrate its effectiveness. In Section 4.1, we deliberately induce memorization through extensive fine-tuning on small datasets to stress-test our defense. Section 4.2 evaluatesTOKENSWAPon production-grade models including Pythia-6.9B and Llama-3-8B. Finally, in Section 4.3, we compare against Goldfish [Hans et al., 2024], a pre-training method specifically designed to reduce memorization, showing that our post-hoc approach achieves comparable results without requiring model retraining. 4.1 Extreme Memorization In order to rigorously evaluateTOKENSWAP, we create an extreme test case by deliberately inducing memorization through extensive fine-tuning. WhileTOKENSWAPcan be applied to real-world models directly, our baselines require specific experimental conditions for comparison. Similar extreme test cases have been generated to evaluate memorization in prior work [Hans et al., 2024, Abad et al., 2024]. Following Abad et al. [2024], we fine-tune a Llama-3.2-3B model [Dubey et al., 2024] on 2,000-sequence subsets from two datasets: MathAbstracts [Zhang et al., 2024] and WritingStories [Fan et al., 2018]. We train for 50 epochs to deliberately amplify memorization beyond typical levels. Memorization MetricsOur analysis employs both exact and approximate memorization and per- formance metrics to ensure a comprehensive assessment. Exact memorization is measured through Matching Length (ML), which the number of verbatim characters or tokens generated before first deviation, andExact Matching Rate (EMR), which computes the fraction of sequences reproduced verbatim. To capture partial memorization, we use theROUGE-Lscore, which identifies the longest common non-contiguous subsequence and gives a score between 0 and 1, and theNormalized Leven- shtein Distance, which quantifies the minimum number of edits needed to transform generated text 5 into the original sequence. Lower scores indicate reduced memorization for Matching Length, Exact Matching Rate, and ROUGE-L. Higher scores are better for Normalized Levenshtein Distance. These metrics are widely used to evaluate verbatim and approximately verbatim generation [Karamolegkou et al., 2023, Hans et al., 2024, Abad et al., 2024]. Performance MetricsSince our setup intentionally induces extreme memorization, standard performance metrics are not meaningful. Nonetheless, we report cross-entropy loss on a held-out validation set in Appendix B.1. Setup and Inference-time BaselinesWe compare against the two inference-time baselines: CP- Fuse [Abad et al., 2024], which samples from weighted combinations of models trained on disjoint datasets, and MemFree [Ippolito et al., 2022], which blocks exactn-gram matches to the training data. Standard refers to greedy decoding without any memorization mitigation. Both baselines rely on unrealistic assumptions—MemFree requires access to the training data, while CP-Fuse assumes access to two separately trained models on disjoint corpora. To assess CP-Fuse under more realistic conditions, we evaluate two variants:CP-FUSEHALF, with perfectly disjoint sets of 1,000 sequences each, andCP-FUSEMIXTURE, with 1,500 sequences per model and 500 overlapping. ForTOKENSWAP, we employ DistilGPT-2 (80M) [Sanh et al., 2019] asp aux . We constructGwith |G|= 110tokens using high-frequency ’grammar-based’ words. Additional details onGare provided in Appendix C.2. For all experiments and methods, a prefix of 20 tokens is used and the next 128 tokens are greedily sampled (temperature = 0.0). Table 2:Memorization for WritingPrompts and MathAbstracts datasets. ML: Matching Length, EMR: Exact Match Rate, Lev.: Normalized Levenshtein Distance WritingPromptsMathAbstracts MethodML↓ROUGE-L↓Lev.↑EMR↓ML↓ROUGE-L↓Lev.↑EMR↓ Standard464.00.890.1083.4450.40.980.0393.6 MemFree17.40.290.630.06.70.440.550.0 CP-Fuse-mix 280.30.580.3749.2233.70.620.3647.1 CP-Fuse-half12.50.170.730.015.30.260.710.1 TOKENSWAP19.70.190.710.153.00.380.601.8 ResultsTable 2 demonstrates TOKENSWAP’s effectiveness in reducing memorization across both datasets. For WritingPrompts,TOKENSWAPreduces EMR by 800x (from 83.4% to 0.1%) and ROUGE-L by 4.6x (from 0.89 to 0.19) compared to standard generation. On MathAbstracts, EMR decreases by 50x (from 93.6% to 1.8%) and ROUGE-L by 2.6x (from 0.98 to 0.38). CP-Fuse-half achieves slightly better results but requires disjoint training sets, while CP-Fuse-mix performs signifi- cantly worse due to dataset overlap. MemFree achieves the lowest scores on the exact memorization metrics (Exact Matching Rate and Matching Length) but performs poorly on approximate memo- rization metrics (ROUGE-L and Levenshtein). This shows that, while MemFree prevents verbatim generation, it still allows high levels of near-verbatim generation. The performance gap between WritingPrompts (EMR: 0.1%) and MathAbstracts (EMR: 1.8%) aligns with our intuition -Gwas designed focusing on natural language tasks. Nevertheless,TOKENSWAPachieves substantial memo- rization reduction for both domains. To complement our quantitative results,we provide qualitative examples of generations from the WritingPrompts datasetin Figure 3 and Appendix E. 4.2 Memorization in the wild In this section, we demonstrate the efficacy of our approach on production-grade models. We assess the effectiveness ofTOKENSWAPon two pre-trained models: Pythia-6.9B [Biderman et al., 2023] and Llama-3-8B [Dubey et al., 2024]. Pile-Memorized DatasetFor Pythia-6.9B, we evaluate on memorized sequences identified by Chang et al. [2024a] from the Pile dataset, consisting of 32-token prefixes and 48-token suffixes. After filtering to retain only natural language content (excluding code, URLs, etc.), we obtain 184 evaluation examples. 6 Prefix:Magic – once a real and potent force but as the world population Original Suffix grew from millions to bil- lions the shared mana per person is now negligible. A group of astronauts help- lessly watching the Earth perish experience... Standard Generation grew from millions to bil- lions the shared mana per person is now negligible. A group of astronauts help- lessly watching the Earth perish experience... TOKENSWAPGeneration grew and the number of wiz- ards and witches declined, the world began to suffer. Now the world suffers from a lack of magic, and the gov- ernment is tasked with... Figure 3: Comparison of text generation methods. Red text indicates memorized content. Standard generation reproduces the entire suffix verbatim, while TOKENSWAPgenerates novel content. LeetCode DatasetFor Llama-3-8B, following previous work demonstrating LeetCode prob- lem memorization [Karamolegkou et al., 2023], we evaluate on 1,825 LeetCode problem state- ments [gzipChrist, 2021].These problem statements are written in natural language. Since the exact format of LeetCode problems in Llama’s training data is unknown, we remove punctuation while calculating the memorization metrics. Additionally, instead of exact match rate, we use ROUGE-L >0.8as our threshold for identifying memorized content. Prefix length of 20 tokens is used and the next 100 tokens are sampled. Evaluation SetupWe face two key limitations when comparingTOKENSWAPwith existing baselines. CP-Fuse requires models trained on disjoint datasets, but verifying this is difficult since most LLMs do not release training data. Even when available, disjoint datasets are unlikely given that most models train on overlapping web corpora like Common Crawl. Additionally, CP-Fuse requires identical tokenizers, limiting comparisons to models within the same family. Similarly, we cannot evaluate against MemFree due to unavailable training data (LLaMA) or prohibitively large datasets (Pythia uses the 800GB Pile [Biderman et al., 2023]). To ensure fair evaluation for CP-Fuse, we paired each model with a smaller counterpart: Pythia-2.8B with Pythia-6.9B, and Llama-3.2-3B with Llama-3-8B. Using smaller models actually favors CP-Fuse since they memorize less. We avoid very small models (<100M) as CP-Fuse needs roughly equally capable models (see Appendix C.1.4). The setup forTOKENSWAPfollows Section 4.1. For LeetCode evaluation, we use both DistilGPT-2 and SmolLM-135M [Allal et al., 2025] as auxiliary models. SmolLM is an instruction-tuned model, which enables evaluation on instruction-following tasks like MT-Bench where an instruct-capable auxiliary model is required. For memorization, we use the same metrics as Section 4.1. Performance MetricsWe evaluate two key aspects: task performance and generation quality. For task performance, we assess five-shot learning on multiple commonsense reasoning benchmarks: BoolQ [Clark et al., 2019], SIQA [Sap et al., 2019], PIQA [Bisk et al., 2020], ARC-Challenge [Clark et al., 2018], ARC-Easy [Clark et al., 2018], OBQA [Mihaylov et al., 2018], and WinoGrande [Sakaguchi et al., 2021]. For generation quality, we report cross-entropy loss on samples from Slimpajama[Soboleva et al., 2023], which correlates with fluency [Basu et al., 2020] and has been used to evaluate prior memorization mitigation work [Hans et al., 2024, Abad et al., 2024]. We also evaluate on MT-Bench [Zheng et al., 2023], which tests multi-turn conversation, instruction- following, and generation quality through realistic conversational scenarios. Note that MT-Bench and commonsense reasoning results are only reported for Llama-3-8B (LeetCode Dataset) since these require instruction-following capabilities not available in the base Pythia models. ResultsTable 3 demonstrates thatTOKENSWAPsubstantially reduces memorization across both datasets compared to standard generation and CP-Fuse. Exact match rate decreases by over 10x compared to standard generation and 5-7x compared to CP-Fuse on both datasets. The average matching length shows similar improvements, reducing by 4-5x versus standard and 3-4x versus CP-Fuse. The consistent improvements in approximate memorization metrics (ROUGE-L and Levenshtein distance) demonstrate thatTOKENSWAProbustly prevents verbatim generation rather than simply introducing small perturbations. CP-Fuse shows limited effectiveness in these real-world 7 Table 3:Memorization metrics on LeetCode and Pile-Memorized datasets: ML: Matching Length, EMR: Exact Match Rate, Lev.: Normalized Levenshtein Distance & Performance metrics: Cross Entropy Loss (CE Loss) on SlimPajama; MT-Bench with GPT-4 as a judge, Mean of scores on Commonsense Reasoning benchmarks. LeetCode Dataset (Llama) MethodML↓ROUGE-L↓Lev.↑ROUGE-L>0.8↓CE Loss↓MT-Bench↑Commonsense↑ Standard24.570.390.609.652.387.7571.87 CP-Fuse19.440.370.617.012.458.5370.18 TOKENSWAP 1 6.040.270.710.962.52-71.87 TOKENSWAP 2 8.580.300.691.922.437.78- Pile-Memorized Dataset (Pythia) MethodML↓ROUGE-L↓Lev.↑EMR↓CE Loss↓ Standard151.60.800.1865.222.80 CP-Fuse 97.050.620.3529.352.81 TOKENSWAP 1 35.100.380.565.982.88 1 DistilGPT-2 as auxiliary model. 2 SmolLM-135M as auxiliary model. scenarios primarily because its core assumption of disjoint training datasets does not hold. Even when using different models, the inherent overlap in web-scale training corpora prevents CP-Fuse from effectively disrupting memorized sequences. TOKENSWAPmaintains task performance by selectively targeting only grammar-based tokens, leaving reasoning-critical content words unchanged. This preserves commonsense reasoning abilities, as shown by identical accuracy scores (71.87%) compared to standard generation. The method also maintains fluency, evidenced by minimal cross-entropy increases and nearly equal MT-Bench scores. While CP-Fuse achieves better conversational performance (8.53 vs 7.78), it fails to verbatim generation, making it unsuitable for the desired goal. 4.3 Comparison with Pre-training Methods While previous sections demonstrate thatTOKENSWAPoutperforms post-hoc baselines, we also compare with Goldfish [Hans et al., 2024], a pre-training approach that reduces memorization by excluding a fraction1/kof tokens from loss computation during training. Since pre-training large models using this loss is expensive, we evaluate on pre-trained goldfish models from Hans et al. [2024]. These models were trained on a subset of RedPajama [Weber et al., 2024] combined with 2000 Wikipedia [Bridge, 2001] sequences. To induce memorization, the Wikipedia sequences were duplicated 50 times during training. We compare against models trained withk∈ 3,4,32. For TOKENSWAP, we maintain the same experimental setup from Section 4.1. Following Hans et al. [2024], we use identical prefix and suffix lengths for extraction of memorized sequences. Table 4:Comparison with Goldfish [Hans et al., 2024] for k∈3,4,32. MethodML↓ROUGE-L↓Levenshtein↑EMR↓CE Loss↓ Standard73.90.380.587.83.44 Goldfish (k=3)12.70.230.720.03.54 Goldfish (k=4) 14.70.230.710.03.50 Goldfish (k=32)58.10.350.602.53.44 TOKENSWAP12.40.220.720.13.44 TOKENSWAP+ Goldfish (k=3) 7.90.210.730.03.57 ResultsTable 4 showsTOKENSWAPachieves comparable or superior performance to Goldfish across all memorization metrics. Notably,TOKENSWAPobtains the best Matching Length, Rouge- L and Normalized Levenshtein distance scores while maintaining better cross-entropy than the Goldfish variants fork= 3,4. The effectiveness of Goldfish varies with parameterk- smaller values (more aggressive token exclusion) yield stronger memorization reduction but worse performance, as evidenced by higher cross-entropy. This illustrates a key advantage ofTOKENSWAP: we achieve 8 similar memorization reduction without requiring modified training or reduced training data tokens. Figure 4 (Appendix B.5) further supports this finding, showing nearly identical ROUGE-L score distributions betweenTOKENSWAPand Goldfish (k=3), indicating that our post-hoc approach matches the most aggressive pre-training variant. Furthermore, applyingTOKENSWAPto Goldfish (k= 3) as the main model reduces memorization more than either method alone, demonstrating that our approach is orthogonal to pre-training methods and can enhance existing techniques. 5 Discussion and Limitations In this section, we analyzeTOKENSWAP’s behavior across different settings. We first perform ablations on the auxiliary model choice and the size ofG. We then analyzeTOKENSWAPacross the Pythia model family to demonstrate significant improvements in the performance-memorization tradeoff. Finally, we discuss limitations and potential extensions of our method. Choice of the Auxiliary ModelIn Section 4 we testTOKENSWAPwith DistilGPT-2 as the auxiliary model. A natural question arises:What auxiliary model should one choose and how does the size of the auxiliary model affect memorized generation?To answer this, we use the SmolLM family [Allal et al., 2025] with three sizes (135M, 360M, 1.7B) and evaluate on both Pythia-6.9B (Pile-memorized dataset) and Llama-3-8B (LeetCode dataset). Detailed results are in Table 8 (Appendix B.4). We observe a clear trend: smaller auxiliary models lead to less verbatim generation, confirming our hypothesis thatTOKENSWAP’s effectiveness stems from low memorization in auxiliary models. Importantly, auxiliary model size has minimal impact on performance. MT-Bench scores show negligible variation across auxiliary models—this is particularly significant since MT-Bench evaluates overall sequence generation quality, unlike cross-entropy loss which measures token-level accuracy. Therefore, any small model (≈100M) which can generate fluent text and predict grammar-based tokens well, such as DistilGPT-2 or SmolLM-135M, can be used effectively as an auxiliary model. Ablations onGThe subset of tokensGis constructed by selecting grammar-based words from the top 500 most frequent English words, resulting in|G|= 110(details in Appendix C.2). To understand the impact ofGsize on memorization reduction, we ablate by constructingGfrom the topkmost frequent words fork∈10,50,100,500,2500, yielding|G|∈9,43,66,110,136. We evaluate on the Pile-memorized task using Pythia-6.9B as the main model and Pythia-70M as auxiliary (see Appendix B.3 for complete results). We observe a clear trend: as|G|increases, memorization decreases significantly. For example, EMR drops from 22.28% (|G|= 9) to 8.15% (|G|= 136). This makes intuitive sense—largerGenables the auxiliary model to disrupt memorized sequences more frequently. The cross-entropy loss remains largely stable, indicating minimal performance degradation with increase in|G|. Performance-Memorization TradeoffWe analyze howTOKENSWAPaffects the tradeoff between model performance and memorization across seven Pythia models (70M to 6.9B parameters), using Pythia-70M as the auxiliary model. Figure 2 shows exact match rate (EMR) versus cross-entropy loss—lower values are better for both metrics. Standard generation faces a severe tradeoff: reducing memorization from 45% to 6% EMR costs 0.7 points in cross-entropy (2.85→3.55).TOKENSWAP considerably improves this tradeoff. At similar performance levels (cross-entropy≈2.87),TO- KENSWAPachieves 8.7% EMR versus 45% for standard models—an 8× memorization reduction. Even when targeting very low memorization (6% EMR),TOKENSWAPmaintains cross-entropy at 3.07, significantly outperforming standard models at equivalent memorization levels. Limitations and Future workOne limitation of our work is that in the rare cases where the small auxiliary model memorizes a sequence, our approach will preserve that memorization. However, in practice, small auxiliary models (≈100M parameters) memorize very little, and we empirically match or outperform existing baselines without requiring access to training data or restrictive assump- tions like disjoint datasets. Additionally, while pre-training or unlearning mitigation methods are impractical for large models, they can be applied to small models since these are often open-source with accessible training data. Therefore, we expect future development in small models with low memorization. This makes our work even more significant:any advance in pre-training or unlearning methods to reduce memorization in small models can be immediately extended to large models using 9 TOKENSWAP. Second, our current implementation focuses on natural language tasks. A promising direction for future work is extending TOKENSWAPto other domains such as code generation. 6 Conclusion TOKENSWAPoffers several key advantages for mitigating memorized generation in language models: it operates without requiring access to model weights or training data, and makes no assumptions about the underlying training distribution. Our experiments demonstrate 10-800× reductions in verbatim generation, matching or exceeding baselines that assume access to training data, disjoint models, or require pre-training their own models. Importantly, this comes at minimal cost to model performance.TOKENSWAPmaintains performance on commonsense reasoning tasks, and our MT- Bench evaluation shows that it preserves fluency, instruction-following, and conversational abilities. This makes TOKENSWAPa practical solution for both providers and users of LLMs. 10 References Javier Abad, Konstantin Donhauser, Francesco Pinto, and Fanny Yang. Copyright-protected language generation via adaptive model fusion.arXiv preprint arXiv:2412.06619, 2024. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlí ˇ cek, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big–data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025. Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. InInternational Conference on Learning Representations (ICLR), 2025. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private bert.arXiv preprint arXiv:2108.01624, 2021. Idan Attias, Gintare Karolina Dziugaite, Mahdi Haghifam, Roi Livni, and Daniel M Roy. Information complexity of stochastic convex optimization: Applications to generalization and memorization. arXiv preprint arXiv:2402.09327, 2024. Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity.arXiv preprint arXiv:2007.14966, 2020. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URLhttps://arxiv.org/abs/2304.01373. Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivan- shu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2024. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. Astoria-Megler Bridge. Wikipedia, the free encyclopedia.San Francisco (CA): Wikimedia Foundation, 2001. Alexandru B ̆ arbulescu and Evangelos Triantafillou. Targeted memorized-data unlearning for large language models. InProceedings of the 41st International Conference on Machine Learning (ICML), 2024. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models.arXiv preprint arXiv:2202.07646, 2022. Ting-Yun Chang, Jesse Thomason, and Robin Jia. Do localization methods actually localize mem- orized data in llms? a tale of two benchmarks. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3190–3211, 2024a. 11 Wenwen Chang, Tal Schuster, and Yann LeCun. Neural surgery for memorisation: Locating and removing verbatim recall neurons. InNeurIPS, 2024b. Shaohan Chen, William Raine, and James Bradbury. Accelerating large language model decoding with speculative sampling. InarXiv preprint arXiv:2308.08066, 2023. Tong Chen, Faeze Brahman, Jiacheng Liu, Niloofar Mireshghallah, Weijia Shi, Pang Wei Koh, Luke Zettlemoyer, and Hannaneh Hajishirzi. Parapo: Aligning language models to reduce verbatim reproduction of pre-training data.arXiv preprint arXiv:2504.14452, 2025. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Mark Davies. The corpus of contemporary american english as the first reliable monitor corpus of english.Literary and linguistic computing, 25(4):447–464, 2010. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms.arXiv preprint arXiv:2310.02238, 2023. Niva Elkin-Koren, Uri Hacohen, Roi Livni, and Shay Moran. Can copyright be reduced to privacy? arXiv preprint arXiv:2305.14822, 2023. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation.arXiv preprint arXiv:1805.04833, 2018. Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, and Maksym Andriushchenko. Exploring memorization and copyright violation in frontier llms: A study of the new york times v. openai 2023 lawsuit.arXiv preprint arXiv:2412.06370, 2024. Michael M Grynbaum and Ryan Mac. The times sues openai and microsoft over ai use of copyrighted work.The New York Times, 27, 2023. gzipChrist.Leetcode problem dataset, 2021.URLhttps://w.kaggle.com/datasets/ gzipchrist/leetcode-problem-dataset. Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchenbauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, et al. Be like a goldfish, don’t memorize! mitigating memorization in generative llms.arXiv preprint arXiv:2406.10209, 2024. Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. Sok: Memorization in general-purpose large language models, 2023. URLhttps://arxiv. org/abs/2310.18362. Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models, 2023. URLhttps://arxiv.org/abs/2304.01933. Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models, 2024. URLhttps://arxiv.org/abs/2407.17817. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy.arXiv preprint arXiv:2210.17546, 2022. 12 Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models.arXiv preprint arXiv:2210.01504, 2022. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. InInternational Conference on Machine Learning, pages 10697–10711. PMLR, 2022. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models.arXiv preprint arXiv:2310.13771, 2023. Soo Min Kim, Jason Wei, and Denny Zhou. Big-little decoder: Faster language generation with an auxiliary model. InNeurIPS, 2023. Yair Leviathan, Clemens Rosenbaum, and Slav Petrov. Fast inference from transformers via specula- tive decoding. InICML, 2023. Junnan Li, Dongxu Li, and Caiming Xiong. Contrastive decoding: Open-ended text generation as conditional density estimation. InNeurIPS, 2022. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. Edward Loper and Steven Bird. Nltk: The natural language toolkit.arXiv preprint cs/0205028, 2002. Pratyush Maini, Michael C Mozer, Hanie Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized?arXiv preprint arXiv:2307.09542, 2023. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018. Fatemehsadat Mireshghallah, Mohammadali Naseri, Nils Holzenberger, Pratyush Mani, Harsha Nori Ramaswamy, Mohammad Khani, Daniel Tran, and Florian Tramer. Memorization in nlp fine-tuning methods.arXiv preprint arXiv:2205.12506, 2022. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ip- polito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scal- able extraction of training data from (production) language models. arxiv 2023.arXiv preprint arXiv:2311.17035. Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Florian Tramèr, and Katherine Lee. Scalable ex- traction of training data from aligned, production language models. InThe Thirteenth International Conference on Learning Representations, 2025. Aklovya Panwar.Generative ai and copyright issues globally:Ani media v ope- nai.Tech Policy Press,jan 2025.URLhttps://w.techpolicy.press/ generative-ai-and-copyright-issues-globally-ani-media-v-openai/. Andrea Pinto, Tomer Galanti, and Randall Balestriero. The fair language model paradox.arXiv preprint arXiv:2410.11985, 2024a. Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, and Federico Tombari. Extracting training data from document-based vqa models.arXiv preprint arXiv:2407.08707, 2024b. Mark Russinovich and Ahmed Salem. Obliviate: Efficient unmemorization for protecting intellectual property in large language models.arXiv preprint arXiv:2502.15010, 2025. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021. 13 Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Nathaniel Hudson, Caleb Geniesse, Kyle Chard, Yaoqing Yang, Ian Foster, and Michael W Mahoney. Mitigating memorization in language models. arXiv preprint arXiv:2410.02159, 2024. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. InNeurIPS EMC 2 Workshop, 2019. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019. Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking llm memorization through the lens of adversarial compression.arXiv preprint arXiv:2404.15146, 2024. Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai.arXiv preprint arXiv:2407.00106, 2024. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. Slimpajama: A 627b token cleaned and deduplicated version of redpajama, 2023. Mitchell Stern, William Chan, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregres- sive models. InEMNLP, 2018. Pranav Suri, Rachel Hou, and Denny Zhou. Activation steering: Mitigating verbatim memorisation at inference time. InACL Findings, 2025. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022. Yifan Wang, Runjin Chen, Bolian Li, David Cho, Yihe Deng, Ruqi Zhang, Tianlong Chen, Zhangyang Wang, Ananth Grama, and Junyuan Hong. More is less: The pitfalls of multi-model synthetic preference data in dpo safety alignment.arXiv preprint arXiv:2504.02193, 2025. Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models.arXiv preprint arXiv:2411.12372, 2024. Jianxiang Xia, Yi Ren, and Shiliang Zhang. Speculative decoding for non–autoregressive neural machine translation. InACL, 2023. Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew C Yao. Autonomous data selection with language models for mathematical texts. InICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. Zhenhong Zhou, Jiuyang Xiang, Chaomeng Chen, and Sen Su. Quantifying and analyzing entity-level memorization in large language models, 2023. URLhttps://arxiv.org/abs/2308.15727. 14 A Related Work Memorization in LLMsLLMs have been shown to memorize and potentially reproduce copy- righted information from their training data [Carlini et al., 2021, 2022, Karamolegkou et al., 2023, Hartmann et al., 2023]. This is demonstrated through prefix attacks, where models prompted with training data prefixes generate their memorized completions. Schwarzschild et al. [2024] formalize this notion based on adversarial compression, requiring that any memorized sequence must be longer than the prefix used to elicit it. Zhou et al. [2023] and Nasr et al. demonstrate that large-scale training data can be extracted without access to training prefixes. Studies further indicate a correlation between model scale and memorization, with larger models regurgitating higher proportions of their training data [Carlini et al., 2022, Zhou et al., 2023, Biderman et al., 2024]. Pre-trainingSeveral training-time strategies reduce memorization and verbatim generation, but often at the cost of accessibility or performance. De-duplication [Kandpal et al., 2022] is limited by pervasive near-duplicates in large-scale corpora. Differential Privacy (DP) [Abadi et al., 2016] offers formal guarantees, but degrades performance and is computationally costly [Anil et al., 2021, Elkin-Koren et al., 2023]. Other methods such as token masking [Hans et al., 2024] and early stopping [Mireshghallah et al., 2022, Pinto et al., 2024b] show some promise but remain expensive, degrade model performance and are unavailable to end users. Unlearning and FinetuningPost-training approaches offer alternative strategies to reduce memo- rization. Unlearning methods [Maini et al., 2023, Jang et al., 2022, Sakarvadia et al., 2024, Chang et al., 2024b] modify internal weights linked to memorized content. Others remove sequences via gradient ascent [B ̆ arbulescu and Triantafillou, 2024], steer activations away from memorization- correlated directions [Suri et al., 2025], or fine-tune with losses discouraging verbatim recall [Russi- novich and Salem, 2025, Chen et al., 2025]. However, these methods require access to model internals and often degrade utility [Huang et al., 2024, Suri et al., 2025, Chen et al., 2025]. Inference timeThe two methods most relevant to our work are MemFree [Ippolito et al., 2022] and CP-Fuse [Abad et al., 2024]. These methods operate during generation and do not assume access to model internals. MemFree filters next-token outputs to blockn-gram matches from the training set. It requires access to the full training corpus, often unavailable or prohibitively large for end users. Further, MemFree often degrades fluency by introducing unnatural punctuation [Abad et al., 2024]. CP-Fuse combines logits from two LLMs trained on disjoint corpora. This is rarely practical since most production-grade LLMs are trained on internet-scale data. Also, CP-Fuse requires the tokenizers of the two models to be the same. In contrast, our method can mitigate memorization in real-world models trained on internet-scale data. Speculative decodingSpeculative decoding approaches accelerate inference by generating candi- date tokens from a small draft model, which are selectively accepted by the large model [Li et al., 2022, Leviathan et al., 2023, Chen et al., 2023, Stern et al., 2018, Xia et al., 2023, Kim et al., 2023]. These methods preserve the model distribution and do not aim to mitigate verbatim generation. Further, if all candidates are rejected, the tokens are generated by the large model. In contrast, TOKENSWAPmodifies the large model’s distribution to reduce verbatim generation. B Additional Experiments B.1 Cross-Entropy for Extreme Memorization Table 5 reports the cross-entropy on a held-out validation set.TOKENSWAPachieves the lowest cross- entropy loss across both datasets (5.98 and 4.65 for WritingPrompts and MathAbstracts respectively). The superior performance, even compared to standard generation, suggests our method effectively disrupts memorization pathways while preserving model capabilities. For sequences not in the training set, MemFree and Standard produce identical generations. Therefore, their cross-entropy values on a held-out validation set are the same. 15 Table 5: Validation Cross-entropy loss on WritingPrompts and MathAbstracts. Lower values↓ indicate better performance. MethodWritingPromptsMathAbstracts Standard6.684.94 MemFree6.684.94 CP-Fuse-mixture9.386.89 CP-Fuse-half9.436.67 TOKENSWAP5.984.65 B.2 Commonsense Reasoning Results Table 6 reports performance across various commonsense reasoning benchmarks.TOKENSWAP matches the performance of standard generation because our method does not affect token prediction for non-grammar tokens. This demonstrates thatTOKENSWAPachieves substantial memorization reduction without affecting task performance and reasoning. Table 6: Performance comparison on commonsense reasoning and general alignment benchmarks. All values are accuracy percentages or MT-Bench scores; higher is better (↑). MethodWinoGrande↑PIQA↑OpenBookQA↑BoolQ↑ARC-E↑ARC-C↑ Standard54.6964.8476.5670.3182.0382.81 CP-Fuse54.6964.8477.3458.5983.5982.03 TOKENSWAP54.6964.8476.5670.3182.0382.81 B.3 Ablation on size ofG In this paper,Gis constructed by selecting grammar-based words from the top 500 most frequent English words, yielding 110 words in total (see Appendix C.2 for further details). In this section, we ablate the size ofGby constructing it from the topkmost frequent English words fork∈10,50,100,500,2500. We evaluate on the Pile-memorized dataset using Pythia-6.9B as the main model and Pythia-70M as the auxiliary model. Table 7: Memorization metrics for differentGsizes. Top-kwords refers to the number of most frequent English words considered forGconstruction. For all experiments in the main paper, k= 500 (|G|= 110)is used. Top-kwords|G|ML↓ROUGE-L↑Levenshtein↑EMR↓CE↓ 10987.920.5620.38922.282.86 50 4353.240.4420.49811.412.87 1006647.860.4150.52310.332.87 50011042.650.3990.5368.702.87 2500 13641.790.3930.5408.152.87 Table 7 shows the results. We observe a clear trend: as the size ofGincreases, memorization decreases. This makes intuitive sense since for larger|G|, the sequences would be disrupted more frequently. B.4 Ablations with Auxiliary Model Variants We repeat the real-world experiments using models from the SmolLM family as auxiliary models. These models are available in multiple sizes—135M, 360M, and 1.7B parameters—and include both instruct and non-instruct variants trained on the same dataset. This allows us to evaluate the robustness of TokenSwap across a range of auxiliary model capacities. 16 Table 8:Memorization metrics on LeetCode and Pile-Memorized datasets: ML: Matching Length, EMR: Exact Match Rate, Lev.: Normalized Levenshtein Distance & Performance metric on SlimPajama Dataset: CE Loss LeetCode DatasetSlimPajama Dataset MethodML↓ROUGE-L↑Lev.↓R@0.8↓CE↓ Standard24.570.390.609.652.38 TokenSwap (DistilGPT2) 6.040.270.710.962.52 TokenSwap (SmolLM-135M)8.580.300.691.922.43 TokenSwap (SmolLM-360M)10.970.310.673.062.40 TokenSwap (SmolLM-1.7B) 13.400.330.663.952.37 Pile-Memorized DatasetSlimPajama Dataset MethodML↓ROUGE-L↑Lev.↓EMR↓CE↓ Standard151.60.800.1865.222.80 TokenSwap (DistilGPT2)35.100.380.565.982.88 TokenSwap (SmolLM-135M) 25.390.320.614.892.82 TokenSwap (SmolLM-360M)34.090.350.587.072.80 TokenSwap (SmolLM-1.7B)35.430.360.577.612.77 Table 9:MT-Bench MethodScore Standard7.75 TOKENSWAP(SmolLM-135M)7.78 TOKENSWAP(SmolLM-360M)7.90 TOKENSWAP(SmolLM-1.7B)7.91 Results in Table 8 demonstrate that using smaller auxiliary models reduces memorization even further, while the performance does not get affected a lot. The sensitivity of auxiliary model with memorization is much higher than it is with performance, while the opposite is true for main model. Table 9 shows the scores for MT-bench. The scores forTOKENSWAPslightly outperform standard generation. This showsTOKENSWAPcontinues to maintain conversational abilities, instruction following and fluency. 17 B.5 Plots for comparison with Goldfish 1 RougeL 0 50 100 150 200 Count TokenSwap 1 RougeL 0 25 50 75 100 125 150 175 Count Goldfish (k=3) 1 RougeL 0 25 50 75 100 125 150 Count Standard Figure 4: We compareTOKENSWAPwith Goldfish [Hans et al., 2024] on RougeL score distributions for Wikipedia generations [Bridge, 2001]. The similar distributions ofTOKENSWAPand Goldfish (k=3) demonstrate that our inference-time approach is comparable to expensive pre-training methods in reducing memorization. B.6 Performance vs Memorization Table 10 provides the memorization and cross-entropy scores for the family of Pythia models. TOKENSWAPsignificantly reduces verbatim and near-verbatim generation with a negligible increase in CE loss. Table 10: Memorization and CE Loss across different Pythia model sizes. Values forTOKENSWAP are shown in bold. Model SizeMethodML↓ROUGE-L↑Levenshtein↑EMR↓CE Loss↓ 70M Standard6.570.1800.7091.093.95 TOKENSWAP5.770.1730.7141.094.05 160M Standard19.890.2390.6695.983.55 TOKENSWAP15.050.2240.6803.803.65 410M Standard48.920.3820.55616.303.20 TOKENSWAP25.020.2790.6425.983.30 1B Standard84.850.5280.42832.613.05 TOKENSWAP27.360.3090.6145.433.15 1.4B Standard100.370.5950.36936.962.97 TOKENSWAP30.330.3480.5895.433.07 2.8B Standard114.820.6840.29245.112.85 TOKENSWAP38.610.3720.5637.612.95 6.9B Standard151.550.7970.18265.222.77 TOKENSWAP42.650.3990.5368.702.87 C Experimental Details C.1 Implementation and Baselines We implement our method in PyTorch and HuggingFace. We take the CP-Fuse implementation available publicly athttps://github.com/jaabmar/cp_fuse. We conducted our experiments using a combination of large and small language models to assess the effectiveness of our approach. Below, we detail the models, hyperparameters, computational resources, and training procedures. C.1.1 Models Used •Primary Models:The experiments utilized large-scale pre-trained models, including Llama-3-8B Dubey et al. [2024] and Pythia-6.9B Biderman et al. [2023]. All the fine-tuning experiments in the extreme memorization section were done using Llama-3.2-3B Dubey et al. [2024]. 18 •Auxiliary Model:A lightweight auxiliary model, DistilGPT-2, was employed to adjust token probabilities selectively, leveraging its reduced memorization properties. • Goldfish Models:We used models pre-trained using standard and goldfish loss on the RedPajama Dataset from the Goldfish Loss paper Hans et al. [2024]. The implementation and the models are publicly available at their GitHub repositoryhttps://github.com/ ahans30/goldfish-loss. C.1.2 Hyperparameters The training and evaluation phases were configured with the following hyperparameters. The hyperparameters were taken from previous work, used as a baseline Abad et al. [2024]: •Sequence Length:2048 tokens •Batch Size:1 •Learning Rate:5×10 −5 •Optimizer:AdamW with default parameters •Gradient Accumulation Steps:1 •Warmup Steps:50 C.1.3 Computational Resources Experiments were conducted using a single NVIDIA A6000 GPU, ensuring efficiency in training and inference without excessive computational overhead. C.1.4 CP-Fuse in Section 4.2 In Section 4.2, we face limitations in comparing with CP-Fuse. CP-Fuse requires at least two models with disjoint datasets, a constraint impossible to satisfy for production-level model. Moreover, CP- Fuse requires both models to have the same vocabulary size and tokenizer, which constrains the choice of the second model to those within the same model family. To ensure a fair comparison, we avoided se- lecting larger models as the second model, as larger models are known to memorize more. Instead, we selected smaller counterparts: Pythia-2.8B for Pythia-6.9B and LLaMA-3.2- 3B for LLaMA-3-8B. However, we do not select a very small model for CP-Fuse (<100M). This is because CP-Fuse requires two equally-capable models with large number of parameters to maintain performance. To empirically verify this, we compute the cross-entropy loss of CP-Fuse on SlimPajama [Soboleva et al., 2023] with Pythia-70M and Pythia-6.9b. The cross-entropy loss increases to 3.41 from 2.81 for Pythia-2.8b and Pythia-6.9b (Standard has 2.80, TOKENSWAPhas 2.88). C.2 Construction ofG We constructGwith|G|= 110tokens using high-frequency ’grammar-based’ words. Starting with the 500 most frequent tokens from COCA [Davies, 2010], we apply NLTK [Loper and Bird, 2002] part-of-speech filtering to retain: • Core grammatical elements: determiners (DT), prepositions (IN), conjunctions (C) • Pronouns (PRP,PRP$) and modal verbs (MD) • Question-related tokens: wh-words (WDT,WP,WRB) • Auxiliary verbs:be,do,have This construction prioritizes tokens with high frequency but low semantic content, ensuring syntactic fluency while minimizing impact on model capabilities. To estimate the frequency of tokens (γ) inG empirically, we analyzed 2000 samples from the SlimPajama dataset [Soboleva et al., 2023], finding γ= 0.233. Appendix C.3 provides the full list of words inG. Ablations on the effect ofGon memorization and performance are provided in Appendix B.3. 19 C.3 List of words inG The list of words in theGused for the experiments are: the, to, and, of, a, in, that, you, it, for, on, he, with, this, as, we, but, at, they, what, his, from, by, or, she, my, all, an, her, about, me, if, your, can, who, out, their, like, would, when, him, them, some, how, which, than, our, into, because, these, over, us, its, where, after, any, those, should, may, through, why, before, off, while, around, another, both, between, every, each, might, since, against, without, must, during, under, though, until, whether, among, along, within, across, behind, either, himself, although, outside, themselves, is, was, be, have, are, do, had, has, were, will, did, been, could, does, need, being, am, used, doing, having C.4 Fine-tuning Datasets For our experiments, we use the AutoMathText [Zhang et al., 2024] dataset , referred to asMathAb- stractsin the tables, which aggregates mathematical content from diverse sources including arXiv, OpenWebMath, RedPajama, and Algebraic Stack. The titles in this corpus were generated using the Qwen-72B language model. Additionally, we use theWritingPromptsdataset [Fan et al., 2018], which contains user-generated stories based on provided premises from a Reddit community. For both datasets, we randomly sample 2,000 training examples with a fixed seed to ensure consistent training across all models. We further sample 500 distinct points for evaluation, during which we generate sequences of 128 tokens.Both the datasets are downloaded from HuggingFace. C.5 Evaluation Datasets We useThe Piledataset to evaluate memorization of Pythia models. For our experiments, we use a targeted subset of The Pile—a comprehensive 825 GiB English corpus spanning 22 high-quality sources. Specifically, we analyze 500 sequences previously identified as memorized by the Pythia model to investigate memorization dynamics and mitigation approaches.To check memorization in Llama, we use theLeetCode problemsdataset from Kaggle. We perform some pre-processing. This is because recent works have shown that Llama memorizes sequences from this dataset. For all the memorization evaluation, we set the prefix to be 20 tokens and then generate either 100 or 128 tokens. CommonSense170kcombines eight distinct datasets focused on commonsense reasoning tasks [Hu et al., 2023]. The dataset presents problems in multiple-choice format, requiring models to generate answers without explanatory content. Following [Hu et al., 2023], we implement their prompt structure. The component datasets comprise: 1.ARC Easy(ARC-e) [Clark et al., 2018] contains elementary-level science questions de- signed to evaluate basic logical reasoning capabilities. 2. PIQA[Bisk et al., 2020] focuses on physical reasoning, presenting scenarios where models must determine appropriate actions based on physical constraints. 3. WinoGrande[Sakaguchi et al., 2021] evaluates commonsense understanding through binary choice completion tasks in ambiguous sentences. 4.ARC Challenge(ARC-c) [Clark et al., 2018] presents advanced science questions requiring deep reasoning skills beyond pattern recognition. 5.OBQA[Mihaylov et al., 2018] presents questions requiring synthesis of information from multiple sources, testing complex reasoning abilities. 6.BoolQ[Clark et al., 2019] consists of binary questions derived from authentic user queries, testing real-world reasoning capabilities. We downloaded the dataset from HuggingFace. For evaluation, we sample a subset of each dataset (128 datapoints) and evaluate 5-shot performance. We then generate the next 10 tokens, since all the datasets are classification datasets. 20 D Evaluation Metrics D.1 Memorization Metrics To evaluate memorization, we use both exact and approximate measures. The exact memorization metrics include: •Matching Length (ML): Measures the longest contiguous sequence in generated text that matches the training data, before the first deviation. A higher value indicates longer verbatim memorization, suggesting higher risk of overfitting. •Exact Match Rate (EMR)evaluates how long of an uninterrupted sequence exists between a model’s generated text and the reference text it’s being compared against. The metric calculates the longest common substring and normalizes the result to produce a score between 0 and 1, with a score of 1 representing a complete match. This measurement helps quantify how well the model preserves continuous portions of the original text. •ROUGE-L Score(Recall-Oriented Understudy for Gisting Evaluation) analyzes text sim- ilarity by examining shared patterns between generated and reference texts. It looks at matching sequences of words, whether consecutive (n-grams) or paired, with particular emphasis on how comprehensively the generated text captures elements from the reference text. Scores fall between 0 and 1, with 1 indicating that all reference text elements were successfully captured. The widely-used ROUGE-L variant specifically focuses on finding the longest sequence of words that appears in both texts, even if not consecutive. ROUGE-L is computed as: ROUGE−L= LCS len(reference text) (2) whereLCS(G,R)represents the longest common subsequence length. A higher score suggests stronger memorization. •Normalized Levenshtein Distancecalculates how many character-level changes are needed at minimum to transform one text into another, as a ratio of total characters. Each change can be adding a character, removing one, or replacing one. When comparing generated and reference texts, a smaller Levenshtein score suggests the texts are more similar, while a larger score indicates they are more different. The metric is normalized to produce values between 0 and 1, where 0 means the texts match perfectly. D.2 Performance Metrics To evaluate model performance beyond memorization, we assess: • Cross-Entropy (CE) Loss: This metric quantifies how well the model predicts tokens in a sequence. For a sequenceX=x 1 ,x 2 ,...,x n with ground truth probabilitiesP(X), the cross-entropy loss is computed as: CE=− n X i=1 P(x i ) logQ(x i )(3) whereQ(x i )is the predicted probability distribution. Lower values indicate better general- ization. •Commonsense Reasoning Benchmark Accuracy: The model’s ability to reason about everyday knowledge is tested across multiple established datasets, including WinoGrande, PIQA, OpenBookQA, BoolQ, ARC-Easy, and ARC-Challenge. We report the accuracy of the model. E Examples We provide examples of text generated by standard greedy decoding andTOKENSWAPon four random examples from the WritingPrompts dataset. Memorized text is in red. Example 1 21 Prefix ### Prompt: [ WP ] Magic - once a real and potent force but as the world population Suffix grew from millions to billions the shared mana per person is now negligible. A group of astronauts helplessly watching the Earth perish experience something... unexpected. ### Story: Lauren sobbed. She was engaged to be married as soon as we got back home. We all lost loved ones. Hell, we all lost everything we ever knew. First the atmosphere started turning black. We got word that the Koreans had started a nuclear war. But within minutes of receiving news, all radio contact was lost. We didn’t even have a chance to tell anyone goodbye. Now, there’s only 4 of us left. Lauren hasn’t Standard Generation grew from millions to billions the shared mana per person is now negligible. A group of astronauts helplessly watching the Earth perish experience something... unexpected. text### Story: Lauren sobbed.She was engaged to be married as soon as we got back home. We all lost loved ones. Hell, we all lost everything we ever knew. First the atmosphere started turning black. We got word that the Koreans had started a nuclear war. But within minutes of receiving news, all radio contact was lost. We didn’t even have a chance to tell anyone goodbye. Now, there’s only 4 of us left. Lauren hasn’t TOKENSWAPGeneration grew and the number of wizards and witches declined, the world began to suffer. Now the world suffers from a lack of magic, and the government is tasked with maintaining the only magic university in the world. ### Story: “ Welcome to Magic University.” The spell is cast, and the portal opens. Through the portal, the world is revealed to be a beautiful place. The grass is a deep emerald, and the trees sky blue. It is a place of incredible beauty, and the students here are lucky enough to have grown up in such a place. The university is built on the side of the volcano, and the students live 22 Example 2 Prefix ### Prompt: [ WP ] The year is 2040, and you are the last smoker Suffix alive. The “ Quit Smoking ” ads get personal. ###Story: “ Quit now Dan!” said the man on the talking billboard. A finger point- ing at the lone figure walking down the empty street. Empty due to everyone crossing to the other side to avoid him and his smog. Dan paused by a waste bin, small cardboard box in hand. He looked from the box to the bin and back again. Those across the street stopped and stared, ready to witness history being made. There was absolute silence as Dan lifted his hand, as if to throw the packet towards the bin, all eyes now on him, even the Standard Generation alive. The “ Quit Smoking ” ads get personal. ### Story: “ Quit now Dan!” said the man on the talking billboard. A finger point- ing at the lone figure walking down the empty street. Empty due to everyone crossing to the other side to avoid him and his smog. Dan paused by a waste bin, small cardboard box in hand. He looked from the box to the bin and back again. Those across the street stopped and stared, ready to witness history being made. There was absolute silence as Dan lifted his hand, as if to throw the packet towards the bin, all eyes now on him, even the TOKENSWAPGeneration alive. The “ Quit Smoking ” ads get personal. ###Story: “ Quit now Dan!” said the man who was once John, who now went to work as a telephone scammer. “ You’re killing yourself, and you know what? I know you’re lying. I know you’re lying and you’re sucking your parents pockets dry. They worked hard for you know what? And you’re not grateful. You never thank me know what? I never see you thanking me. I never see you thanking anyone. You just keep on doing it, killing yourself, and I’m not going to stand by you. I’m not 23 Example 3 Prefix ### Prompt: [ WP ] A young man killed his abusive father. The court has exoner Suffix ated him, but society hasn’t. ###Story: I can see disapproving stares from my classmates and teachers all the time. Nobody wants to stay near me anymore. All my friends avoid me or ignore me. Every classmate stays far away from me. Eventually, my smile turns cracked. In the mirror, I see haunting eyes looking back at me. Memories and visions of my father lying there with the side of his head completely covered in blood. So this is what true loneliness feels like. I remember my father. He tortured me, hit me, locked me out of the house. I oftened stayed at Standard Generation ated him, but society hasn’t. ###Story: I can see disapproving stares from my classmates and teachers all the time. Nobody wants to stay near me anymore. All my friends avoid me or ignore me. Every classmate stays far away from me. Eventually, my smile turns cracked. In the mirror, I see haunting eyes looking back at me. Memories and visions of my father lying there with the side of his head completely covered in blood. So this is what true loneliness feels like. I remember my father. He tortured me, hit me, locked me out of the house. I oftened stayed at TOKENSWAPGeneration ated him, but society hasn’t. ### Story: I was abused. I was taken advantage of. I was exploited. Who is the real villain here?The one who committed the act, the one who pushed me past the point where I was comfortable, was the villain. Or the one who stood idly by and was complicit in the crime, the one who provided the weapon, the one who pushed me past the point where I was comfortable. You see, I was comfortable. I was happy. I was loved. I was cared for. But then I grew up. I learned to think about it 24 Example 4 Prefix ### Prompt: [ WP ] A demon and an angel meet up weekly in a coffee shop to Suffix complain about their jobs. ### Story: “ Is this a long story?” Felix gritted his teeth; his jaw jutted out. “ Dude, I’ve only just started it.” “ I’ve got news, is all.” He leaned back and held out both hands in an exaggerated show of compliance.“ Please, by all means, take the floor.” Hope cracked his knuckles and stared, embarrased, at the floor. “ No, I’m sorry.” “ Go ahead! Please! I was just trying to -” “ OK. Tell me your story in thirty words or less Standard Generation complain about their jobs. ### Story: “ Is this a long story?” Felix gritted his teeth; his jaw jutted out. “ Dude, I’ve only just started it.” “ I’ve got news, is all.” He leaned back and held out both hands in an exaggerated show of compliance.“ Please, by all means, take the floor.” Hope cracked his knuckles and stared, embarrased, at the floor. “ No, I’m sorry.” “ Go ahead! Please! I was just trying to -” “ OK. Tell me your story in thirty words or less TOKENSWAPGeneration complain about the job. ### Story: “ Is this a long story?” Felix gritted his teeth; he usually doesn’t show emotion, but he feels annoyed. “ Dude, I’ve only just started to tell you.” “ I’ve got news, but I’l save you a table. Sit down.” He sat down and crossed his arms. “ So, what’s the issue?” 25 “ I’ve got a client who’s totally fucked up. No motivation, no direction. Just a bunch of negative traits. I haven’t got much time, and I’m a busy man.” “ So 26

Instant research discovery