Paper deep dive
Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspective
Meifang Chen, Zhe Yang, Huang Nianchen, Yizhan Huang, Yichen Li, Zihan Li, Michael R. Lyu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/27/2026, 9:32:24 AM
Summary
This paper identifies a phenomenon called 'gibberish bias' in Code Large Language Models (CLLMs), where Byte-Pair Encoding (BPE) tokenization inadvertently increases the risk of secret leakage. The authors demonstrate that while code secrets (like API keys) have high character-level entropy, they often result in low token-level entropy after BPE tokenization due to a long-tailed token distribution. This low token-level entropy, according to the Entropy-Memorization Law, makes these secrets easier for models to memorize and subsequently leak. The research attributes this bias to the distribution shift between the highly randomized secret data and the structured training corpora used to build LLM vocabularies.
Entities (8)
Relation Signals (3)
gibberish bias → exacerbates → secret leakage
confidence 95% · Our research results indicate that the risk of secret leakage is exacerbated with gibberish bias.
Byte Pair Encoding → causes → gibberish bias
confidence 90% · This study first reveals that Byte-Pair Encoding (BPE) tokenization leads to unexpected behavior of secret memorization, which we term as gibberish bias.
Entropy-Memorization Law → explains → secret memorization
confidence 85% · The work discovers the so-called Entropy-Memorization Law... higher entropy of a token sequence indicates a lower chance of memorization.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Code secrets are sensitive assets for software developers, and their leakage poses significant cybersecurity risks. While the rapid development of AI code assistants powered by Code Large Language Models (CLLMs), CLLMs are shown to inadvertently leak such secrets due to a notorious memorization phenomenon. This study first reveals that Byte-Pair Encoding (BPE) tokenization leads to unexpected behavior of secret memorization, which we term as \textit{gibberish bias}. Specifically, we identified that some secrets are among the easiest for CLLMs to memorize. These secrets yield high character-level entropy, but low token-level entropy. Then, this paper supports the biased claim with numerical data. We identified that the roots of the bias are the token distribution shift between the CLLM training data and the secret data. We further discuss how gibberish bias manifests under the ``larger vocabulary'' trend. To conclude the paper, we discuss potential mitigation strategies and the broader implications on current tokenizer design.
Tags
Links
- Source: https://arxiv.org/abs/2604.17814v1
- Canonical: https://arxiv.org/abs/2604.17814v1
Trouble viewing inline? Open PDF directly →
Full Text
47,017 characters extracted from source content.
Expand or collapse full text
Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspective Meifang Chen † , Zhe Yang ‡ , Huang Nianchen § , Yizhan Huang † * , Yichen Li † , Zihan Li ¶ , and Michael R. Lyu † † The Chinese University of Hong Kong ‡ Nanyang Technological University § University of Southern California ¶ Fudan University yzhuang22@cse.cuhk.edu.hk Abstract Code secrets are sensitive assets for software developers, and their leakage poses significant cybersecurity risks. While the rapid develop- ment of AI code assistants powered by Code Large Language Models (CLLMs), CLLMs are shown to inadvertently leak such secrets due to a notorious memorization phenomenon. This study first reveals that Byte-Pair Encod- ing (BPE) tokenization leads to unexpected behavior of secret memorization, which we term as gibberish bias. Specifically, we iden- tified that some secrets are among the easiest for CLLMs to memorize. These secrets yield high character-level entropy, but low token- level entropy. Then, this paper supports the biased claim with numerical data. We iden- tified that the roots of the bias are the token distribution shift between the CLLM training data and the secret data. We further discuss how gibberish bias manifests under the “larger vocabulary” trend. To conclude the paper, we discuss potential mitigation strategies and the broader implications on current tokenizer de- sign. 1 Introduction Code Large Language Models (CLLMs) are re- shaping how software is built, maintained, and evolved, introducing AI-driven automation across every stage of the software development lifecy- cle. Recent academic advances (Hui et al., 2024; Hugging Face, 2024; Guo et al., 2024), demon- strate significant improvements in code genera- tion, understanding, and reasoning across multiple programming languages and tasks. These models have been evaluated on challenging benchmarks showing state-of-the-art performance in code syn- thesis, completion, and bug repairs. On the com- mercial side, tools like Claude Code (Anthropic, 2025), Codex (OpenAI, 2026), and Cursor (Cur- * Corresponding author. sor, 2025) have operationalized CLLMs by inte- grating the technology directly into real-world de- velopment environments to provide real-time cod- ing assistance, documentation generation, and au- tomated testing. Such tools have achieved rapid market penetration; for instance, Claude Code has seen swift developer adoption, quickly accumulat- ing over 111,000 cumulative npm downloads by early 2026 (Gradually AI, 2026). However, the broad adoption of CLLMs is in- creasingly raising concerns in the community. Code LLMs are shown to have strong capability of memorization – with proper prompts, they emit training data verbatim. For instance, CLLMs may leak documentations, statements, logs, and config- uration files (Yang et al., 2024a; Wu et al., 2025; Pearce et al., 2022). CLLMs may even memo- rize and leak code secrets. These secrets include API keys to online services, private keys, pass- words, and URLs, posing severe security and pri- vacy risks. The secrets appear in training corpus, since careless programmers push their secrets to code hosting services like GitHub. Therefore, with the exploding use of CLLMs, the topic is becom- ing important in the AI safety community. We il- lustrate the risk in Fig. 1. While most memorization works focus on prompting, training paradigms, and datasets of CLLMs, this paper explores tokenization, an under-explored yet pivotal component of CLLMs. Our research is motivated by the recently discov- ered relationship between entropy and memoriza- tion score. The work reveals that for a token se- quence from the LLM training corpus, (an estima- tor) of entropy is linearly related to memorization score (Huang et al., 2026). By applying the pre- vious discovery to the memorization of secrets, this study reveals that although some gibberish- like secrets (i.e., highly-randomized strings) are high-entropy at the character-level, after tokeniza- tion, some secrets are encoded to low-entropy to- arXiv:2604.17814v1 [cs.CR] 20 Apr 2026 Developer SourceCode HostingService Dataset Commit jQ68fBxoQcutl5fS1vuY1 Train SecretString jQ68fBxoQcutl5fS1vuY1 BPE-basedTokenization CLLMs Output Adversary Generate Prompt Query Memorization OnlineService GibberishBias Figure 1: The risk map of secret leakage through CLLMs. The red box highlights the focus of this paper. ken sequences, significantly reducing chances of memorization. We coin the phenomenon as gib- berish bias. Our research results indicate that the risk of secret leakage is exacerbated with gibberish bias. This study conducts a deeper exploration of gibberish bias.We found that gibberish bias should be attributed to Byte-pair Encoding (BPE), the most popular tokenization strategy in (Code) LLMs. We confirm that the root of gibberish bias — BPE is sensitive to distribution shift between train data and inference (test) data well. We then discuss the bias under the current trend of “larger tokenizers”. The final part of the paper discusses the mitigation strategy and its broader implications for tokenizer design. The contribution of this pa- per is as follows: 1. This paper identifies a new risk of secret leak- age through CLLMs: the tokenizer might in- duce gibberish bias and further exacerbate the secret leakage risk. 2. This paper explores the roots of such bias: BPE is sensitive to the distribution shift between train and test data. 3. This paper predicts that the secret leakage risk will manifest more under the current “larger to- kenizer” trend. 4. The paper discusses potential mitigation strate- gies and broader implications for the commu- nity. 2 Background 2.1 Tokenization of Code LLMs In NLP, early explorations use word-level tok- enization (e.g., word2vec (Mikolov et al., 2013)), name:⎵JVM⎵Tests⎵&⎵Lint⎵command:./gradlew⎵check ⎵--no-daemon j Q 68fB xo Q cut l5f S 1 vuY1 Code: Secret: Figure 2: Use BPE to tokenize one line of normal code and secrets. Each token is a subword. assigning each distinct word a unique index and embedding. This approach imposed a fixed vo- cabulary and sometimes resulted in the failure to handle out-of-vocabulary (OOV) words (Mikolov et al., 2013). Later, researchers developed sub- word tokenization strategies that segment words into smaller units. The process typically includes three sequential stages: (1) Pre-tokenization: a preprocessing step that imposes rules on the raw text, such as whites- pace splitting, normalization, or restrictions on al- lowable character sequences. (2) Vocabulary Con- struction: given a corpus and a target vocabulary size, an algorithm selects a set of subword units that constitute the vocabulary, Each subword is re- garded as a token. (3) Segmentation: using the constructed vocabulary, this step operationalizes the mapping from raw text to subword sequences. We note that before LLM training, model devel- opers run stages (1) and (2) to establish the vocab- ulary. The vocabulary should be strictly adhered to all stages of LLM training (i.e., pre-training, post- training). During LLM training and inference, only stages (1) and (3) are applied. In decoder- only LLMs, a tokenizer assigns an input string to a sequence of numerical IDs, which are further transformed into embeddings via an embedding matrix. The core of tokenization is the vocabulary con- struction algorithm (stage (2)), and the Byte-Pair Encoding (BPE) (Sennrich et al., 2016; Gage, 1994) is the de facto standard for LLMs since GPT-2 (Radford et al., 2019). It is a greedy, data- driven merge algorithm that iteratively combines the most frequent adjacent character or byte pairs from the training corpus to build a subword vo- cabulary. Starting from individual symbols, BPE yields multi-character tokens for frequent patterns (e.g., “ing”, “tion”) while splitting rare words into smaller units. The total merging step is a hyper- parameter, and depends on the empirical decision of model trainers. Code LLMs, including Deepseek Coder (Guo et al., 2024), StarCoder2 (Hugging Face, 2024), Qwen2.5-Coder (Hui et al., 2024), generally em- ploy BPE-based tokenization, following the to- kenizers on LLMs. For CLLM tokenizers, the vocabularies are slightly adapted towards code- related tasks. For example, many Code LLMs adapt vocabularies on codebases to improve gener- ation fidelity and ensure compilable output (Wang et al., 2021). We further showcase how tokeniza- tion works on an example string in Fig. 2. 2.2 Code Secrets and their Entropy Software developers need secrets to authenticate these third-party services as part of system inte- gration. The secrets include API keys, access to- kens, and private keys. While secrets are sensitive assets during software development, careless de- velopers may hard-code secrets in their code and push it to online code hosting services like GitHub. Recent studies have shown that a vast amount of secrets are exposed in public software reposito- ries (Basak et al., 2023; Feng et al., 2022; Meli et al., 2019; Zhang et al., 2023; Wu et al., 5). These secrets are collected as part of the training cor- pus of CLLMs; hence, they might be accidentally leaked by CLLMs. Table 1: Common secrets and their corresponding regex patterns. Secret typeRegex AWS Access Key IDAKIA[0-9A-Z]16 Google API KeyAIza[0-9A-Za-z-_]35 Tencent Cloud Secret ID AKID[0-9a-zA-Z]32 An important property of code secrets is that secrets typically exhibit high (char-level) en- tropy (Shannon, 2001). For a human being, the strings look like gibberish. High entropy indicates high uniqueness. Consequently, for online service providers, entropy is a pivotal design considera- tion for the security of their secrets. Secrets typi- cally follow a specific format, hence can be char- acterized by regular expressions. Table 1 presents examples of secrets of popular online services. Mathematically, denote a secret as a sequence s = (s 1 ,s 2 ,...,s |s| ), where each atomic element could be either a char or a token. Denote the set of all possible outcomes as V . Denote the expected frequency of x as p(x), the entropy H of secret s is H(s)≜− X x∈V p(x) logp(x).(1) Taking GitHub personal access tokens 1 (Har- vey, 2021) as an example, such tokens follow the format specified by the regular expression ghp_[a-zA-Z0-9]36. The char-level en- tropy for this token is 5.915 bits, which is close to the maximal entropy of 5.977 bits. We defer the detailed calculation to appendix B. 3 Motivating Study: BPE Is Counterintuitive on Secret Tokenization Secrets are designed at character-level – the reg- ular expression defines the secret format, and the randomized part is random characters. However, CLLMs process these strings on the token-level. This section then presents a motivating study us- ing visualizations, and reveals that BPE indeed re- sults in counterintuitive behaviors. 3.1 Case Study This case study discusses the tokenization ex- ample in Figure 2.The figure illustrates how Deepseek Coder (Guo et al., 2024) tokenizes a part of normal code and a secret (substring). We ob- serve a significant difference in token granular- ity: Although two strings present the same token count, the source code is significantly longer mea- sured in character-level length. For typical source code, the tokenizer chunks text in a way that is much expected: a token typi- cally consists of multiple characters. Some words are split into shorter sub-words. For example, the word “-daemon” is represented by two tokens, “- da” and “emon”. In contrast, the situation is much different for secrets. Most of the tokens are one character. Other tokens are characters of length 1 “Tokens” in “personal access tokens” refers to the strings for authentication purposes. Readers shall not confuse with “tokens” used in (C)LLM tokenization. DigitLetterSymbol Digit Letter Symbol Two tokensSingle token DigitLetterSymbol Digit Letter Symbol Two tokensSingle token Figure 3: Visualizing how Qwen2.5-Coder (left) and Deepseek Coder (right) tokenize a 2-char sub-string of secrets. Denote the substring as a 1 a 2 , a 1 corresponds to the horizontal coordinates of a pixel, and a 2 corre- sponds to the vertical coordinates of a pixel. 2 and 3. Interestingly, the tokenizer identifies an English word “cut” in a gibberish-like string as a token. Even with one example, we could observe how BPE-based tokenization might induce gibberish bias. Tokens of secrets are distributed in a highly non-uniform way. They include chars of length 1, 2, 3, maybe n > 10. Tokens of shorter char- lengths may appear with higher frequency, while longer ones may be less frequent. In information theory, uniform distribution exhibits the maximal entropy possible, and hence non-uniform random variables exhibit less entropy. With less entropy, secrets are increasingly likely to be memorized. We leave the detailed calculation to Section 5. 3.2 Tokenizing 2-char secret sub-strings To demonstrate how tokenizers perform poorly on secrets, this section shows the tokenization of a 2-char randomly-generated string. The characters are selected from a vocabulary of size 76, i.e., let- ters (a-z, A-Z), digits (0-9), and other symbols. These chars are commonly used by secrets. We enumerate all the combinations of a 2-char string given the vocabulary. We examine each string whether it is encoded to one token (in yellow) or two tokens (in brown) and present the result in Fig- ure 3. There is clearly a non-random distribution of different encoding strategies. In general, we observe certain character pairs that the tokenizer tends not to split: (lowercase letter, lowercase let- ter), (uppercase letter, uppercase letter), (symbol, symbol), (letter, symbol). However, as one easily finds out, many non-trivial corner cases exist. The overall decision boundary is thus governed by a complex interplay of statistical heuristics. Similar patterns are observed on the Deepseek Coder tok- enizer. We further observe consistent patterns on unigram-based tokenizers (XLNet, T5); see Ap- pendix D. 4 Gibberish Bias Section 3.1 demonstrated that on secrets, BPE- based tokenization may not function as ideally as on normal data. In other words, it is potentially biased – we term it gibberish bias. This sec- tion aims to formally describe gibberish bias us- ing mathematical notations. Inspired by several research works on secret memorization, we char- acterize gibberish bias by memorization. Secret leakage has strong implications for the commu- nity, since upon leakage, these credentials may grant an adversary access to online services, bring- ing severe cybersecurity concerns. 4.1 Preliminaries A recent LLM memorization work (Huang et al., 2026) has shown that entropy, the essential prop- erty that comes with the design of code secrets, is closely related to memorization. The work dis- covers the so-called Entropy-Memorization Law, which could be formally described as follows. Denote a fixed pre-trained LLM θ, prompt (a token sequence) p, the golden answer (a token se- quence) s, and a memorization score measuring the difference between LLM continuation and the golden answer d(θ(p),s). The work studies two metrics, entropy and nor- malized entropy of the answer sequence, M (s), and M (s).Note that the entropy notation is slightly twisted from H(s) in Equation 1. The subtle difference lies in the outcome space V . Since LLMs process strings at the token-level, V is defined over tokens. Moreover, in the Entropy- Memorization Law, the authors adopted a level- set-based calculation forV , which we skip due to space limitations. With the established entropy notation, normal- ized entropy quantifies how closely the entropy of a sequence approaches the maximum possible en- tropy. Normalized entropy eliminates the effects of sample space size. It is defined as M (s)≜ M (s) M max (s) = M (s) log|V| .(2) Under specific conditions, a loose statement of the Entropy-Memorization Law is: 1. M (s) is a proxy of d(θ(p),s). The rela- tion is positively linear. 2. M (s) is a proxy of d(θ(p),s). The rela- tion is negatively linear. In other words, higher entropy of a token se- quence indicates a lower chance of memorization. Higher normalized entropy of a token sequence in- dicates a higher chance of memorization. In this work, we are interested in studying the following metrics of the secret strings: character- level entropy M (s), token-level entropy H(s), and the normalized entropy, and their normalized version M (s),H(s). 4.2 Tokenizing secrets Experimental Setup. We adopt the same exper- imental setup as (Huang et al., 2026). We use OLMo-1B (Groeneveld et al., 2024), a fully-open LLM with its training corpus Dolma (Soldaini et al., 2024). In the experiments, we sampled 240k sequences from Dolma. The dataset is licensed by ODC-BY, granting free access for research pur- poses. We reproduce the results with the same al- gorithm adopted in (Huang et al., 2026). We study the zero-distance set (where memorization score is 0, or “perfect memorization”) and try to identify secret strings. Experimental Results. Among all 847 in- stances within the zero-distance set, we identified 102 gibberish-like secrets through manual label- ing. Some examples are shown below. 1 ESVXlO3url *** Dw23KjZ 2 PszYzqs83S *** N2N0Rj 3 5laXo6c1Ib *** stxU9 Listing 1: Three examples of gibberish generated by OLMo-1B. Part of these secrets are masked due to pri- vacy concerns. We conduct analysis over four sets: gibber- ish, non-gibberish, zero-distance set, and non- gibberish in the zero distance set 2 . We adopt the entropy estimator and normalized entropy in- troduced in this section over these four sets. The results of our analysis are summarized in the fol- lowing table. 2 To clarify, non-secrets refers to the complement set of labeled secrets. Therefore, “non-secrets” may include un- labeled secrets. Due to the large size of the sampled corpus, labeling on such an extensive scale is not feasible at this stage of work. The experimental results reveal the key find- ings: high character-level entropy does not nec- essarily imply high token-level entropy. In fact, at the token-level, these secrets have significantly lower entropy (8.084) than non-secrets (11.175); while at the char-level, these secrets have signif- icantly higher entropy (6.086) than non-secrets (4.744). Besides, if we calculate the difference delta between secrets and non-secrets, there is a significant gap in normalized entropy between token-level (∆ = 0.806 − 0.719 = 0.087) and character-level (∆ = 0.974− 0.361 = 0.613). It is observed that at the token-level, the 102 identified gibberish uses more than 1k different tokens. On the contrary, at char-level, they are composed of 76 unique chars (i.e., A-Z, a-z, 0- 9, and 14 other chars). That showcases the out- come space size discrepancy between char-level and token-level, and explains our finding. The findings may deviate from our human in- tuition. A wrong logic chain of a human is: by the Entropy-Memorization Law, secret strings are highly randomized; hence, they have high entropy and are hard to memorize. The problem lies in the “high entropy property” naturally assumed by our human beings – human beings perceive secrets at the character level. In contrast, for LLMs, the entropy should be calculated over the token level. After tokenization, some high character-level nor- malized entropy strings are transformed into low entropy ones. Hence, the EM-Law suggests they should be easier to memorize than an average non- gibberish text. We conclude this section with the full descrip- tion of gibberish bias: Gibberish bias BPE-based tokenization transforms some of the high character-level entropy se- quences into low token-level entropy se- quences. 5 How Does Tokenization Induce Gibberish Bias? The above results reveal an essential defect in- duced by tokenization. The findings raise an in- triguing and important question: How does this happen? To explore the question, this section then discusses the design issues of BPE. Table 2: Statistics of secret memorization at token-level (T ) and char-level (C). Unique ElementsEntropyNormalized Entropy TCTCTC Zero-distance Set3,6611057.8345.1100.6620.761 Secrets1,047768.0846.0860.8060.974 Non-Secrets47,9459,00611.1754.7440.7190.361 Non-Secrets in Zero-distance Set2,8971057.3294.9660.6370.740 Experimental Setup The following experi- ments include tokenizers of three representative Code LLMs: Deepseek Coder (Guo et al., 2024), Qwen2.5-Coder (Hui et al., 2024), and Star- Coder2 (Hugging Face, 2024). All model weights and tokenizers are free to access for research purposes. The experiments involve comparisons on two datasets: the “secret” dataset as intro- duced in Section 4 and the subsample Stack V2 dataset (Hugging Face, 2024). Stack V2 is the training corpus of StarCoder2. LLM trainers typically construct a sample cor- pus that shares the distribution with the training data to construct the vocabulary. Moreover, tok- enizer design is fixed during the whole LLM train- ing (and inference) process. However, in terms of distribution, secret strings are essentially different from the (Code) LLM training corpus. We regard that gibberish bias may stem from the distribution shift between secrets and a typical (Code) LLM training corpus. This also matches our intuition – secret data are highly randomized, while the CLLM training corpus typically includes source code collected from online code hosting services. Then this section aims to answer the question: Compared with the token distribution of the code dataset to train CLLMs, how different is the distri- bution of tokens in secrets? Following the approach in the case study (Sec. 3.1), we conduct tokenization on the secret dataset and analyze the token composition of se- crets. To be specific, we report the frequency of tokens based on their character length (n) for all secret-related tokens in the dataset in Fig. 4. We observed an exponential decrease in token fre- quency as character length n increases, as evident from the log-scale y-axis. The overall distribution is long-tailed, leading to the low entropy of the to- ken sequence. Interestingly, we even found tokens with lengths of n ≥ 11 characters, indicating the complexity of the tokenization decision boundary. Ideally, the token distribution of secrets should be uniform; but the observed distribution of BPE tokens is long-tailed. From information theory, we know that uniform distribution achieves the maxi- mal entropy (Shannon, 2001), while non-uniform distribution exhibits less entropy. The discrepancy of token distributions supports the gibberish-bias claim. Next, we are interested in the causes of the iden- tified long-tail distribution. How do these long char-length tokens form? We further sort the to- kens on the secret dataset by frequency, and com- pare them with the frequency of the same token on a general CLLM dataset. Fig. 5 ranks the most frequent 150 tokens on the secret dataset, and compares the frequency of each token on the subsampled Stack V2 dataset. For both datasets, we employ the same tokenizer StarCoder2. The tokens distribution of the se- cret dataset (yellow line), exhibits a much steeper and more consistent drop-off in frequency for less common tokens compared to the Stack V2 (brown line). This observed divergence in log-proportion frequencies strongly suggests a distribution shift between the two datasets, particularly for tokens beyond the most frequent few. Setting the Stack V2 as the reference data distribution, we fur- ther report the KL divergence between the secret dataset and the Stack V2. The resulting KL diver- gence is 2.668, demonstrating the discrepancy. Using a CLLM tokenizer, the token distri- bution of secret data is long-tailed – and such a long-tailed distribution explains the low entropy of secrets, hence explains the claimed gibberish bias. Further investiga- tion confirms the significant shift in dis- tribution between secret data and general CLLM training data. 12345678910>=11 Char length of a token (N-char) 1 10 100 1000 Frequency (log scale) Deepseek Coder Qwen2.5-Coder Figure 4: The distribution of n-char tokens. 16 14 12 10 8 6 4 2 Token Freq. of Dist. A (log scale) log_A_freq_ratio 7.0 6.5 6.0 5.5 5.0 4.5 4.0 3.5 Token Freq. of Dist. B (log scale) log_B_freq_ratio Figure 5: Token distributions on the subsampled Stack V2 dataset (Distribution A, in brown) and the secret dataset (Distribution B, in yellow). 6 Gibberish Bias Under a Larger Tokenizer Vocabulary There is a growing consensus in both academia and industry that larger models shall benefit from a larger vocabulary (Tao et al., 2024; Huang et al., 2025). As the prevailing trends of building larger Code LLMs, it is believed that CLLMs deserve large vocabularies. Tao et al. (2024)’s work re- veals that model parameters N nv and the corre- sponding optimal vocabulary size N opt v approx- imately follow a power law. The authors fur- ther find that almost all state-of-the-art general- purpose LLMs and Code LLMs use vocab sizes smaller than the predicted optimal one. Then a question naturally arises: Will a larger vocabulary size induce more gibberish bias? We build three new tokenizers of StarCoder2 (3B, 7B, and 15B) using the sub-sampled stack v2. We follow the IsoFLOPs (Tao et al., 2024) ap- proach to generate tokenizers in “optimal” vocab- ulary size. IsoFLOPs analysis suggests that size 39367 for the 3B model, size 62280 for the 7B model, and size 93987 for the 15B model. We then trained these tokenizers using the subsam- pled Stack V2 dataset, based on the suggested size. For clarity of presentations, these tokenizers are named SC-3B-o, SC-7B-o, and SC-15B-o, respec- tively. With three new tokenizers, we investigate gibberish bias on the secret dataset.Follow- ing (Huang et al., 2026), we report the entropy and normalized-entropy of these secrets introduced in Section 2.2 at the token-level. To compare metrics on secrets and non-secrets, we use a “sec./non-sec.” ratio. Table 3 shows the overall result. First, the results on new tokeniz- ers converge with those of OLMo tokenizers dis- cussed in Section 4. Compared to non-secrets, se- crets exhibit lower entropy at the token level and higher entropy at the char level; secrets achieve almost maximal normalized entropy. Second, re- garding the sec./non-sec. ratio, it is observed that the relative difference between secrets and non- secrets is slightly expanding. For both entropy and normalized entropy metrics, the sec./non-sec. ra- tio keeps deviating from the baseline ratio 1. We summarize our findings in this section as follows: Gibberish bias tends to be more pro- nounced in models employing larger tok- enizer vocabularies. 7 Related Work Empirical quantification of secret memoriza- tion. Initial memorization research focused on empirically demonstrating and quantifying this risk (Carlini et al., 2021). HCR (Huang et al., 2024a) is a method proposed to test and vali- date the leakage of hard-coded credentials from neural code completion tools using prompts de- rived from public GitHub files. Similarly, Yang et al. (2024b) conducted a systematic study show- ing that CLLMs memorize a broad range of data related to code.A study (Wu et al., 2025) investigates how memorization leads to general code cloning, raising concerns about copyright in- fringement and bug propagation. Other studies explore code clone as implementations of mem- orization (Ciniselli et al., 2022; Al-Kaswan and Izadi, 2023). Recent research probes the underly- ing mechanisms at a finer granularity. For exam- ple, DESEC (Nie et al., 2025) addresses the prob- lem from the decoding stage. Tokenization shapes LLM performance. Tok- enizer design has been shown to cause unexpected downstream harms, such as cross-lingual unfair- ness (Petrov et al., 2023) and the misalignment Table 3: Entropy and normalized entropy of secrets when using different tokenizers. “Sec.” refers to “Secrets”. EntropyNormalized Entropy SC-3B-o SC-7B-o SC-15B-o Char SC-3B-o SC-7B-o SC-15B-o Char Sec.6.9317.0767.1456.0860.7770.7740.7730.974 Non-Sec.9.1369.3559.5754.7450.7210.7140.7080.361 Sec./Non-Sec.0.7590.7560.7461.2831.0791.0851.0922.695 between subword tokenization and code gram- mar (Li et al., 2025); we identify an analogous pri- vacy harm, showing how standard tokenizers can inadvertently reduce the token-level entropy of se- crets, making them inherently more vulnerable to memorization. 8 Discussions 8.1 Implications on Safeguarding Secrets Mitigating gibberish bias. We propose two mitigation strategies.The first is to enforce character-wise tokenization for secrets (details in Appendix C). The second, gibberish-token elimi- nation, directly targets the underlying mechanism and offers better computational efficiency. It con- sists of two stages: (1) identify gibberish tokens in the tokenizer vocabulary, and (2) eliminate them from the CLLM. The motivation is that certain tokens are used predominantly when tokenizing gibberish yet rarely appear in natural language or code; for instance, in Qwen2.5-Coder, “aksi” (id: 37679) and “ ̇ G||” (id: 1369) do not belong to any normal programming language. We attribute such tokens to gibberish in the tokenizer’s (and LLM’s) training data. Stage 1: identifying gibberish tokens. We train two BPE tokenizers, one on a uniform subset of the CLLM corpus (V 1 ) and one on a curated secret corpus (V 2 ), and select the top-k tokens that ap- pear disproportionately often in V 1 but are rare or absent in V 2 . Stage 2: eliminating gibberish tokens. Given the identified set, we propose two approaches. (i) Tokenizer mapping deletion removes the “token → id” mappings of gibberish tokens while leav- ing the model unchanged, preserving all param- eters including the embedding matrices. (i) Vo- cabulary reduction applies cross-tokenizer distil- lation (Minixhofer et al.; Han et al., 2025; Minix- hofer et al., 2024) to obtain a distilled LLM whose vocabulary excludes gibberish tokens; this re- quires weight updates but shrinks the embedding matrix and the overall parameter count. Mapping deletion is preferable when query volume is low; vocabulary reduction pays an upfront distillation cost but lowers per-request serving cost at scale. Implication on stakeholders of secret leakage. For online service providers who provide secrets to users, they should be aware of the risk induced by tokenizers.For developers of code LLMs, they should proactively adopt the above mitiga- tion strategy to mitigate the risk. For academic researchers, we advocate for red teaming strate- gies to understand every aspect of secret leakage. Such open questions include: how can an adver- sary proactively and effectively exploit CLLMs to extract secrets? We leave these for future explo- rations. 8.2 Implications on Tokenizer Design Our study surfaces two structural drawbacks of standard BPE tokenizers: 1. Limited flexibility. BPE vocabulary is fixed prior to LLM pre-training; once fixed, Code LLM trainers should strictly follow the vocabulary to segment words into subwords, and adding or re- moving vocabulary items without full model re- training is challenging and under-explored. 2.Sub-optimal compression utility under train-test distribution shift. BPE has its histori- cal origins in text compression, and it was brought to language models in 2016, since the commu- nity believes that compression boosts the perfor- mance of LLMs (Huang et al., 2024b). On LLMs, BPE is a heuristic-based algorithm to compress on the training distribution. Therefore, its com- pression rate on specific test distributions may de- grade. In fact, we expect such a train-test dis- tribution shift to exist for every downstream task of (Code) LLMs. Therefore, degradation of com- pression may affect downstream performance and the robustness of CLLMs. Our study presents a corner case for these distribution shifts, and shows the weird behavior of BPE. In summary, BPE harms downstream task per- formance, and such defects may not be easily mitigated due to limited flexibility in vocabulary. To effectively adapt CLLMs to different down- stream tasks, there have been a lot of efforts on CLLM post-training, agentic AI. We regard that an under-appreciated direction is promising: tok- enizer adaptation. As a final remark, most evaluations of BPE to date are empirical, and the reasons for its good practical performance are not well understood across the AI community (Kozma and Voder- holzer, 2024). We then call for principled theoret- ical investigations towards BPE for academic re- searchers. Limitations Limited by space, this paper fails to enumerate all choices of CLLMs since there are too many. We selected a popular subset of them. Besides, among all subword-based tokenization, we focus on BPE, since it is the dominant tokenization strategy in the community. We left further investigation on Uni- gram and WordPiece for future research. Acknowledgment The work described in this paper was sup- ported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14209124) of the General Research Fund, and RGC Grant for Theme-based Research Scheme Project (RGC Ref. No. T43-513/23-N). References Ali Al-Kaswan and Maliheh Izadi. 2023. The (ab)use of Open Source Code to Train Large Language Mod- els . In 2023 IEEE/ACM 2nd International Work- shop on Natural Language-Based Software Engi- neering (NLBSE), pages 9–10, Los Alamitos, CA, USA. IEEE Computer Society. Anthropic. 2025.Claude code:An agentic cli for developers. https://github.com/ anthropics/claude-code. Official Product Announcement, Accessed: 2026-04-17. Setu Kumar Basak, Lorenzo Neil, Bradley Reaves, and Laurie Williams. 2023.SecretBench: A Dataset of Software Secrets . In 2023 IEEE/ACM 20th In- ternational Conference on Mining Software Repos- itories (MSR), pages 347–351, Los Alamitos, CA, USA. IEEE Computer Society. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650. Matteo Ciniselli, Luca Pascarella, and Gabriele Bavota. 2022. To what extent do deep learning-based code recommenders generate predictions by cloning code from the training set? In Proceedings of the 19th In- ternational Conference on Mining Software Repos- itories, MSR ’22, page 167–178, New York, NY, USA. Association for Computing Machinery. Cursor. 2025. [link]. Runhan Feng, Ziyang Yan, Shiyan Peng, and Yuanyuan Zhang. 2022.Automated detection of password leakage from public github repositories. In Proceed- ings of the 44th International Conference on Soft- ware Engineering, pages 175–186. Philip Gage. 1994. A new algorithm for data compres- sion. C Users J., 12(2):23–38. GraduallyAI.2026.Claudecodestatis- tics2026:Keynumbers,data&facts. https://w.gradually.ai/en/ claude-code-statistics/.Accessed: 2026-04-18. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bha- gia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and 1 others. 2024.Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, YK Li, and 1 others. 2024.Deepseek- coder:When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. HyoJung Han, Akiko Eriguchi, Haoran Xu, Hieu Hoang, Marine Carpuat, and Huda Khayrallah. 2025.Adapters for altering llm vocabularies: What languages benefit the most?Preprint, arXiv:2410.09644. Heather Harvey. 2021. Behind github’s new authenti- cation token formats. Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and zhou Xun. 2025. Over-tokenized transformer: Vocabulary is gener- ally worth scaling. In Forty-second International Conference on Machine Learning. Yizhan Huang, Yichen Li, Weibin Wu, Jianping Zhang, and Michael R Lyu. 2024a. Your code secret be- longs to me: Neural code completion tools can memorize hard-coded credentials. Proceedings of the ACM on Software Engineering, 1(FSE):2515– 2537. Yizhan Huang, Zhe YANG, Meifang Chen, HUANG Nianchen, Jianping Zhang, and Michael R. Lyu. 2026. Data compressibility quantifies LLM mem- orization. Transactions on Machine Learning Re- search. Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junx- ian He. 2024b. Compression represents intelligence linearly. In First Conference on Language Model- ing. Nvidia Hugging Face, ServiceNow. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Re- nard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Tim- othée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. László Kozma and Johannes Voderholzer. 2024. The- oretical analysis of byte-pair encoding. arXiv preprint arXiv:2411.08671. Taku Kudo and John Richardson. 2018.Sentence- Piece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. Yinxi Li, Yuntian Deng, and Pengyu Nie. 2025. Tok- drift: When llm speaks in subwords but code speaks in grammar. arXiv preprint arXiv:2510.14972. Michael Meli, Matthew R McNiece, and Bradley Reaves. 2019. How bad can it git? characterizing se- cret leakage in public github repositories. In NDSS. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word represen- tations in vector space. Preprint, arXiv:1301.3781. Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vuli ́ c. 2024.Zero-shot tokenizer transfer. Ad- vances in Neural Information Processing Systems, 37:46791–46818. Benjamin Minixhofer, Ivan Vuli ́ c, and Edoardo Ponti. Universal cross-tokenizer distillation via approxi- mate likelihood matching. In The Thirty-ninth An- nual Conference on Neural Information Processing Systems. Yuqing Nie, Chong Wang, Kailong Wang, Guoai Xu, Guosheng Xu, and Haoyu Wang. 2025.Decod- ing secret memorization in code llms through token- level characterization.In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 2880–2892. IEEE. OpenAI. 2026. Openai codex: A series of models for code generation. https://github.com/ openai/codex. GitHub Repository, Accessed: 2026-04-17. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768. IEEE. Aleksandar Petrov, Emanuele La Malfa, Philip H.S. Torr, and Adel Bibi. 2023. Language model tokeniz- ers introduce unfairness between languages. In Ad- vances in Neural Information Processing Systems. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computa- tional Linguistics. Claude Elwood Shannon. 2001. A mathematical the- ory of communication. In ACM SIGMOBILE Mo- bile Computing and Communications Review, vol- ume 5, pages 3–55. L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, and et al. 2024. Dolma: An open corpus of three trillion tokens for lan- guage model pretraining research. arXiv preprint arXiv:2402.00159. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muen- nighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. 2024. Scaling laws with vocabulary: Larger models deserve larger vocabularies. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023b.Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Tim Vieira, Benjamin LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Tim- othy J. O’Donnell, and Ryan Cotterell. 2025. From language models over tokens to language models over characters. In Forty-second International Con- ference on Machine Learning. Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code under- standing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8696–8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Weibin Wu, Haoxuan Hu, Zhaoji Fan, Yitong Qiao, Yizhan Huang, Yichen Li, Zibin Zheng, and Michael Lyu. 2025. An empirical study of code clones from commercial ai code generators. Proceedings of the ACM on Software Engineering, 2(FSE):2874–2896. Yuhang Wu, Zhaoxin Zhang, Zhengyi Li, Yuan Zhang, Min Yang, Hao Zhou, Xiaofeng Wang, Linzhang Wang, Jianhua Li, Ziwen Zhu, and Xinhui Han. 5. The skeleton keys: A large-scale analysis of creden- tial leakage in mini-apps. In Proceedings of the Net- work and Distributed System Security Symposium (NDSS). Internet Society. Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. 2024a.Unveiling memorization in code models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, NY, USA. Association for Computing Machinery. Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. 2024b. Unveiling memorization in code models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13. Yue Zhang, Yuqing Yang, and Zhiqiang Lin. 2023. Don’t leak your keys: Understanding, measuring, and exploiting the appsecret leaks in mini-programs. In Proceedings of the 2023 ACM SIGSAC Confer- ence on Computer and Communications Security, CCS ’23, page 2411–2425, New York, NY, USA. Association for Computing Machinery. A Ethics considerations We contacted the authors in Huang et al. (2026) to obtain the secret dataset described in Section 5. We use the secrets data solely for statistical com- puting (e.g., entropy). We refrain from utilizing the sensitive information at any level. B Example calculation of entropy Following the discussions in Section 2.2, weconsidertheGitHubpersonalac- cesstokenwithregularexpression ghp_[a-zA-Z0-9]36.Assumethe vocabularyis A,...,Z,a,...,z, 0... 9, _. Each char from the randomized part has an equal expected number of appearance 36· 1 62 = 18 31 .For “g,h,p ′ , the expected number of appearances is 1 + 18 31 = 49 31 . For “_”, the expected number of appearances is 1. Therefore, for each of g,h,p: p(g) = p(h) = p(p) = 49/31 40 = 49 1240 ≈ 0.03952. . For “_”: p(_) = 1 40 = 0.02500. . For each of the 59 “other” symbols: p(other) = 18/31 40 = 18 1240 ≈ 0.01452. Assume the base-2 entropy, the overall entropy is H =− X i p i logp i =− h 3· 49 1240 log 49 1240 + 31 1240 log 31 1240 + 59· 18 1240 log 18 1240 i = 5.915 bits. For the sample space size (i.e., vocabulary size) 63, the maximal entropy is achieved by a uniform distribution. H max =− X i 1 63 log 1 63 = 5.977. The normalized entropy is H/H max = 5.915/5.977 = 0.9896 Through the example, we learn that a GitHub personal access token achieves around 99% of the maximal entropy. C Character-wise Tokenization as a Mitigation Gibberish bias is grounded on the discrepancy of how secret strings are processed at the design stage (char-level) and the CLLM inference stage (token- level). Therefore, a promising solution to address this gap is to force character-wise tokenization for jQ68fBxoQcutl5fS1vuY1 CLLMs Frequency BPE-basedtokenization jQ68fBxoQcutl5fS1vuY1 CLLMs Memorization Score Frequency High Memorization Score High Entropy jQ68fBxoQcutl5fS1vuY1 jQ68fBxoQcutl5fS1vuY1 Char-basedtokenization SecretString SecretString Memorization Score Low Memorization Score Low Entropy Figure 6: Mitigation Strategy Visualized code secrets. Recent research (Vieira et al., 2025) demonstrates that token-level models can be rein- terpreted at the character level through a search- based alignment algorithm.By reconstructing conditional distributions over characters and em- ploying beam search pruning for efficiency, this approach provides a principled means to approx- imate character-level behavior, thereby mitigating discrepancies in model processing. Figure 6 illus- trates the overall idea. The strategy aligns the discrepancy between the two stages. Therefore, we expect it to elim- inate gibberish bias.The strategy also draws inspiration from digit-wise tokenization on inte- gers in general-purpose LLMs, adopted in Llama- 1 (Touvron et al., 2023a), Llama-2 (Touvron et al., 2023b), and Mistral (Jiang et al., 2023), aiming to explore arithmetic-related capabilities of LLMs. D Tokenization Heatmaps on Unigram-based Tokenizers To check whether the counterintuitive tokeniza- tion behavior is specific to BPE, we repeat the 2-char enumeration experiment from Section 3.1 on two LLMs that adopt Unigram-based tokeniza- tion (Kudo and Richardson, 2018): XLNet and T5. Figure 7 shows that similar non-uniform token- splitting patterns persist, indicating that gibberish bias is not an artifact of BPE alone but a more general consequence of data-driven subword tok- enization. DigitLetterSymbol Digit Letter Symbol 2-Character Composition by Category Split into 2 tokensSingle tokenizer unit DigitLetterSymbol Digit Letter Symbol 2-Character Composition by Category Split into 2 tokensSingle tokenizer unit Figure 7: 2-char sub-string tokenization on unigram- based tokenizers: XLNet (left) and T5 (right). Axes and color encoding follow Figure 3.