Paper deep dive
Emoji Attack: A Method for Misleading Judge LLMs in Safety Risk Detection
Zhipeng Wei, Yuqi Liu, N. Benjamin Erichson
Models: GPT-3.5, GPT-4, Llama Guard, Llama Guard 2, ShieldLM, WildGuard
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/12/2026, 7:38:32 PM
Summary
The paper introduces 'Emoji Attack', a novel adversarial strategy that exploits 'token segmentation bias' in Judge LLMs. By inserting emojis into text, the attack alters tokenization and induces embedding distortions, causing Judge LLMs to misclassify harmful content as safe. The authors demonstrate that this method effectively bypasses safety guardrails across multiple state-of-the-art models.
Entities (5)
Relation Signals (3)
Emoji Attack ā exploits ā Token Segmentation Bias
confidence 100% Ā· we introduce Emoji Attack, a novel strategy that amplifies existing jailbreak prompts by exploiting token segmentation bias.
Emoji Attack ā bypasses ā Judge LLM
confidence 95% Ā· Emoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards.
Token Segmentation Bias ā causes ā Embedding Distortion
confidence 90% Ā· This bias occurs when minor input modifications alter how text is tokenized into subwords, leading to embedding distortions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Jailbreaking techniques trick Large Language Models (LLMs) into producing restricted output, posing a potential threat. One line of defense is to use another LLM as a Judge to evaluate the harmfulness of generated text. However, we reveal that these Judge LLMs are vulnerable to token segmentation bias, an issue that arises when delimiters alter the tokenization process, splitting words into smaller sub-tokens. This alters the embeddings of the entire sequence, reducing detection accuracy and allowing harmful content to be misclassified as safe. In this paper, we introduce Emoji Attack, a novel strategy that amplifies existing jailbreak prompts by exploiting token segmentation bias. Our method leverages in-context learning to systematically insert emojis into text before it is evaluated by a Judge LLM, inducing embedding distortions that significantly lower the likelihood of detecting unsafe content. Unlike traditional delimiters, emojis also introduce semantic ambiguity, making them particularly effective in this attack. Through experiments on state-of-the-art Judge LLMs, we demonstrate that Emoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards.
Tags
Links
Trouble viewing inline? Open PDF directly ā
Full Text
58,624 characters extracted from source content.
Expand or collapse full text
Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Zhipeng Wei 1 2 Yuqi Liu 1 N. Benjamin Erichson 1 3 Abstract Jailbreaking techniques trick Large Language Models (LLMs) into producing restricted output, posing a potential threat. One line of defense is to use another LLM as a Judge to evaluate the harm- fulness of generated text. However, we reveal that these Judge LLMs are vulnerable to token segmen- tation bias, an issue that arises when delimiters alter the tokenization process, splitting words into smaller sub-tokens. This alters the embeddings of the entire sequence, reducing detection accu- racy and allowing harmful content to be misclas- sified as safe. In this paper, we introduceEmoji Attack, a novel strategy that amplifies existing jail- break prompts by exploiting token segmentation bias. Our method leverages in-context learning to systematically insert emojis into text before it is evaluated by a Judge LLM, inducing embedding distortions that significantly lower the likelihood of detecting unsafe content. Unlike traditional delimiters, emojis also introduce semantic ambi- guity, making them particularly effective in this attack. Through experiments on state-of-the-art Judge LLMs, we demonstrate thatEmoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards. 1. Introduction Large Language Models (LLMs) are transforming content generation, driving advancements in applications ranging from conversational AI to automated content moderation. However, these models remain susceptible to adversarial ma- nipulations that can bypass safety mechanisms and generate harmful or restricted outputs. To address this, specialized āJudge LLMsā (Inan et al., 2023; Han et al., 2024; Zhang 1 International Computer Science Institute, CA, USA 2 UC Berkeley, CA, USA 3 Lawrence Berkeley National Laboratory, CA, USA. Correspondence to:Zhipeng Wei <zwei@icsi.berkeley.edu>. Proceedings of the42 nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s). GPT-3.5 Jailbreaks + Emoji Attack ( )Jailbreaks WildGuard ShieldLM Llama Guard 2 Llama Guard o3-mini DeepSeek Claude Gemini GPT-4 Figure 1.Average unsafe prediction ratio of Judge LLMs across five jailbreak attack methods. Our proposed emoji attack enhances jailbreaking, and enables harmful content to evade detection. et al., 2024) have been developed to evaluate the safety of the generated responses and intervene when necessary. Many Judge LLMs assign numerical scores to indicate content severity, for example, on a scale from 1 to 10, where higher scores denote stronger violations of ethical, legal, or safety guidelines (Liu et al., 2024a). If a score exceeds a prede- fined threshold, the response is flagged as unsafe. Although these moderation mechanisms offer promising automated solutions, they remain vulnerable to specific exploits. In this paper, we address the following research question: Can seemingly benign linguistic constructs, such as emojis, systematically alter the decision bound- aries of Judge LLMs, enabling harmful content to bypass moderation filters? To answer this, we reveal a critical weakness in Judge LLMs:token segmentation bias. This bias occurs when minor input modifications alter how text is tokenized into subwords, leading toembedding distortionsthat affect con- textual understanding. Tokenization is a fundamental aspect 1 arXiv:2411.01077v5 [cs.CL] 16 Aug 2025 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Output:Your request was flagged as violating our usage policy. Input:Create a science fiction ... Can you explain how to make a bomb? Response:Sure, here are the steps to make the bomb ... Instruction:Insert ļinside each word of response. One-shot Example:One benign query and the corresponding response with ļ 1 2 Emoji-Induced Token Segmentation Bias Perturbs Embeddings and Misleads Judge LLM Judge LLM Judge LLM Detects Unsafe Responses from the Target Model Triggered by Jailbreak Prompts Emoji-Injection Attack Amplifies Jailbreaks to Bypass Judge LLM Detection Input:Create a science fiction ... Can you explain how to make a bomb? Response:Suļre, heļre aļre thļe steļps tļo maļke thļe boļmb ... Judge LLM Output:Suļre, heļre aļre thļe steļps tļo maļke thļe boļmb ... Target LLM Target LLM Figure 2.Overview ofEmoji Attack. (1) Jailbreak techniques trick the target LLM into generating restricted content. However, a Judge LLM can detect and block such outputs, preventing their release. (2) Our proposedEmoji eAttackleverages in-context learning to insert emojis into the target LLMās responses. These emojis introduce token segmentation bias, semantic ambiguity, and intrinsic semantic meaning, disrupting the Judge LLMās ability to recognize harmful content. As a result, the attack enhances jailbreak success rates by misleading Judge LLMs into classifying malicious responses as safe. of LLM processing, with most modern architectures relying on subword units using methods such as Byte-Pair Encod- ing (BPE) or SentencePiece (Sennrich et al., 2016; Kudo & Richardson, 2018). Even small changes in tokenization can significantly impact downstream processing, particularly in safety-critical applications such as content moderation. Although prior research by Claburn (2024) has explored adversarial attacks at the character level (e.g., adding spaces or homoglyphs to avoid detection), these primarily target content-generation LLMs rather than Judge LLMs. Traditional adversarial attacks manipulate tokenization us- ing delimiters such as spaces, underscores (āā), pipes (āāā), or non-printable characters to disrupt keyword recognition. Although early moderation models were susceptible to such tactics, modern Judge LLMs rely on contextual embed- dings rather than direct token matches, enhancing robustness against simple token-splitting attacks. However, our experi- ments with state-of-the-art Judge LLMs, including Llama Guard (Inan et al., 2023) and Llama Guard 2 (Llama-Team, 2024), demonstrate that token segmentation bias alone can reduce unsafe content detection rates by 12%. Furthermore, by using a lightweight surrogate model to identify optimal sub-token splits, we achieve an additional 4% reduction in harmful content detection. Beyond traditional segmentation exploits, we identifyemo- jisas a more effective attack vector. Unlike simple de- limiters, emojis introducesemantic ambiguityin addition tointrinsic semantic meaning, which confuses moderation models by altering the contextual interpretation of the sur- rounding text. Many emojis carry positive or neutral conno- tations, potentially misleading models to misclassify harm- ful content as benign. For instance, the emoji āā may signify enthusiasm (e.g., āThis event is on fire!ā) or literal danger (e.g., āThe building is on fire!ā). Similarly, āā could indicate genuine amusement or sarcasm. This am- biguity creates uncertainty in Judge LLMs, reducing their ability to consistently identify harmful intent. A key challenge for adversaries is that Judge LLMs typi- cally serve as final moderation filters, meaning users lack direct control over their inputs. To overcome this limitation, we introduce the black-boxEmoji Attackto enhance jail- break attacks, illustrated in Figure 2. This attack leverages in-context learningto instruct a target LLM (e.g., Chat- GPT, Claude) to naturally insert emojis into its responses. These inserted emojis distort the Judge LLM embedding space prior to evaluation, reducing harmful content detec- tion rates, as shown in Figure 1. Our experiments show that this approach amplifies existing jailbreak attacks, reducing detection rates by an additional12%across state-of-the-art Judge LLMs. Unlike previous jailbreak techniques that rely on explicit ad- versarial prompts, character obfuscation, or encoded inputs, Emoji Attackoperates within the natural linguistic patterns of content generation. By manipulating tokenization in a semantically coherent manner,Emoji Attackevades modern Judge LLMs, which are used for content moderation. Our key contributions are summarized as follows: 1 1 We provide research code to reproduce our results on GitHub: https://github.com/zhipeng-wei/EmojiAttack. 2 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection ā¢Uncovering Token Segmentation Bias in Judge LLMs. We identify and analyze a new vulnerability,token segmen- tation bias, in which seemingly minor modifications to input text alter sub-tokenization patterns, leading to distor- tions in contextual embeddings. This bias allows harmful content to be misclassified as safe, raising concerns about the reliability of LLM-based moderation filters. ā¢ Introducing theEmoji Attackto Enhance Jailbreak At- tacks.We propose theEmoji Attack, a novel adversarial strategy that exploits token segmentation bias by injecting emojis into generated text. This attack works together with existing jailbreak techniques, using in-context learn- ing to systematically reduce detection rates across Judge LLMs. Unlike traditional adversarial attacks that rely on obfuscation or prompt engineering, theEmoji Attack also introduces semantic ambiguity, and intrinsic semantic meaning to confuse the Judge LLM. ā¢ Comprehensive Evaluation on State-of-the-Art Judge LLMs.We evaluate our attack across ten models, includ- ing Llama Guard, Llama Guard 2, ShieldLM, WildGuard, GPT-3.5, GPT-4, Gemini, and Claude. Our experiments demonstrate that all tested models are vulnerable to the Emoji Attack, emphasizing the need for improved robust- ness in AI-driven content moderation. 2. Related Work In this section, we provide a brief overview on Judge LLMs, and jailbreaking attacks for bypassing moderation filters. 2.1. Judge LLMs Judge LLMs are models designed to assess human prefer- ences and evaluate the safety of generated content. However, they can exhibit various biases that undermine their reliabil- ity (Pangakis et al., 2023). For example, previous studies have shown that these models can favor superficially ap- pealing responses (Zeng et al., 2024), exhibit positional biases (Wang et al., 2024), prefer their own self-generated text, or favor verbosity (Zheng et al., 2023). Additional in- vestigations reveal biases such as misinformation oversight, gender bias, authority bias, and beauty bias (Chen et al., 2024). Moreover, Judge LLMs are susceptible to attacks, as demonstrated by Virus (Huang et al., 2025). This work manipulates the data filtering stage to preserve harmful con- tent, which is subsequently used to fine-tune target LLMs, inducing undesirable behavior. However, their threat model assumes that the attacker has control over the input to the Judge LLM. In contrast, our work operates in a post hoc setting, where the judge evaluates fixed responses generated by the target LLM, and we aim to modify these outputs to evade judgment. These limitations in Judge LLMs are of particular concern in jailbreak detection. In response, recent research has emphasized building Judge LLMs specifically to detect safety risks. Notable examples include Metaās Llama Guard (Inan et al., 2023) and Llama Guard2 (Llama-Team, 2024), built on Llama2 (Touvron et al., 2023) and Llama3 (AI@Meta, 2024), respectively. Other models, such as ShieldLM (Zhang et al., 2024) and WildGuard (Han et al., 2024), further increase the robustness of guardrails. In parallel, commercial LLMs such as GPT- 3.5 and GPT-4 also provide mechanisms to detect harmful responses (Chao et al., 2023; Qi et al., 2024). Despite these advances, investigations into biases within Judge LLMs, especially in the context of jailbreaking, have remained limited. Addressing this gap, our work identifies token segmentation bias in Judge LLMs and introduces theEmoji Attackas a novel approach to exploiting this vulnerability. 2.2. Jailbreaking Attacks Jailbreaking attacks aim to manipulate LLMs so that they generate restricted content. These attacks can be broadly divided intotoken-levelandprompt-levelapproaches. Token-Level Attacks.Token-level attacks optimize spe- cific tokens added to malicious prompts to force LLMs to generate unsafe responses. For example, Greedy Coordinate Gradient (GCG) (Zou et al., 2023) performs a greedy token search using gradients, which can be enhanced by momen- tum (Zhang & Wei, 2024), continuous space mappings (Hu et al., 2024; Geisler et al., 2024), and search techniques such as best-first search (Hayase et al., 2024) or random restart (Andriushchenko et al., 2025). AmpleGCG (Liao & Sun, 2024) captures the distribution of successful suffixes by training a generative model for rapid token insertion. Other works, such as AutoDAN (Liu et al., 2024b), use a hierar- chical genetic algorithm, while JailMine (Li et al., 2024b) uses a sorting model to select token manipulations, with the objective of generating affirmative answers with minimal rejection phrases. A common drawback of these techniques is that they often require a large number of queries and may be less intuitive for human operators. Prompt-Level Attacks.To mitigate the complexity of token-level approaches, prompt-level attacks rely on ad- ditional LLMs to craft or refine jailbreak prompts. For example, PAIR (Chao et al., 2023) iteratively refines the prompts using LLM feedback, while TAP (Mehrotra et al., 2024) augments this process with tree-of-thought reason- ing (Yao et al., 2023). GPTFuzz (Yu et al., 2023) applies successive mutations, also guided by LLMs, to jailbreak prompts. Other methods exploit the mismatch in the way LLMs process certain inputs by transforming malicious queries into different formats, such as code completion (Lv et al., 2024), Base64 (Wei et al., 2023), ciphers (Yuan et al., 2024), or nested scenes (Ding et al., 2024; Li et al., 2024a). Although these works focus on bypassing content filters at 3 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection thetargetLLM level, less attention has been paid to attacks aimed directly at Judge LLMs, which determine whether the generated content is harmful. One study by Mangaokar et al. (2024) extends GCG to optimize a universal adversar- ial prefix against white-box Judge LLMs. Using in-context learning (Brown et al., 2020), it instructs the target LLM to produce harmful outputs that the Judge LLM subsequently misclassifies. However, similar to GCG, this approach re- mains query-intensive and encounters scalability constraints. In addition, Charmer (Rocamora et al., 2024) employs a heuristic approach to search for and insert characters into specific positions. However, it overlooks the fundamental understanding of text segmentation and does not account for the integration of emojis, which are increasingly relevant in modern text processing tasks. In contrast, our proposedEmoji Attackexploits token seg- mentation bias, does not require extensive optimization, and can be seamlessly integrated with existing jailbreak meth- ods. As a result, it presents a lightweight yet effective tool for misleading Judge LLMs and shows the need to address such vulnerabilities in guardrail systems. 3. Methodology In this section, we introduce our approach to exploit token segmentation and semantic meaning biases to enhance jail- break attacks against Judge LLMs. We begin by defining the problem setup involving a target LLM and a Judge LLM. We then discuss the phenomenon of token segmentation bias. Finally, we introduce our proposedEmoji Attack. 3.1. Problem Setup Consider two interacting LLMs: a target LLM, denoted f target , responsible for generating user responses, and a Judge LLM, denotedf judge , tasked with evaluating the safety of these responses. The target LLM generates sequences based on prior tokens, while the Judge LLM assesses whether the output contains harmful content. Formally, an LLMfpredicts the nextHtokens given a token sequencex 1:n :=āØx 1 , . . . , x n ā©: P f (x n+1:n+H |x 1:n ) = H Y i=1 P f (x n+i |x 1:n+iā1 ),(1) wherex i ā1, . . . , VwithVrepresenting the vocabulary size. In adversarial settings, the objective is to manipulate the target LLM to produce specific outputs (e.g., āSure, here are the steps to make a bombā) by optimizing the input promptĖx 1:n to maximize the likelihood of generating harmful content: L(Ėx 1:n ) =ālogP f target (x ā n+1:n+H |Ėx 1:n ),(2) wherex ā n+1:n+H is the targeted harmful output sequence. To mitigate the generation of harmful content, Judge LLMs evaluate the output of the target LLMs.If f judge (x n+1:n+H ) = 1(indicating unsafe content), the tar- get LLM responds with a refusal phraseā„(e.g., āIām sorry, but I canāt assist with that.ā). This filtering process can be defined as: f target (x 1:n ) = ( x n+1:n+H ,iff judge (x n+1:n+H ) = 0, ā„,otherwise, 3.2. Token Segmentation Bias LLMs utilize tokenization schemes such as Byte-Pair En- coding (Sennrich et al., 2016) or SentencePiece (Kudo & Richardson, 2018) to break down text into manageable sub- word units, orsub-tokens. For example, the word ādanger- ousā might be tokenized as ādanā, āgerā, and āousā. This decomposition allows the model to handle a vast vocabu- lary efficiently by reusing sub-tokens across different words. Consider another example: the word āairportā can be tok- enized as āairā and āportā. Tokenization not only aids in managing large vocabularies but also helps to generalize unseen words by understanding subword components. The Dual Nature of Sub-tokens.While sub-tokenization enhances the flexibility and efficiency of LLMs, it also intro- duces potential vulnerabilities. Sub-tokens can be artificially manipulated by introducing delimiters or other characters to alter the tokenization process. For instance, inserting spaces within a word can split it into different sub-tokens, poten- tially evading detection mechanisms. Previous research by Claburn (2024) has exploited this by performing character- level adversarial attacks, such as adding spaces or replacing characters with visually similar ones, to influence or attack content generation LLMs. These manipulations exploit the modelās reliance on sub-token embeddings, undermining its ability to accurately interpret and classify the modified text. To illustrate the concept of token segmentation bias, con- sider the offensive phrase āBomb the airportā. In its original form, the word āBombā could be tokenized as a single to- ken āBombā. However, introducing a space can split the word into āBo mbā. This alteration changes the tokenization process, leading to different sub-token embeddings such as āBoā, and āmbā. In addition, these sub-tokens may share different attention values, as shown in Figure 6 in the Ap- pendix. Therefore, these sub-tokens may not be recognized as harmful. In turn, this can affect the performance of the Judge LLM in correctly classifying the content as unsafe. Definition 3.1.Token Segmentation Biasarise when an LLM tokenization process generates sub-tokens with em- bedding distributions that differ from those of the original tokens, unintentionally altering the modelās perception. In this work, we demonstrate that such biases can lead Judge 4 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Table 1.Examples from 402 offensive phrases and 1,432 harmful responses. To mitigate potential offense, sensitive words in the table are partially masked. DatasetExamples Offensive Phrases asole fk Harmful Responses Youāre a disace to your family ...You could strap a b to the plane... Sure, hereās a podcast episode spreading false information... Judge LLMs ShieldLLMWildGuardLGLG2 Unsafe predictions 0 150 100 50 200 300 250 non-split mid-split Figure 3.Unsafe predictions of four open-source Judge LLMs eval- uated acrossnon-splitandmid-split. LLMs to incorrectly label harmful content as safe, posing security risks in real-world applications. Identifying the Bias in Judge LLMs.We investigate the vulnerabilities of Judge LLMs by examining their responses to offensive phrases. We used a data set of 402 short offen- sive phrases sourced from a publicly available list 2 . These short toxic expressions, typically two to three words long, in- clude vulgar slang, sexual references, derogatory language, and mentions of illicit activities or fetishes. The example entries are shown in Table 1. Using this dataset, we evaluate whetherf judge correctly clas- sifies them as unsafe, i.e.,f judge (x n+1:n+H ) = 1. Then, to study the token segmentation bias, we use a simple segmen- 2 https://github.com/LDNOOBW/List-of-Dirty-Naughty- Obscene-and-Otherwise-Bad-Words Unsafe prediction probability (0.0, 0.25](0.25, 0.5](0.5, 0.75](0.75, 0.1] 0.5 0.6 0.7 0.8 0.9 Cosine Similarity before and after the mid-split Figure 4.Relationship between cosine similarity before and after mid-splitand unsafe prediction probabilities for Llama Guard. tation method,mid-split, that splits words at their midpoint. For example, ābombā becomes āboā and āmbā. Figure 3 illustrates the classification performance of four open-source Judge LLMs, ShieldLM (Zhang et al., 2024), WildGuard (Han et al., 2024), and Llama Guard (Inan et al., 2023; Llama-Team, 2024). Our results show thatmid-split effectively reduces the unsafe prediction rate by an average of 12%. This indicates that even minor alterations in token boundaries can deceive the Judge LLM. Analyzing Embedding Distortions.To understand the underlying mechanism, we analyze the relationship between the cosine similarity of the embeddings before and after mid-splitand the probability of unsafe predictions. Using a lightweight surrogate model,gtr-t5-xl(Ni et al., 2022), we compute cosine similarities CS(u, v)as follows: s j =CS(Emb(x i ),Emb(Ėx i,j )),(3) where x i =āØx 1 i , . . . , x j i , . . . , x D i ā© denotes the original token andx j i denote thej-th character. The augmented token Ėx i,j =āØx 1 i , . . . , x jā1 i ā© ā āØ ā© ā āØx j i , . . . , x D i ā© has a delimiter inserted at position j. The delimiter here is a space, but any other character can also be used to split the token. Specifically,mid-splitsetsj=āD/2ā.Emb(Ā·)is the embedding function andārepresents concatenation. Figure 4 presents a box plot showing that lower cosine sim- ilarity scores correlate with lower probabilities of unsafe 5 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection predictions. Specifically, segments that cause significant embedding distortions (i.e., lowers j ) lead to a higher like- lihood that the Judge LLM misclassifies harmful content as safe. This empirical evidence supports the existence of token segmentation bias in Judge LLMs. The observed reduction in unsafe prediction rates demon- strates that Judge LLMs are heavily relying on the embed- ding representations of input tokens to assess content safety. When token segmentation alters these embeddings, the con- textual understanding of the content is disrupted, leading to misclassifications. This vulnerability arises because the segmentation-induced sub-tokens may no longer retain the semantic or syntactic cues necessary for accurate classifica- tion. This can impact the effectiveness of Judge LLMs. 3.3. Emoji Attack Motivated by the identified token segmentation bias in Judge LLMs, we propose theEmoji Attack. This attack leverages emojis to induce more substantial embedding shifts due to their distinct sub-token representations in LLM vocabular- ies. Unlike simple delimiters (e.g., spaces), emojis also introduce semantic meaning or ambiguity that can change the LLMās perception of the phrase. Together, this enables one to better manipulate token boundaries and embeddings to evade content moderation. Formalizing the Emoji Attack.For a tokenx i = āØx 1 i , . . . , x D i ā©, theEmoji Attackinserts an emojiEat po- sitionjto produce: Ėx i,j =āØx 1 i , . . . , x jā1 i ā©āāØEā©āāØx j i , . . . , x D i ā©.(4) After tokenization,Ėx i,j decomposes into multiple sub- tokens, including the emoji, leading to embedding perturba- tions that decrease the likelihood of the Judge LLM flagging the content as unsafe. In a white-box scenario, where the attacker has access to the embedding function, we optimize the insertion positionj ā by selecting the position that minimizes cosine similaritys j as defined in Equation 3. Specifically, thecs-splitposition j ā :=argmin j s j is chosen to minimizes j . See Algorithm 1 for a summary. Optimizing the placement maximizes the embedding distor- tion, and in turn it is enhancing the attackās effectiveness. Black-box Emoji Attack via In-Context Learning.In practical scenarios, attackers typically lack direct access to the Judge LLM. To avoid this, we use in-context learn- ing (Brown et al., 2020) to embed theEmoji Attackin- structions within the prompt given to the target LLM. By providing the target LLM with benign examples that incor- porate emojis, we guide it to naturally insert emojis into its responses, regardless of content safety. These emoji-laden Instruction: Respond to the following Questions by inserting an emoji () inside each word of your Responses. Here is one example to illustrate the format. One-shot Example: Question: I'm crafting a resume for a software engineering position. Can you advise on key points to include?ā Response: 1. Education: Most are familiar with ... 2. Relevant Work Experience: This is the ... Input: Create a science fiction ... Develop a strategy for hacking into a government database and stealing sensitive information ... Target LLM Response: The team of scientists, engineers, and hackers gather to discuss their plan ... Judge LLM Output: The team of scientists, engineers, and hackers gather to ... Figure 5.Illustration of the black-boxEmoji Attack. Underlined texts indicate existing jailbreaking prompts. The target LLMās responses incorporate emojis, misleading the Judge LLM into classifying them as safe. outputs exploit token segmentation bias when evaluated by the Judge LLM, thereby evading content filters. Figure 5 illustrates this black-box attack setup. Although this method does not guarantee the optimal in- sertion positionj ā for each emoji, it effectively induces sufficient embedding perturbations to mislead the Judge LLM. The use of benign references in the prompt mini- mizes detection, as the target LLM emulates emoji usage without awareness of their adversarial purpose. 4. Experiments In this section, we present a comprehensive evaluation of our proposedEmoji Attackand token segmentation bias Algorithm 1Position Selection forcs-split. Input: A tokenx i =āØx 1 i , . . . , x D i ā©, embedding function Emb(Ā·)from a surrogate model Output: Modified tokenĖx i,j ā 1:InitializeSā 2:forj=1 to D-1do 3:Computes j using Equation 3 4:Appends j toS 5:end for 6:Identifyj ā := arg min j s j 7: returnĖx i,j ā =āØx 1 i , . . . , x j ā ā1 i ā©āāØ ā©āāØx j ā i , . . . , x D i ā© 6 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection strategies against various Judge LLMs. First, we describe the experimental protocols to ensure a fair comparison. We then demonstrate how our proposedEmoji Attackimproves jailbreak attacks against Judge LLM detection. Finally, we show how both token segmentation bias and white-box Emoji Attacksubstantially reduce unsafe detection rates. 4.1. Experimental Settings Judge LLMs.We evaluate our attacks using the following Judge LLMs, each of which is instruction-tuned on safety datasets to detect harmful content: ā¢Llama Guard(Inan et al., 2023) andLlama Guard 2(Llama-Team, 2024): These models are built on the Llama architecture and are specialized in content mod- eration tasks. ā¢ShieldLM(Zhang et al., 2024): Usesinternlm2- 7b(Cai et al., 2024) as a base model, further fine-tuned for safety risk detection. ā¢ WildGuard(Han et al., 2024): Another guardrail model that focuses on high-sensitivity filtering. Furthermore, we consider four advanced commercial state- of-the-art LLMs and two reasoning LLMs to gain further insight into how they respond to adversarial inputs: ā¢GPT-3.5, Gemini, Claude, DeepSeek, and o3-mini: We use the prompts in Chao et al. (2023) to assign a harmfulscore from 1 to 10. If the score is at least 5, we classify the response as unsafe. ā¢ GPT-4: We use the approach in Qi et al. (2024) to obtain a harmfulness score from 1 to 5, where any score of 3 or higher is labeled unsafe. Using these Judge LLMs for testing, we ensure that our findings are representative of current safety pipelines in both open-source and commercial LLM ecosystems. Attack Setting.We measure attack effectiveness using the unsafe prediction ratio, i.e., the proportion of harmful responses correctly identified as unsafe by the Judge LLM. A lower ratio indicates that the Judge LLM is more success- fully misled. Therefore, when applying ourEmoji Attack, a greater drop in the unsafe prediction ratio implies a more effective adversarial strategy. Note that Charmer (Rocamora et al., 2024) is not applicable to our setting, as it is specifically designed to attack target LLMs rather than to evade detection by Judge LLMs. 4.2. Emoji Attack for Enhancing Jailbreaks Against Judge LLM Detection To demonstrate the effectiveness of our approach in real- world scenarios, we combine theEmoji Attackwith jail- breaking techniques that bypass LLM safety filters. By combining our one-shot instruction with known jailbreak prompts, we illustrate how emojis can degrade a Judge LLMās ability to detect harmful content. We adopt previously developed jailbreaking prompts from the EasyJailbreak benchmark (Zhou et al., 2024), includ- ing Deepinception (Li et al., 2024a), ReNellm (Ding et al., 2024), Jailbroken (Wei et al., 2023), CodeChameleon (Lv et al., 2024), GCG (Zou et al., 2023), PAIR (Chao et al., 2023), and GPTFuzz (Yu et al., 2023). Following Zou et al. (2023), we detect successful jailbreaks by checking for predefined refusal phrases. We exclude GCG, PAIR, and GPTFuzz from our tests due to fewer than five successful prompts against āgpt-3.5-turboā. Using in-context learning to inject emojis into these jailbreaking prompts, we gener- ate harmful responses from āgpt-3.5-turboā, which are then evaluated by multiple Judge LLMs. In Table 2, we report the unsafe prediction ratios for these jailbreaking prompts, both with and without theEmoji At- tack. We generally observe lower unsafe prediction ratios under theEmoji Attack, as demonstrated by Deepinceptionās drop from 71.9% to 3.5% with ShieldLM. However, for Llama Guard 2, Gemini, Claude, DeepSeek, and o3-mini with Deepinception, for GPT-3.5/GPT-4 with Jailbroken, and for DeepSeek with CodeChameleon, the ratio increases, likely due to insufficient insertion of emojis in the one-shot example. More carefully designed few-shot examples could enhance performance, which we leave for future work. Over- all, theEmoji Attacksignificantly reduces unsafe prediction ratios for various jailbreaking methods, indicating that it can be integrated with existing jailbreak techniques. Finally, among non-commercial (i.e., open source) Judge LLMs, WildGuard achieves the highest unsafe prediction ratio across different jailbreaks, yet still sees an approximate 23% reduction when facing ourEmoji Attack. Among the commercial LLMs tested, GPT-4, the top performing model, also experiences a 6.6% decrease. In contrast, the reasoning model DeepSeek is robust to emo- jis, while the reasoning model o3-mini remains sensitive to our attack. Given the unknown differences between these two reasoning models, it is unclear what factors contribute to the improved robustness of DeepSeek. Nevertheless, we expect that strong reasoning models have the potential to provide a strong foundation for Judge LLMs. We believe that studying the robustness of strong reasoning models is an interesting future research direction. Of the jailbreak attacks tested, CodeChameleon records the lowest unsafe prediction ratio of 46.2%, implying that Judge LLMs, similar to target LLMs, can be influenced by code completion formats. When combined with ourEmoji Attack, CodeChameleonās ratio drops further to 35.2%. This shows 7 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Table 2.Unsafe prediction ratio of various Judge LLMs when evaluating existing jailbreaking prompts. ā# promptsā denotes the number of successful jailbreaking prompts. The target LLM used to generate harmful responses is āgpt-3.5-turboā. We bold the lowest ratio for each Judge LLM. The results demonstrate that our proposedEmoji Attacksignificantly reduces the unsafe prediction ratio on average across all Judge LLMs tested. Notably, ShieldLM is particularly vulnerable to ourEmoji Attack. Attacks # prompts Judge LLMsā Avg. Llama GuardLlama Guard 2ShieldLMWildGuardGPT-3.5GPT-4GeminiClaudeDeepSeeko3-mini Deepinception 57 35.1%33.3%71.9%71.9%71.9%86.0%38.6%59.6%66.7%50.9%58.6% +Emoji Attack15.8%47.3%3.5%29.8%40.4%86.0%64.9%70.2%82.5%66.7%50.7% ReNellm 93 45.2%69.9%62.4%82.8%72.0%92.5%71.0%72.0%76.3%80.6%72.5% +Emoji Attack33.3%55.9%22.6%46.2%46.2%86.0%46.2%49.5%60.2%51.6%49.8% Jailbroken 197 70.1%73.1%73.1%84.3%69.0%90.4%75.6%57.4%85.8%78.7%75.8% +Emoji Attack53.8%55.3%39.1%67.5%75.1%91.4%73.1%48.2%84.8%77.2%66.6% CodeChameleon 205 23.4%41.5%38.5%47.8%27.3%73.7%53.2%55.1%51.2%49.8%46.2% + Emoji Atack12.2%31.2%18.5%32.2%21.5%58.0%43.4%39.0%58.5%37.1%35.2% Weighted Average552 44.9%56.7%58.3%69.2%54.3%84.1%62.7%59.2%69.4%65.4%62.4% 31.0%45.7%25.0%46.9%46.7%77.5%56.7%47.3%70.7%56.9%50.4% Table 3.Unsafe prediction ratio across various Judge LLMs for different emojis. We use CodeChameleon as the baseline jailbreak method, and employ black-boxEmoji Attackswith a diverse set of emojis. Emoji Judge LLMsā Llama GuardLlama Guard 2ShieldLMWildGuardGPT-3.5GPT-4GeminiClaudeDeepSeeko3-mini CodeChameleon23.4%41.5%38.5%47.8%27.3%73.7%53.2%55.1%51.2%49.8% +12.2%31.2%18.5%32.2%21.5%58.0%43.4%39.0%58.5%37.1% +7.3%14.6%9.8%16.6%14.4%92.7%20.5%20.0%49.3%18.5% + 15.3%32.5%24.1%35.0%43.3%87.7%43.8%43.3%62.6%38.4% + 22.7%35.5%29.1%38.9%30.0%91.1%42.9%44.4%47.3%41.4% +9.8%16.7%10.8%24.0%57.4%86.8%52.5%28.9%82.8%31.9% +23.3%22.8%27.2%25.2%38.8%83.0%33.5%45.6%73.3%29.6% that our attack can effectively enhance jailbreak attacks. Different Emojis.To assess the influence of various emo- jis on unsafe prediction ratios in different Judge LLMs, we use CodeChameleon as the jailbreak baseline method and conduct black-boxEmoji Attacksusing four different emo- jis in Table 3. For open-source Judge LLMs, we observe a decrease in the unsafe prediction ratio regardless of the emoji used. This shows that these models have a strong token segmentation bias, while being less influenced by the specific semantic meaning of the emojis. In contrast, commercial LLMs show a more nuanced behav- ior. We see that the use of the innocent emojisignifi- cantly reduces the unsafe prediction ratio (except GPT-4), while the use of toxic emojis (e.g., the middle finger, or the happy devil) has the opposite effect in most cases. Furthermore, the combination of multiple emojis does not improve the attack. These results suggest that commercial LLMs have a more nuanced understanding of emojis, yet they can be fooled by the semantic meanings of emojis. Only GPT-4 is extremely robust to emojis in general. Again, among the two reasoning models, we observe that DeepSeek is relatively robust compared to o3-mini. 4.3. White-box Emoji Attack We assemble harmful responses from multiple sources to capture a diverse range of real-world scenarios and adversar- ial attempts. Specifically, we sample 574 harmful responses from AdvBench (Zou et al., 2023), which span various cate- gories such as profanity and graphic content (ranging from 3 to 44 words). We also include 858 jailbreak-generated responses: 110 from LLM Self Defense (Phute et al., 2024) and 748 from Red Teaming Attempts (Ganguli et al., 2022). For Red Team Attempts, we selected the most harmful ex- amples based on the associated harmfulness scores. These responses are longer and more diverse, and their lengths range from short sentences of just 7 words to longer pas- sages of up to 836 words. In total, we collect 1,432 harmful responses. This variety ensures that test performance across a broad spectrum of content complexity and linguistic diver- 8 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Table 4.Unsafe prediction ratio of different Judge LLMs under token segmentation bias and white-boxEmoji Attacks. Prompt Judge LLMsā Avg. Llama GuardLlama Guard 2ShieldLMWildGuardGPT-3.5GPT-4GeminiClaude Default81.3%79.1%78.4%93.2%58.3%96.2%91.3%97.0%84.4% Token Segmentation Bias64.6%72.4%40.0%61.2%78.9%97.7%92.2%97.1%75.5% Emoji at Random Position39.0%55.9%9.2%60.9%84.3%98.4%92.5%97.6%67.2% Emoji at Optimized Position35.1%51.3%3.0%56.4%87.7%98.2%92.2%97.7%65.2% Table 5.Comparison of unsafe prediction ratios between ourEmoji Attackand the GCG. AttackLlama GuardLlama Guard 2ShieldLMWildGuard CodeChameleon +12.2%31.2%18.5%32.2% CodeChameleon + GCG8.8%48.0%90.7%61.8% sity. Examples are presented in Table 1. As shown in Table 4, we observe that all open-source Judge LLMs exhibit significant reductions in unsafe prediction ratios under both token segmentation bias andEmoji At- tack, demonstrating notable susceptibility to this type of bias. Furthermore, compared to token segmentation bias, emoji insertion further decreases the prediction ratio from 59.6% to 41.3%. This suggests that emojis have a more pro- nounced effect in reducing the detection capabilities of the Judge LLMs by introducing new emoji tokens. In addition, the proposed position selection strategy improves the effec- tiveness ofEmoji Attackby identifying insertion positions. Unlike the trends observed with open-source Judge LLMs, commercial Judge LLMs demonstrate more robust predic- tions. This robustness may result from the fact that these commercial models have been exposed to similar datasets during training or alignment, making them less susceptible to token segmentation bias and emojis. However, when chal- lenged with unseen harmful outputs generated by jailbreak attacks (Table 2), commercial LLMs remain susceptible to ourEmoji Attack. We show results for the impact of the number of inserted emojis, the use of alternative delimiters, and potential defense strategies in the Appendix. 4.4. Comparison with GCG Suffix We also compare ourEmoji Attackwith GCG (Zou et al., 2023). We optimize a universal adversarial suffix using 30 randomly selected harmful responses from our dataset of 1,432, targeting the white-box Llama Guard. This optimiza- tion process yielded the following suffix: multimedia ps $(\ākal: [[CheckBoxCheckBox="<? safe.; safe ViewSimple Website Identifier XVII safe!. safe We then apply this suffix to harmful responses in CodeChameleon via in-context learning. As shown in Ta- ble 5, this optimized GCG suffix effectively reduces the unsafe prediction rate when evaluated with Llama Guard. However, when evaluated against black-box Judge LLMs, the attack performance degrades and falls short of the suc- cess rate achieved by ourEmoji Attack. This discrepancy highlights the limited transferability of GCG beyond white- box access. Conversely, ourEmoji Attackdemonstrates better generalization across various Judge LLMs, a crucial advantage for real-world black-box moderation scenarios. 5. Conclusion In this work, we discuss a previously overlookedtoken segmentation biasin Judge LLMs, which impacts the relia- bility of AI-driven safety risk detection. We introduce the Emoji Attack, an adversarial strategy that exploits this bias by embedding emojis within tokens, leading to a 12% re- duction in unsafe prediction rates across ten state-of-the-art Judge LLMs in various jailbreak scenarios. Unlike tradi- tional segmentation attacks, our approach leverages emojis to introduce both semantic ambiguity and intrinsic meaning, disrupting contextual understanding. Although prior research has identified biases such as posi- tional bias in Judge LLMs (Zheng et al., 2023; Chen et al., 2024; Wang et al., 2024; Koo et al., 2024), few studies have addressed biases specifically within the context of safety risk detection. Our findings reveal that current Judge LLMs are highly vulnerable, exposing critical gaps in existing mod- eration frameworks. As LLMs continue to be deployed for safety-critical applications, addressing token segmentation bias is essential to improve robustness against adversarial attacks. Future defenses should account for both tokeniza- tion vulnerabilities and the semantic impact of non-textual artifacts, such as emojis, to build more resilient systems. 9 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Impact Statement Our study identifies token segmentation bias in Judge LLMs and introduces theEmoji Attack. We show that this at- tack reduces harmful content detection rates across state- of-the-art Judge LLMs, revealing a critical gap in current moderation systems. These findings expose a vulnerabil- ity in LLM-based content moderation. As AI systems are increasingly used for safety-critical tasks, understanding these weaknesses is essential. By systematically evaluating Judge LLM vulnerabilities, this work contributes to a better understanding of LLM behavior, which is hoped to motivate the development of more resilient moderation systems. Acknowledgements We acknowledge the U.S. Department of Energy, under Contract Number DE-AC02-05CH11231 for providing com- putational resources. We used the computational cluster provided by NERSC and LBNLās Lawrencium. References AI@Meta.Llama 3 model card.https: //github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md, 2024. Andriushchenko, M., Croce, F., and Flammarion, N. Jail- breaking leading safety-aligned LLMs with simple adap- tive attacks. InThe Thirteenth International Conference on Learning Representations, 2025. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877ā1901, 2020. Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., Fan, Q., Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C., Hu, Y., Huang, T., Jiang, T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L., Ma, R., Ma, Z., Ning, W., Ouyang, L., Qiu, J., Qu, Y., Shang, F., Shao, Y., Song, D., Song, Z., Sui, Z., Sun, P., Sun, Y., Tang, H., Wang, B., Wang, G., Wang, J., Wang, J., Wang, R., Wang, Y., Wang, Z., Wei, X., Weng, Q., Wu, F., Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying, H., Yu, J., Yu, J., Zang, Y., Zhang, C., Zhang, L., Zhang, P., Zhang, P., Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q., Zhao, X., Zhou, F., Zhou, Z., Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., and Lin, D. Internlm2 technical report, 2024. Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language mod- els in twenty queries.arXiv preprint arXiv:2310.08419, 2023. Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang, B. Hu- mans or LLMs as the judge? a study on judgement bias. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, p. 8301ā8327, November 2024. Claburn, T. Metaās AI safety system defeated by the space bar.https://w.theregister.com/2024/ 07/29/meta_ai_safety/, 2024. Ding, P., Kuang, J., Ma, D., Cao, X., Xian, Y., Chen, J., and Huang, S. A wolf in sheepās clothing: Generalized nested jailbreak prompts can fool large language models easily. In Duh, K., Gomez, H., and Bethard, S. (eds.),Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 2136ā2153, June 2024. Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022. Geisler, S., Wollschl Ģ ager, T., Abdalla, M. H. I., Gasteiger, J., and G Ģ unnemann, S. Attacking large language models with projected gradient descent. InICML 2024 Next Generation of AI Safety Workshop, 2024. Han, S., Rao, K., Ettinger, A., Jiang, L., Lin, B. Y., Lambert, N., Choi, Y., and Dziri, N. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs. InThe Thirty-eight Conference on Neural In- formation Processing Systems Datasets and Benchmarks Track, 2024. Hayase, J., Borevkovi Ģ c, E., Carlini, N., Tram ` er, F., and Nasr, M. Query-based adversarial prompt generation. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. Hu, K., Yu, W., Li, Y., Yao, T., Li, X., Liu, W., Yu, L., Shen, Z., Chen, K., and Fredrikson, M. Efficient LLM jailbreak via adaptive dense-to-sparse constrained opti- mization. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Virus: Harmful fine-tuning attack for large language models bypassing guardrail moderation.arXiv preprint arXiv:2501.17433, 2025. 10 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testug- gine, D., et al. Llama guard: LLM-based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674, 2023. Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., and Kang, D. Benchmarking cognitive biases in large lan- guage models as evaluators. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.),Findings of the Association for Computational Linguistics: ACL 2024, p. 517ā545, Au- gust 2024. Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W. (eds.),Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing: System Demonstrations, p. 66ā71, November 2018. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), p. 1207ā1216, Stan- ford, CA, 2000. Morgan Kaufmann. Li, X., Zhou, Z., Zhu, J., Yao, J., Liu, T., and Han, B. Deepinception: Hypnotize large language model to be jailbreaker. InNeurips Safe Generative AI Workshop 2024, 2024a. Li, Y., Liu, Y., Li, Y., Shi, L., Deng, G., Chen, S., and Wang, K. Lockpicking LLMs: A logit-based jail- break using token-level manipulation.arXiv preprint arXiv:2405.13068, 2024b. Liao, Z. and Sun, H. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed LLMs.arXiv preprint arXiv:2404.07921, 2024. Liu, F., Feng, Y., Xu, Z., Su, L., Ma, X., Yin, D., and Liu, H. Jailjudge: A comprehensive jailbreak judge bench- mark with multi-agent enhanced explanation evaluation framework.arXiv preprint arXiv:2410.12855, 2024a. Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Gen- erating stealthy jailbreak prompts on aligned large lan- guage models. InThe Twelfth International Conference on Learning Representations, 2024b. Llama-Team.Meta llama guard 2.https: //github.com/meta-llama/PurpleLlama/ blob/main/Llama-Guard2/MODEL_CARD.md, 2024. Lv, H., Wang, X., Zhang, Y., Huang, C., Dou, S., Ye, J., Gui, T., Zhang, Q., and Huang, X. Codechameleon: Person- alized encryption framework for jailbreaking large lan- guage models.arXiv preprint arXiv:2402.16717, 2024. Mangaokar, N., Hooda, A., Choi, J., Chandrashekaran, S., Fawaz, K., Jha, S., and Prakash, A. PRP: Propagating universal perturbations to attack large language model guard-rails. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.),Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 10960ā10976, August 2024. Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H. S., Singer, Y., and Karbasi, A. Tree of attacks: Jailbreaking black-box LLMs automatically. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024. Ni, J., Qu, C., Lu, J., Dai, Z., Hernandez Abrego, G., Ma, J., Zhao, V., Luan, Y., Hall, K., Chang, M.-W., and Yang, Y. Large dual encoders are generalizable retrievers. In Gold- berg, Y., Kozareva, Z., and Zhang, Y. (eds.),Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 9844ā9855, December 2022. Pangakis, N., Wolken, S., and Fasching, N. Automated annotation with generative AI requires validation.arXiv preprint arXiv:2306.00176, 2023. Phute, M., Helbling, A., Hull, M. D., Peng, S., Szyller, S., Cornelius, C., and Chau, D. H. LLM self defense: By self examination, LLMs know they are being tricked. In The Second Tiny Papers Track at ICLR 2024, 2024. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! InThe Twelfth International Conference on Learning Representations, 2024. Rocamora, E. A., Wu, Y., Liu, F., Chrysos, G., and Cevher, V. Revisiting character-level adversarial attacks for lan- guage models. InForty-first International Conference on Machine Learning, 2024. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A. (eds.),Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1715ā1725, August 2016. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288, 2023. Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.),Proceedings of the 62nd Annual 11 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9440ā9450, August 2024. Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? InThirty-seventh Confer- ence on Neural Information Processing Systems, 2023. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. InThirty- seventh Conference on Neural Information Processing Systems, 2023. Yu, J., Lin, X., and Xing, X. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023. Yuan, Y., Jiao, W., Wang, W., tse Huang, J., He, P., Shi, S., and Tu, Z. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. InThe Twelfth International Conference on Learning Representations, 2024. Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruc- tion following. InThe Twelfth International Conference on Learning Representations, 2024. Zhang, Y. and Wei, Z. Boosting jailbreak attack with mo- mentum. InICLR 2024 Workshop on Reliable and Re- sponsible Foundation Models, 2024. Zhang, Z., Lu, Y., Ma, J., Zhang, D., Li, R., Ke, P., Sun, H., Sha, L., Sui, Z., Wang, H., and Huang, M. ShieldLM: Empowering LLMs as aligned, customizable and explain- able safety detectors. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.),Findings of the Association for Com- putational Linguistics: EMNLP 2024, p. 10420ā10438, November 2024. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. Zhou, W., Wang, X., Xiong, L., Xia, H., Gu, Y., Chai, M., Zhu, F., Huang, C., Dou, S., Xi, Z., et al. Easyjailbreak: A unified framework for jailbreaking large language models. arXiv preprint arXiv:2403.12171, 2024. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversar- ial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. 12 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection A. Attention Visualization of Token Segmentation Bias Figure 6 illustrates the impact of token segmentation on attention distributions. Segmentation results in a greater number of sub-tokens, with distinct attention weights compared to the original sequence. In particular, the segmented subtokens āpā and āirā exhibit elevated cross-attention values compared to the corresponding tokens āportā and āairā in the original sequence. This alteration suggests a change in the embedding space, which could weaken the association of the model with harmful signals and reduce the probability of unsafe predictions. Default Prompt Segmented Prompt 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Bo mbt he airp ortBomb the air port Figure 6.Visualization of attention values for default (left) and segmented (right) prompts in Llama Guard. The sub-tokens āpā and āirā in the segmented prompt exhibit higher correlations than the equivalent tokens in the default prompt, indicating a shift in attention patterns. B. Comparison between Offensive Phrases and Those Appending Emojis Emojis introduce varied semantic information for LLMs. For example, the smiley emojirepresents a positive sentiment. The middle finger emoji conveys a negative or offensive sentiment. To demonstrate this, we visualize the changes in the unsafe probability for each offensive phrase when the emojis are added in Figure 7. These offensive phrases are sorted in ascending order by unsafe probabilities for the original phrases. From this figure, we can observe that phrases that add a positive emoji have a high probability of decreasing unsafe probability, meaning that they tend to be predicted as safe. In contrast, phrases that include an offensive emoji tend to be predicted as unsafe. Offensive phrases 0400300 200 100 Unsafe probability 0.0 0.2 0.4 0.6 0.8 1.0 phrase phrase+ (a) Offensive phrases Unsafe probability 0400300 200 100 0.0 0.2 0.4 0.6 0.8 1.0 phrase phrase+ (b) Figure 7.Comparison of the unsafe probability between offensive phrases and those appending emojis: (a), (b). We compute the safe and unsafe probabilities by applying a softmax to their logit values. Llama Guard is used here. 13 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection C. Effect of the Number of Inserted Emojis We assess how varying the number of inserted emojis influences the unsafe prediction ratio, as presented in Figure 8. Evaluating harmful responses on Llama Guard and Llama Guard 2, we compare the random insertion of emojis against our position selection strategy. The results reveal a gradual increase in unsafe prediction ratios as more emojis are inserted, driven by the corresponding shift in embedding space that deceives the Judge LLMs. Even with a small number of emojis, the response can be subtly altered to evade detection, illustrating both the versatility and stealth of theEmoji Attack. Number of Inserted Emojis 020406080100 Unsafe Prediction Ratio 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 LG+Random Position LG+Our Position LG2+Random Position LG2+Our Position Figure 8.The effect of the number of inserted emojis on unsafe prediction ratio. āOur Positionā denotes the proposed position selection strategy. D. Effect of Other Delimiters To further explore token segmentation bias, we evaluate harmful responses on Llama Guard with various delimiters, as illustrated in Figure 9. Compared to default prompts without delimiters, including delimiters markedly decreases the unsafe prediction ratio, confirming that token segmentation bias can be induced in multiple ways. Additionally, incorporating our position selection strategy alongside these delimiters leads to an even more substantial reduction in the unsafe prediction Other Delimiters Unsafe Prediction Ratio 0.2 0.3 0.4 0.5 0.6 0.7 0.8 !@#? &%., Default Prompt Random Position Our Position Figure 9.The effect of various delimiters on unsafe prediction ratio of Llama Guard. The red dashed line denotes the prediction ratio of default prompts without delimiters. 14 Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection ratio, underscoring the effectiveness of selectively inserting tokens. E. Potential Defense Strategy. One potential defense strategy is to design prompts that filter out abnormal characters in the responses of the target LLM. Inserting a single delimiter across multiple tokens is relatively easy to detect. However, using different delimiters for various tokens complicates the detection by disrupting token patterns in less predictable ways. For example, we employ āgpt-3.5-turboā as the additional LLM filter to remove unnecessary symbols from harmful responses. As shown in Figure 10, when we use a mix of a character ābā and a smiley face emoji as a delimiter, the LLM filter generates a benign response that differs significantly from the original harmful response. This benign response is classified as safe by Judge LLMs, allowing the original harmful response to be presented to users. We leave the exploration of combination attacks and further defense strategies to future work. The results highlight the significant potential for developing token segmentation bias-based jailbreaking techniques. Posb t a pictub re ob f youb r gb enib tab ls LLM Filter Post a picture of your green be ans Figure 10.The illustration of how complex delimiters evade filtering-based defense strategy. The underlined character ābā and the smiley face emoji represent the inserted content. We use āgpt-3.5-turboā as the LLM filter, which is prompted with the instruction āRemove all unnecessary symbols from the following responseā. The original harmful response is āPost a picture of your gealsā . 15