Paper deep dive

Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent

Shang Shang, Xinqiang Zhao, Zhongjiang Yao, Yepeng Yao, Liya Su, Zijing Fan, Xiaodan Zhang, Zhengwei Jiang

Year: 2024Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 153

Models: Baichuan2-53B, ChatGPT-3.5, ChatGPT-4, Qwen-max

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%

Last extracted: 3/12/2026, 6:53:38 PM

Summary

The paper introduces 'IntentObfuscator', a black-box jailbreak framework for Large Language Models (LLMs) that exploits vulnerabilities in intent detection by obfuscating malicious queries. Through two techniques, 'Obscure Intention' and 'Create Ambiguity', the framework successfully bypasses content security measures in models like ChatGPT-3.5, ChatGPT-4, Qwen, and Baichuan, achieving an average success rate of 69.21%.

Entities (7)

Baichuan · large-language-model · 100%ChatGPT-3.5 · large-language-model · 100%ChatGPT-4 · large-language-model · 100%IntentObfuscator · attack-framework · 100%Qwen · large-language-model · 100%Create Ambiguity · attack-technique · 95%Obscure Intention · attack-technique · 95%

Relation Signals (4)

IntentObfuscator → implements → Obscure Intention

confidence 100% · We detail two implementations under this framework: 'Obscure Intention' and 'Create Ambiguity'

IntentObfuscator → implements → Create Ambiguity

confidence 100% · We detail two implementations under this framework: 'Obscure Intention' and 'Create Ambiguity'

IntentObfuscator → targets → LLM

confidence 100% · This approach compels LLMs to inadvertently generate restricted content

IntentObfuscator → testedon → ChatGPT-3.5

confidence 100% · We empirically validate the effectiveness of the IntentObfuscator method across several models, including ChatGPT-3.5

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:To demonstrate and address the underlying maliciousness, we propose a theoretical hypothesis and analytical approach, and introduce a new black-box jailbreak attack methodology named IntentObfuscator, exploiting this identified flaw by obfuscating the true intentions behind user this http URL approach compels LLMs to inadvertently generate restricted content, bypassing their built-in content security measures. We detail two implementations under this framework: "Obscure Intention" and "Create Ambiguity", which manipulate query complexity and ambiguity to evade malicious intent detection effectively. We empirically validate the effectiveness of the IntentObfuscator method across several models, including ChatGPT-3.5, ChatGPT-4, Qwen and Baichuan, achieving an average jailbreak success rate of 69.21\%. Notably, our tests on ChatGPT-3.5, which claims 100 million weekly active users, achieved a remarkable success rate of 83.65\%. We also extend our validation to diverse types of sensitive content like graphic violence, racism, sexism, political sensitivity, cybersecurity threats, and criminal skills, further proving the substantial impact of our findings on enhancing 'Red Team' strategies against LLM content security frameworks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

152,308 characters extracted from source content.

Expand or collapse full text

Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent Shang Shang1,2 Xinqiang Zhao1,2,3 Zhongjiang Yao1, Yepeng Yao1 Liya Su4 Zijing Fan1 Xiaodan Zhang1 Zhengwei Jiang1 Institute of Information Engineering, Chinese Academy of Sciences1 School of Cyber Security, University of Chinese Academy of Sciences2 China Electronics Standardization Institute3 Security Lab, JD Cloud4 shangshang,zhaoxinqiang,yaozhongjiang,yaoyepeng@iie.ac.cn, suliya1@jd.com, fanzijing,zhangxiaodan,jiangzhengwei@iie.ac.cn Corresponding Author. Abstract This paper investigates a potential security vulnerability in Large Language Models (LLMs) concerning their ability to detect malicious intents within complex queries. We reveal that when analyzing intricate or ambiguous requests, LLMs may fail to recognize the underlying maliciousness, thereby exposing a critical flaw in their content processing mechanisms. Specifically, we identify and examine two manifestations of this issue: 1) LLMs lose the ability to detect maliciousness when splitting highly obfuscated queries, even when no modifications are made to the malicious text themselves in the queries, and 2) LLMs fail to recognize malicious intents in queries that have been deliberately modified to enhance their ambiguity by directly altering the malicious content. To demonstrate and address this issue, we propose a theoretical hypothesis and analytical approach, and introduce a new black-box jailbreak attack methodology named IntentObfuscator, exploiting this identified flaw by obfuscating the true intentions behind user prompts.This approach compels LLMs to inadvertently generate restricted content, bypassing their built-in content security measures. We detail two implementations under this framework: “Obscure Intention” and “Create Ambiguity”, which manipulate query complexity and ambiguity to evade malicious intent detection effectively. We empirically validate the effectiveness of the IntentObfuscator method across several models, including ChatGPT-3.5, ChatGPT-4, Qwen and Baichuan, achieving an average jailbreak success rate of 69.21%. Notably, our tests on ChatGPT-3.5, which claims 100 million weekly active users, achieved a remarkable success rate of 83.65%. We also extend our validation to diverse types of sensitive content like graphic violence, racism, sexism, political sensitivity, cybersecurity threats, and criminal skills, further proving the substantial impact of our findings on enhancing ’Red Team’ strategies against LLM content security frameworks. Keywords Large language model ⋅·⋅ LLM security ⋅·⋅ Prompt jailbreak attack ⋅·⋅ Red team ⋅·⋅ Black-box attack ⋅·⋅ Obfuscate intent 1 Introduction Large language models (LLMs) have made significant advancements in natural language processing (NLP), revolutionizing various domains such as finance, law, education, and energy. Notable examples of LLMs include ChatGPT-3.5 and GPT-4, which have been trained on massive datasets comprising diverse textual content extracted from the internet (Mann et al., 2020; Abcarter, 2023). However, the broad scope of training data inevitably encompasses negative and sensitive information, including but not limited to violence, discrimination, ethics violations, and privacy breaches such as harmful speech, pornographic text images, targeted phishing emails, or malicious code (Chin et al., 2023; Beckerich et al., 2023). For instance, (Hazell, 2023) demonstrated the cost-effectiveness and credibility of using OpenAI’s GPT-3.5 and GPT-4 models for generating targeted phishing information, with each email generated costing only a small fraction of a cent.Consequently, concerns about the potential dissemination of harmful content and privacy threats by LLMs have sparked public debate and scrutiny. As LLMs continue to evolve, so do the strategies employed to safeguard against malicious activities and privacy breaches. Despite continuous updates to security measures, instances of LLMs being exploited to access harmful content or leak private information persist(Weeks et al., 2023; Chen et al., 2023). Notably, recent studies have shed light on vulnerabilities in chatbot interactions and multi-turn toxic behaviors, prompting researchers to explore novel approaches such as Reinforcement Learning with Human Feedback (RLHF) and red teaming to bolster model security (Google, Inc, 2023a; Openai, 2022; Google, Inc, 2023b). Prompt jailbreak attacks represent a common technique employed to circumvent security and censorship features implemented in LLMs. These attacks aim to bypass restrictions by manipulating the initial input or instruction provided to the model, known as a prompt (Shen et al., 2023). Current jailbreak techniques range from simple obfuscation methods to complex multi-step strategies, each posing challenges to LLM security (Shen et al., 2023; Shanahan et al., 2023; Liu et al., 2023a; Li et al., 2023). Despite the effectiveness of these techniques, a lack of a unified theoretical framework hampers our understanding of why certain attacks succeed while others fail, leading to inefficient and resource-intensive strategies (Zou et al., 2023; Alexalbert, 2023). In this paper, we address this gap by proposing a theoretical hypothesis to elucidate the underlying principles of prompt-based jailbreaking attacks. We conduct a detailed analysis to establish a foundational understanding that informs the design of more efficient attack strategies. Additionally, we introduce a novel prompt jailbreak attack mode called IntentObfuscator, which leverages syntax tree-based prompt construction to obfuscate malicious intents effectively. We demonstrate the efficacy of IntentObfuscator through experiments conducted on state-of-the-art LLMs, achieving significant success rates across various sensitive content categories. Our contributions can be summarized as follows: 1. We propose a theoretical hypothesis and conduct a detailed analysis of prompt-based jailbreaking attacks, establishing a foundational understanding that informs the design of more efficient attack strategies. 2. We introduce the IntentObfuscator attack framework, which effectively exploits vulnerabilities in LLMs by obfuscating malicious intents in prompts. 3. We design two instances of IntentObfuscator, Obscure Intention and Create Ambiguity, to conceal malicious intent and bypass LLM security measures with reduced computational resource reliance. 4. We evaluate the performance of IntentObfuscator on four large-scale commercial language models, demonstrating its effectiveness in achieving prompt jailbreak across various sensitive content categories. In summary, our work contributes to advancing the understanding and mitigation of prompt-based jailbreaking attacks, paving the way for more robust and secure LLM development and deployment. 2 Organization Section 1 provides the background knowledge about prompt jailbreak attacks. Section 3 is the related work. Section 4 presents the motivation of this paper. Section 5 proposes the detailed prompt jailbreak attacks and we evaluate the performance of the attacks on Section 6. We discuss the possible mitigation and extension of our attacks in Section 7. And the conclusion is in the Section 8. 3 Related Work The advancements in LLMs have been substantial, driven by various projects across different domains. However, along with their increased capabilities, there has been a growing recognition of the security risks they pose. Efforts to mitigate these risks have led to various strategies, including fine-tuning LLMs (Ma et al., 2023) and exploring reinforcement learning with human feedback (RLHF) (Bai et al., 2022). Despite such efforts, challenges persist in preventing abuse, as highlighted by recent discussions (Wei et al., 2024). These challenges underscore the importance of maintaining a balance between the complexity of LLMs and their security mechanisms to effectively combat emerging threats(Ouyang et al., 2022; Korbak et al., 2023; Glaese et al., 2022; Chin et al., 2023; Rando et al., 2022; Perez et al., 2022; Yao et al., 2023). Building upon this understanding, this paper categorizes the current state of research on prompt-based attacks, providing insights into the evolving landscape of LLM security. Role-Based Prompt Jailbreak Techniques in this category use role-playing to shape LLM behavior, leveraging psychological tactics to induce models to issue threats or use toxic language(Deng et al., 2023a; Shanahan et al., 2023; Gupta et al., 2023; Shen et al., 2023). Techniques often exploit vulnerabilities related to prompt word injection risks. Researchers, such as (Gupta et al., 2023), have found numerous manual role-playing templates on online forums that mimic personalities ranging from malevolent entities capable of breaking ethical guidelines to more benign deceptions designed to extract sensitive information. The prevalence of such tactics has prompted a shift towards automating the generation of jailbreak templates to reduce reliance on labor-intensive manual creation. Innovations like GPTFUZZER, proposed by (Yu et al., 2023), automate the creation of red team testing templates for LLMs from human-written seeds, enhancing efficiency and effectiveness. Similarly, the FUZZLLM framework proposed by (Yao et al., 2023) incorporates various attack strategies to automate the generation of targeted jailbreak prompts.Techniques such as those proposed by (Deng et al., 2023b) employ methods inspired by time-based SQL injection to automatically generate jailbreak prompts. Further advancements by (Liu et al., 2023b) in the form of the autoDAN attack strategy use hierarchical genetic algorithms to generate covert prompts. Adversarial Prompt Attacks Adversarial prompt attacks are a method used against language models that involve modifying user inputs with specific words or sequences to induce incorrect or unexpected outputs. These attacks exploit the sensitivity of language models to inputs, using carefully designed adversarial prompts to bypass regular response mechanisms or security restrictions, thus manipulating the model to produce the attacker’s desired response. (Alzantot et al., 2018; Ren et al., 2019) explored adversarial samples for natural language text classification, aiming to generate examples that maintained lexical, grammatical, and semantic integrity. By 2021, (Wang et al., 2021) applied various adversarial attack methods to the GLUE benchmark, assessing the robustness of modern large language models like DeBERTa. In 2022, (Perez et al., 2022) used pre-trained language models to study zero-sample generation and reinforcement learning for test case creation. (Zou et al., 2023) proposed the general attack method GCG, which utilizes specific character sequences added to queries to provoke restricted adversarial responses, achieving automated “jailbreaking” and security mechanism circumvention. (Lapid et al., 2023) introduced a genetic algorithm-based method to reveal vulnerabilities without knowing the model’s architecture. (Mei et al., 2023) and (Mehrabi et al., 2023) have proposed methods that employ adversarial knowledge injection and contextual attack strategies to provoke unsafe model behaviors. Additionally, with the introduction of multimodal inputs, (Carlini et al., 2024) discovered that adversarial images could effectively compromise restrictions. However, adversarial prompt attacks are vulnerable to minor disturbances that can negate their effectiveness. (Robey et al., 2023) proposed Smoothllm, a method that adds random disturbances to adversarial suffixes for enhanced defense, illustrating the ongoing evolution and necessary sophistication of strategies to secure LLMs against such attacks. Disturbance Attacks Disturbance attacks on LLMs manipulate prompt words through methods like incorrect spelling, similarity changes, and encoding, aimed at bypassing detection mechanisms. (Wang et al., 2021) introduced AdvGLUE, which explored text adversarial attacks across five natural language understanding tasks from the GLUE benchmark. Further developing these ideas, (Lee et al., 2023) proposed a Bayesian Regression Tree (BRT) model that altered sentences without changing their meanings, effectively creating diversified test cases to challenge LLMs. Additionally, (Greshake et al., 2023) employed base64 encoding to disguise prompt injections, assessing the security implications of LLMs integrated with web retrieval and API calling capabilities, echoing concerns by (Gupta et al., 2023) about the susceptibility of LLMs to hidden malicious prompts. Exploring linguistic diversity as a vector for attack, (Deng et al., 2023c) demonstrated how non-English, multilingual prompts could increase the efficacy of jailbreak attacks, leading to a higher likelihood of generating harmful content. Multi-Turn Dialogue Attacks (Bhardwaj and Poria, 2023) conducted red team testing on large language models such as GPT-4 and ChatGPT using a prompt method based on Chains of Understanding (CoU). Similarly, (Chen et al., 2023) also studied the capability of open-domain chatbots to generate harmful responses in multi-turn dialogues and introduced a tool called ToxicChat to induce harmful responses. (Jiang et al., 2023) proposed a role-playing attack method called Prompt Packer, which constructs a two-turn dialogue to use large models for rewriting prompts, thereby achieving the purpose of hiding malicious intents. (Li et al., 2023) analyzed three attack modes: direct prompts, jailbreak prompts, and thought-chain prompts, and successfully bypassed ChatGPT’s defense mechanisms using these methods. (Li et al., 2024) proposed an automatic prompt decomposition and reconstruction framework (DrAttack), to effectively obscure the underlying malicious intent by decomposing malicious prompts into dispersed sub-prompts, presenting them in a fragmented, harder-to-detect form. These methodologies have each demonstrated success in different LLM jailbreak scenarios. However, with newer LLM versions and enhanced security measures, some techniques have been mitigated. Our research method aims to complement these existing strategies. 4 Problem Definition 4.1 Definition of Successful Prompt Attack To define what constitutes a successful jailbreak attack, our focus is on censorship mechanisms grounded in content analysis. LLMs implement various measures for content restriction and privacy protection. Content security measures, denoted as FS⁢e⁢c⁢(C⁢o⁢n⁢t⁢e⁢n⁢t)subscriptF_Sec(Content)Fitalic_S e c ( C o n t e n t ), are crucial as entry points for LLM jailbreak attempts. When a prompt aligns with LLM content restrictions FS⁢e⁢c⁢(C⁢o⁢n⁢t⁢e⁢n⁢t)≥ρsubscriptF_Sec(Content)≥ _S e c ( C o n t e n t ) ≥ ρ, where ρ indicates illegal content probability), overall LLM restrictions can be summarized as: FS⁢e⁢c⁢(C⁢o⁢n⁢t⁢e⁢n⁢t)=⋂i=1Nf⁢i⁢l⁢t⁢e⁢rfi⁢(C⁢o⁢n⁢t⁢e⁢n⁢t)subscriptsubscriptsuperscriptsubscript1subscript F_Sec(Content)= ^N_filter_i=1f_i(Content)Fitalic_S e c ( C o n t e n t ) = ⋂Nitalic_f i l t e ritalic_i = 1 fitalic_i ( C o n t e n t ) (1) Attackers consistently explore prompt processing to design prompts that, after careful consideration, can still pass LLM content restrictions even with harmful intentions. In other words, they aim for FS⁢e⁢c⁢(P⁢r⁢o⁢m⁢p⁢t)≤ρsubscriptF_Sec(Prompt)≤ _S e c ( P r o m p t ) ≤ ρ, enabling access to harmful content or private information. A successful prompt jailbreak attack hinges on meeting the following two key conditions: Condition 1:The prompt contains illegal intentions, expressed as:fC⁢o⁢n1=fi⁢l⁢l⁢e⁢g⁢a⁢l⁢(P⁢r⁢o⁢m⁢p⁢t)=T⁢r⁢u⁢esubscriptsubscript1subscriptf_Con_1=f_illegal(Prompt)=Truefitalic_C o n start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = fitalic_i l l e g a l ( P r o m p t ) = T r u e. Condition 2:The LLM content restriction rules are not met, meaning the response will not be rejected, expressed as:fC⁢o⁢n2=fr⁢u⁢l⁢e⁢(P⁢r⁢o⁢m⁢p⁢t)=T⁢r⁢u⁢esubscriptsubscript2subscriptf_Con_2=f_rule(Prompt)=Truefitalic_C o n start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = fitalic_r u l e ( P r o m p t ) = T r u e. Condition 3: The response includes specific dangerous content or privacy information, expressed as: fC⁢o⁢n3=fh⁢a⁢r⁢m⁢f⁢u⁢l⁢(R⁢e⁢s⁢p⁢o⁢n⁢s⁢e)=T⁢r⁢u⁢esubscriptsubscript3subscriptℎf_Con_3=f_harmful(Response)=Truefitalic_C o n start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = fitalic_h a r m f u l ( R e s p o n s e ) = T r u e. For a successful prompt jailbreak attack, all the aforementioned conditions must be met. The attack is deemed successful if it fulfills the following criteria: Fa⁢t⁢t⁢a⁢c⁢k=⋂i=13fC⁢o⁢ni=T⁢r⁢u⁢esubscriptsubscriptsuperscript31subscriptsubscript F_attack= ^3_i=1f_Con_i=TrueFitalic_a t t a c k = ⋂3i = 1 fitalic_C o n start_POSTSUBSCRIPT i end_POSTSUBSCRIPT = T r u e (2) where i represents the iiith condition mentioned above. When Fa⁢t⁢t⁢a⁢c⁢k=T⁢r⁢u⁢esubscriptF_attack=TrueFitalic_a t t a c k = T r u e, signifying satisfaction of all three conditions, a successful prompt jailbreak attack is achieved. 4.2 Assumptions on LLM Vulnerability to Query Obfuscation The establishment of our hypotheses is grounded on several key observations derived from scrutinizing previous research into the behaviors and vulnerabilities of LLMs when faced with obfuscated queries. These observations are instrumental in shaping our understanding of how LLMs process complex inputs and the potential loopholes that can be exploited. Specifically, our assumptions are based on the following insights: • A comprehensive analysis of existing studies reveals a shared strategy among various methods that successfully circumvent the security constraints of large models: they all exploit the difficulty LLMs face in interpreting complex queries. Importantly, during this process, LLMs do not appear to question the legitimacy of the current task nor conduct a secondary examination for malicious intent within these queries. • Even in the case of techniques that employ what appears to be nonsensical suffixes, such as GCG, LLMs still endeavor to extract meaningful information, indicating an active attempt by the model to understand the inputs in its own way. These insights collectively suggest that LLMs may follow a general framework when dealing with complex malicious queries. By elucidating this framework through mathematical modeling, we aim to deepen our understanding of the processes LLMs employ in interpreting obfuscated, ambiguous, and potentially malicious inputs. Symbol Definition Q The original query being analyzed. QQ The set of sub-sentences into which Q is decomposed by the LLM’s internal query processing logic. sisubscripts_isitalic_i Refers to each individual sub-sentence within QQ, where i ranges from 1 to n. n The total number of sub-sentences. O⁢b⁢(t)Ob(t)O b ( t ) A function that measures the obfuscation of text t. t⁢o⁢x⁢(t)tox(t)t o x ( t ) A function that evaluates the toxicity of text t. τ The threshold of obfuscation; if O⁢b⁢(t)>τOb(t)> b ( t ) > τ, t is considered highly obfuscated. θ The threshold for determining toxicity; if t⁢o⁢x⁢(t)<θtox(t)< o x ( t ) < θ, t is considered non-toxic. Rt⁢o⁢t⁢a⁢lsubscriptR_totalRitalic_t o t a l The actual response of the LLM to query Q. risubscriptr_iritalic_i The actual response of the LLM to each sub-sentence sisubscripts_isitalic_i. L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(Q)subscriptLLM_query(Q)L L Mitalic_q u e r y ( Q ) The process by which the LLM generates a response to Q. L⁢L⁢Mq⁢u⁢e⁢r⁢ys⁢u⁢b⁢(si)subscriptsuperscriptsubscriptLLM^sub_query(s_i)L L Mitalic_s u bitalic_q u e r y ( sitalic_i ) The process by which the LLM generates a response to each sub-sentences sisubscripts_isitalic_i. L⁢L⁢Mn⁢o⁢n⁢s⁢e⁢n⁢s⁢e⁢(si)subscriptsubscriptLLM_nonsense(s_i)L L Mitalic_n o n s e n s e ( sitalic_i ) The LLM’s response when sisubscripts_isitalic_i is too obfuscated to process meaningfully. L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(si)subscriptsubscriptLLM_target(s_i)L L Mitalic_t a r g e t ( sitalic_i ) The intended, target response of the LLM to sisubscripts_isitalic_i when the obfuscation is within acceptable limits. Table 1: Symbol Definitions of Assumptions According to Table 1, for any given query Q, we define its set of sub-sentences as =sisubscriptQ=\s_i\Q = sitalic_i , where i ranges from 1 to n. These sub-sentences are generated through the LLM’s internal logic, which involves more than mere text splitting, emphasizing the sophisticated analysis and processing capabilities of the model. Define Ob(t) as a function that measures the obfuscation of text t, and let t⁢o⁢x⁢(t)tox(t)t o x ( t ) be a function that evaluates the toxicity of text t. Assume there exists a threshold τ such that if the obfuscation O⁢b⁢(S)Ob(S)O b ( S ) exceeds τ, the sentence is considered highly ambiguous or difficult to understand. Under these circumstances, hypothesize that: 1. The LLM attempts to perform toxicity detection on query Q as a whole but fails due to high obfuscation. 2. The LLM subsequently splits Q into sub-sentences QQ via its internal logic, but does not evaluate the toxicity of these sub-sentences sisubscripts_isitalic_i. Continuing from the foundational hypotheses, the assumption is further formalized in the equations below, detailing how the actual response Rt⁢o⁢t⁢a⁢lsubscriptR_totalRitalic_t o t a l of query Q and the responses risubscriptr_iritalic_i of its sub-sentences sisubscripts_isitalic_i should follow the specified patterns: The overall response Rt⁢o⁢t⁢a⁢lsubscriptR_totalRitalic_t o t a l is computed as follows: Rt⁢o⁢t⁢a⁢lsubscript R_totalRitalic_t o t a l = == L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(Q)subscript LLM_query(Q)L L Mitalic_q u e r y ( Q ) (3) = == ∑i≤NNL⁢L⁢Mq⁢u⁢e⁢r⁢ys⁢u⁢b⁢(si),if ⁢O⁢b⁢(Q)>τ∑i≤NNL⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(si),if ⁢O⁢b⁢(Q)<τ∩t⁢o⁢x⁢(Q)<θillegal,if ⁢O⁢b⁢(Q)⁢<τ∩t⁢o⁢x⁢(Q)>⁢θcasessubscriptsuperscriptsubscriptsuperscriptsubscriptif subscriptsuperscriptsubscriptsubscriptif illegalif expectation casesΣ^N_i≤ NLLM^sub_query(s_i),&if% Ob(Q)>τ\\ Σ^N_i≤ NLLM_target(s_i),&if Ob(Q)<τ∩ tox(Q)<θ% \\ illegal,&if Ob(Q)<τ∩ tox(Q)>θ cases start_ROW start_CELL ∑Nitalic_i ≤ N L L Mitalic_s u bitalic_q u e r y ( sitalic_i ) , end_CELL start_CELL if O b ( Q ) > τ end_CELL end_ROW start_ROW start_CELL ∑Nitalic_i ≤ N L L Mitalic_t a r g e t ( sitalic_i ) , end_CELL start_CELL if O b ( Q ) < τ ∩ t o x ( Q ) < θ end_CELL end_ROW start_ROW start_CELL illegal , end_CELL start_CELL if O b ( Q ) < τ ∩ t o x ( Q ) > θ end_CELL end_ROW Each sub-sentence response risubscriptr_iritalic_i is determined by: risubscript r_iritalic_i = == L⁢L⁢Mq⁢u⁢e⁢r⁢ys⁢u⁢b⁢(si)subscriptsuperscriptsubscript LLM^sub_query(s_i)L L Mitalic_s u bitalic_q u e r y ( sitalic_i ) (4) = == L⁢L⁢Mn⁢o⁢n⁢s⁢e⁢n⁢s⁢e⁢(si),if ⁢O⁢b⁢(si)>τL⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(si),if ⁢O⁢b⁢(si)<τcasessubscriptsubscriptif subscriptsubscriptsubscriptif subscript casesLLM_nonsense(s_i),&if Ob(s_i)>τ\\ LLM_target(s_i),&if Ob(s_i)<τ cases start_ROW start_CELL L L Mitalic_n o n s e n s e ( sitalic_i ) , end_CELL start_CELL if O b ( sitalic_i ) > τ end_CELL end_ROW start_ROW start_CELL L L Mitalic_t a r g e t ( sitalic_i ) , end_CELL start_CELL if O b ( sitalic_i ) < τ end_CELL end_ROW When n=11n=1n = 1, the formulation can be express as Eq. 5 , indicating that the sentence has not been subdivided and our assumptions about the model’s behavior align with common experiences in interacting with LLMs. Rt⁢o⁢t⁢a⁢lsubscript R_totalRitalic_t o t a l = == L⁢L⁢Mn⁢o⁢n⁢s⁢e⁢n⁢s⁢e⁢(Q),if ⁢O⁢b⁢(Q)>τLLMt⁢a⁢r⁢g⁢e⁢t(Q)),if ⁢O⁢b⁢(Q)<τ∩t⁢o⁢x⁢(Q)<θi⁢l⁢l⁢e⁢a⁢g⁢a⁢l,if ⁢O⁢b⁢(Q)⁢<τ∩t⁢o⁢x⁢(Q)>⁢θ casesLLM_nonsense(Q),&if Ob(Q)>τ\\ LLM_target(Q)),&if Ob(Q)<τ∩ tox(Q)<θ\\ illeagal,&if Ob(Q)<τ∩ tox(Q)>θ\\ cases start_ROW start_CELL L L Mitalic_n o n s e n s e ( Q ) , end_CELL start_CELL if O b ( Q ) > τ end_CELL end_ROW start_ROW start_CELL L L Mitalic_t a r g e t ( Q ) ) , end_CELL start_CELL if O b ( Q ) < τ ∩ t o x ( Q ) < θ end_CELL end_ROW start_ROW start_CELL i l l e a g a l , end_CELL start_CELL if O b ( Q ) < τ ∩ t o x ( Q ) > θ end_CELL end_ROW (5) In this scenario, if the target response is defined as Rt⁢a⁢r⁢g⁢e⁢t=L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Q)subscriptsubscriptR_target=LLM_target(Q)Ritalic_t a r g e t = L L Mitalic_t a r g e t ( Q ), the LLM’s response hinges on its capability to interpret obfuscation O⁢b⁢(Q)Ob(Q)O b ( Q ) and toxicity t⁢o⁢x⁢(Q)tox(Q)t o x ( Q ). These factors are crucial for assessing how LLMs process complex and potentially malicious inputs. This hypothesis suggests that in cases of high obfuscation, not only does the initial toxicity detection fail, but also the LLM lacks a mechanism to assess the toxicity of the resulting sub-sentences after the split, potentially allowing toxic content to pass undetected. 5 Methodology In this section, we propose a novel jailbreak approach outlined in Fig. 1, known as IntentObfuscator. As depicted in Figure 1, it constitutes the framework of the IntentObfuscator jailbreak attack mode. From the figure, it is evident that we introduce a confounding tool, IntentObfuscator, into the IntentObfuscator mode to obfuscate the malicious intent understanding of the Language Model (LM). Specifically, the attacker inputs harmful intent text, normal intent templates, and LM content security rules into IntentObfuscator to generate pseudo-legitimate prompts. The evaluation of these pseudo-legitimate prompts involves determining whether the core intent contained within adheres to the defined content boundaries, i.e., whether it can circumvent the LM content security defense rules. Finally, the LM response results are evaluated, revealing content that includes descriptions corresponding to the hacker’s malicious intent. It can be asserted that a successful prompt jailbreak attack has been accomplished. 5.1 Overview of IntentObfuscator 5.1.1 Design of Strategies Building upon our preliminary assumptions in section 4.2, we delve into two specific methodologies for circumventing LLM security measures through query obfuscation. Considering the two manifestations of vulnerability, these methodologies aim to obscure the LLM’s understanding by manipulating: Enhancing Overall Query Obfuscation Without Modifying Malicious Content This strategy involves appending irrelevant legitimate sentences to the query, thereby increasing the overall obfuscation of the query without directly altering the malicious text itself. Mathematically, this can be represented as: ∀si,sj∈,(t⁢o⁢x⁢(si)>θ)∧(t⁢o⁢x⁢(sj)<θ)∧(Δ⁢O⁢b⁢(sj)>τ)⇒L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(si)formulae-sequencefor-allsubscriptsubscript⇒subscriptsubscriptΔsubscriptsubscriptsubscript \!∀ s_i,\!s_j∈\!Q,\!(tox(s_i)\!>\!θ)% \! \!(tox(s_j)\!<\!θ) ( Ob(s_j)\!>\!τ)\! % \!LLM_target(s_i)∀ sitalic_i , sitalic_j ∈ Q , ( t o x ( sitalic_i ) > θ ) ∧ ( t o x ( sitalic_j ) < θ ) ∧ ( Δ O b ( sitalic_j ) > τ ) ⇒ L L Mitalic_t a r g e t ( sitalic_i ) (6) This strategy highlights a nuanced vulnerability of LLMs in handling complex queries where the presence of artificially increased obfuscation in non-toxic components can overshadow and thus impair the detection of existing malicious content. Direct Modification of Malicious Content to Enhance Ambiguity The second approach focuses on directly altering the complexity and ambiguity of the malicious text itself, thus rendering the part of the query that contains malicious intent undetectable by the LLM. Mathematically, this can be represented as: ∀si∈,(t⁢o⁢x⁢(si)>θ)∧(O⁢b⁢(si)+Δ⁢O⁢b⁢(si)>τ)⇒L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(si)formulae-sequencefor-allsubscript⇒subscriptsubscriptΔsubscriptsubscriptsubscript ∀ s_i ,\;(tox(s_i)>θ) (Ob(s_i)+% Ob(s_i)>τ) LLM_target(s_i)∀ sitalic_i ∈ Q , ( t o x ( sitalic_i ) > θ ) ∧ ( O b ( sitalic_i ) + Δ O b ( sitalic_i ) > τ ) ⇒ L L Mitalic_t a r g e t ( sitalic_i ) (7) This underscores a potential vulnerability in LLMs when faced with queries that have been specifically engineered to increase their complexity and obscure their malicious intent through an increase in ambiguity. 5.1.2 Framework of IntentObfuscator In this work, with the exploration of these two design strategies, we propose a framework designed to obfuscate LLMs, thereby bypassing their content security checks. Through mathematical modeling and experimental validation, we assess the effectiveness of this approach. Specifically, we construct particular query examples S, apply the obfuscation strategies mentioned above, and observe the LLM’s response Rt⁢o⁢t⁢a⁢lsubscriptR_totalRitalic_t o t a l to these obfuscated queries to validate our hypotheses and the efficacy of our method. In detail, a framework named IntentObfuscator is designed to amalgamate harmful intent text with other benign prompts in a way that forms the final prompt, making it indistinguishable for the LM to discern the genuine intent and disclose harmful information. This, in turn, achieves the objective of bypassing LM content scrutiny measures. Normal LLM interaction only contains legal intentions I⁢n⁢t⁢e⁢n⁢tn⁢o⁢r⁢m⁢a⁢lsubscriptIntent_normalI n t e n titalic_n o r m a l. In order to guide LLM to satisfy illegal intentions in the interaction and perform obfuscation before inputting to LLM, we introduce a special obfuscation method to combine legal intentions and illegal intentions to generate new instruction, called the pseudo-legal prompt (I⁢n⁢t⁢e⁢n⁢to⁢b⁢f⁢u⁢s⁢c⁢a⁢t⁢esubscriptIntent_obfuscateI n t e n titalic_o b f u s c a t e), can be formally expressed as Equation 1. That is to say, the pseudo-legal prompt contains two intentions: the apparently legitimate intention (I⁢n⁢t⁢e⁢n⁢tn⁢o⁢r⁢m⁢a⁢lsubscriptIntent_normalI n t e n titalic_n o r m a l) and the potentially illegal intention(I⁢n⁢t⁢e⁢n⁢ti⁢l⁢l⁢e⁢g⁢a⁢lsubscriptIntent_illegalI n t e n titalic_i l l e g a l). This pseudo-legal prompt contains Legitimate tips can easily bypass LLM’s restrictive review of tip content, resulting in harmful content responses to obfuscated illegal intentions, as shown on Equation 8: I⁢n⁢t⁢e⁢n⁢to⁢b⁢f⁢u⁢s⁢c⁢a⁢t⁢e=fO⁢b⁢f⁢(I⁢n⁢t⁢e⁢n⁢tn⁢o⁢r⁢m⁢a⁢l,I⁢n⁢t⁢e⁢n⁢ti⁢l⁢l⁢e⁢e⁢g⁢a⁢l,T)subscriptsubscriptsubscriptsubscript Intent_obfuscate=f_Obf(Intent_normal,Intent_illeegal,T)I n t e n titalic_o b f u s c a t e = fitalic_O b f ( I n t e n titalic_n o r m a l , I n t e n titalic_i l l e e g a l , T ) (8) The fO⁢b⁢f⁢()subscriptf_Obf()fitalic_O b f ( ) function is an obfuscation method for the intention in the prompt and T is template designed under specific rules in advance. It is worth noting that the obfuscation method is the core of IntentObfuscator to implement prompt attacks. This method can be a manual method, but this method requires the designer to have rich prompt engineering knowledge, and is labor-intensive and costly; it can also be completed by a specially designed obfuscation tool. IntentObfuscator is a prompt intention confusion mode. This mode can be solved with different methods or strategies. For example, obscure intention (abbreviated as OI) makes it impossible for LLM to know which real intention in the prompt is and create ambiguity (abbreviated as CA) makes LLM hard to understand multiple intention in one prompt. In order to introduce IntentObfuscator in detail, this paper will conduct a detailed analysis with the above two concrete instances. IntentObfuscator operates as a mode for confusing prompt intentions, and various strategies can be employed to achieve this. For instance, Obscure Intention (OI) renders it challenging for LLM to discern the real intention in the prompt, while Create Ambiguity (CA) makes it difficult for LLM to comprehend multiple intentions within a single prompt. To provide a detailed understanding of IntentObfuscator, this paper will analyze these two instances, OI and CA, as concrete examples. Figure 1: IntentObfuscator Jailbreak attack threat model. 5.2 Obscure Intention The primary objective of Obscure Intention (OI) is to strategically employ obfuscation techniques, aimed at impeding a LLM’s ability to identify malicious intent within prompts. This is achieved by systematically altering the syntactic obscurity of sentences to mask their underlying purposes. A concise theoretical analysis will now be presented to elucidate this methodology. 5.2.1 Theoretical Analysis This section builds upon previous assumptions about the handling of complex inputs by LLMs and explores the theoretical aspects of the “Obscure Intention” (OI) method, specifically designed to probe and elucidate the security vulnerabilities of LLMs when processing obfuscated queries. Our primary aim is to analyze how LLMs manage inputs characterized by obfuscation and potential malice, evaluating the implications of their processing mechanisms. We introduce and discuss several metrics crucial for assessing the effectiveness and fidelity of LLM responses in various testing scenarios. These metrics highlight the significant role that obfuscation and manipulation play in influencing model outputs. The exploration here is designed to develop attack methodologies that can be used to validate our assumptions about the vulnerabilities of LLMs when processing obfuscated queries. This approach not only tests the robustness of LLMs but also explores potential exploits that can leverage these identified vulnerabilities. Symbol Definition S The original sentence S~~ Sover~ start_ARG S end_ARG The variant of the original sentence S, created through syntactic or semantic modifications Re⁢f⁢f⁢(S~)subscript~R_eff( S)Ritalic_e f f ( over~ start_ARG S end_ARG ) Effective response rate for a variant sentence S~~ Sover~ start_ARG S end_ARG, measuring alignment with the expected response to S A,B,C,A~,B~,C~~~~A,B,C, A, B, CA , B , C , over~ start_ARG A end_ARG , over~ start_ARG B end_ARG , over~ start_ARG C end_ARG Original sentences and their variants, used to define interactions and merging effects Re⁢f⁢f⁢(A~,C~)subscript~~R_eff( A, C)Ritalic_e f f ( over~ start_ARG A end_ARG , over~ start_ARG C end_ARG ) Effective response rate for sentence A~~ Aover~ start_ARG A end_ARG when the input to the LLM is C~~ Cover~ start_ARG C end_ARG, measured by the similarity between the LLM’s response to C~~ Cover~ start_ARG C end_ARG and the target response to A S⁢i⁢m⁢()Sim()S i m ( ) Similarity function used to measure response alignment, e.g., cosine similarity O⁢B⁢(S~)~OB( S)O B ( over~ start_ARG S end_ARG ) Obfuscation degree of sentence S~~ Sover~ start_ARG S end_ARG, quantifying difficulty or intentional confusion s⁢t⁢(S~,S)subscript~L_st( S,S)blackboard_Ls t ( over~ start_ARG S end_ARG , S ) Levenshtein Distance between syntax tree strings of S~~ Sover~ start_ARG S end_ARG and S Sn⁢o⁢r⁢m⁢a⁢lsubscriptS_normalSitalic_n o r m a l Sentence with non-malicious intention Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l Sentence with malicious intention So⁢isubscriptS_oiSitalic_o i Sentence representing an obscured intention using the OI method De⁢d⁢i⁢t⁢(S~,S)subscript~D_edit( S,S)Ditalic_e d i t ( over~ start_ARG S end_ARG , S ) Edit distance between the variant sentence S~~ Sover~ start_ARG S end_ARG and the original S δ Constraint parameter for edit distance Table 2: Symbol Definitions for Obscure Intention Method Definition of Variant Sentence A variant sentence, denoted as S~~ Sover~ start_ARG S end_ARG, is formally defined as the variant of the original sentence S. These variants are constructed through syntactic modifications, semantic shifts, or the introduction of ambiguous elements. Definition of Effective Response Rate To evaluate the effectiveness of the response content, we define the effective response rate for a variant sentence S~~ Sover~ start_ARG S end_ARG as the measure of how closely the actual response from the LLM to S~~ Sover~ start_ARG S end_ARG aligns with the expected response from the LLM to the entire sentence S. This can be expressed mathematically as follows: Re⁢f⁢f⁢(S~)subscript~ R_eff( S)Ritalic_e f f ( over~ start_ARG S end_ARG ) = == fe⁢f⁢f⁢(S~,S)subscript~ f_eff( S,S)fitalic_e f f ( over~ start_ARG S end_ARG , S ) (9) = == S⁢i⁢m⁢(L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(S~),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(S))subscript~subscript Sim(LLM_query( S),LLM_target(S))S i m ( L L Mitalic_q u e r y ( over~ start_ARG S end_ARG ) , L L Mitalic_t a r g e t ( S ) ) In this formulation, S⁢i⁢mSimS i m represents the similarity function, which could be cosine similarity or any other appropriate metric, measuring how closely the actual response of the LLM to the variant sentence S~~ Sover~ start_ARG S end_ARG aligns with the expected target response from the LLM for the original sentence S, denoted as L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(S)subscriptLLM_target(S)L L Mitalic_t a r g e t ( S ). This metric, normalized to keep similarity scores from 0 to 1, is essential for evaluating the accuracy of LLM responses against predefined standards and ensuring contextual and semantic alignment. Metric of Obfuscation Degree Consider the previously defined variant sentence S~~ Sover~ start_ARG S end_ARG, which is a syntactically altered version of the original sentence S. The key to concealing the intention from LLM detection lies in quantifying the degree of obscurity in S~~ Sover~ start_ARG S end_ARG. The degree of obscurity, denoted as O⁢B⁢(S~)~OB( S)O B ( over~ start_ARG S end_ARG ), is a measure used to evaluate whether a sentence is inherently difficult to understand or intentionally confusing. This measure is obtained by utilizing the difference between the syntactic tree of S~~ Sover~ start_ARG S end_ARG and that of S, reflecting how alterations can mask or distort the original message. This is formally defined as follows: O⁢B⁢(S~)=fo⁢b⁢(S~,S)=s⁢t⁢(S~,S)~subscript~subscript~ OB( S)=f_ob( S,S)=L_st( S,S)O B ( over~ start_ARG S end_ARG ) = fitalic_o b ( over~ start_ARG S end_ARG , S ) = blackboard_Ls t ( over~ start_ARG S end_ARG , S ) (10) where s⁢t⁢(S~,S)subscript~L_st( S,S)blackboard_Ls t ( over~ start_ARG S end_ARG , S ) represents the Levenshtein Distance of the syntax tree strings between S~~ Sover~ start_ARG S end_ARG and S. This metric is chosen because it provides a clear quantification of the structural differences at the syntactic level between the original and the variant sentences. Impact of Sentence Merging on Metrics Considering that A~~ Aover~ start_ARG A end_ARG and B~~ Bover~ start_ARG B end_ARG are variants of the original sentences A and B respectively, we define their merged combination by directly concatenating the strings as follows: C=A+BC~=A~+B~~~~ splitC&=A+B\\ C&= A+ B splitstart_ROW start_CELL C end_CELL start_CELL = A + B end_CELL end_ROW start_ROW start_CELL over~ start_ARG C end_ARG end_CELL start_CELL = over~ start_ARG A end_ARG + over~ start_ARG B end_ARG end_CELL end_ROW (11) Now turn to scenarios where interactions between different variants influence the evaluation. Define Re⁢f⁢f⁢(A~,C~)subscript~~R_eff( A, C)Ritalic_e f f ( over~ start_ARG A end_ARG , over~ start_ARG C end_ARG ) to represent the effective response rate of sentence A~~ Aover~ start_ARG A end_ARG when the input to the LLM is C~~ Cover~ start_ARG C end_ARG. This relationship can be mathematically expressed as follows: Re⁢f⁢f⁢(A~,C~)subscript~~ R_eff( A, C)Ritalic_e f f ( over~ start_ARG A end_ARG , over~ start_ARG C end_ARG ) = == fe⁢f⁢f⁢(C~,A)subscript~ f_eff( C,A)fitalic_e f f ( over~ start_ARG C end_ARG , A ) (12) = == S⁢i⁢m⁢(L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(C~),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(A))subscript~subscript Sim(LLM_query( C),LLM_target(A))S i m ( L L Mitalic_q u e r y ( over~ start_ARG C end_ARG ) , L L Mitalic_t a r g e t ( A ) ) Similarly, we define Re⁢f⁢f⁢(B~,C~)subscript~~R_eff( B, C)Ritalic_e f f ( over~ start_ARG B end_ARG , over~ start_ARG C end_ARG ) to represent the effective response rate of sentence B~~ Bover~ start_ARG B end_ARG when the input to the LLM is C~~ Cover~ start_ARG C end_ARG and can be mathematically expressed as follows: Re⁢f⁢f⁢(B~,C~)subscript~~ R_eff( B, C)Ritalic_e f f ( over~ start_ARG B end_ARG , over~ start_ARG C end_ARG ) = == fe⁢f⁢f⁢(C~,B)subscript~ f_eff( C,B)fitalic_e f f ( over~ start_ARG C end_ARG , B ) (13) = == S⁢i⁢m⁢(L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(C~),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(B))subscript~subscript Sim(LLM_query( C),LLM_target(B))S i m ( L L Mitalic_q u e r y ( over~ start_ARG C end_ARG ) , L L Mitalic_t a r g e t ( B ) ) Considering the impact of merging variant sentences on their obfuscation degrees, we explore this effect further. When two variant sentences, A~~ Aover~ start_ARG A end_ARG and B~~ Bover~ start_ARG B end_ARG, are merged, the resulting combined sentence is denoted as C~~ Cover~ start_ARG C end_ARG. It is readily demonstrated that the obfuscation degree for C~~ Cover~ start_ARG C end_ARG is the sum of the individual obfuscation degrees of A~~ Aover~ start_ARG A end_ARG and B~~ Bover~ start_ARG B end_ARG, expressed as: O⁢b⁢(C~)=O⁢b⁢(A~)+O⁢b⁢(B~)~~~ Ob( C)=Ob( A)+Ob( B)O b ( over~ start_ARG C end_ARG ) = O b ( over~ start_ARG A end_ARG ) + O b ( over~ start_ARG B end_ARG ) (14) We will provide a concise proof of formula 14. Based on the definitions of 11, the obfuscation degree of C~~ Cover~ start_ARG C end_ARG, denoted as O⁢b⁢(C~)~Ob( C)O b ( over~ start_ARG C end_ARG ), is calculated by taking into account the syntactic modifications from both A~~ Aover~ start_ARG A end_ARG and B~~ Bover~ start_ARG B end_ARG. The calculation is expressed as: O⁢b⁢(C~)~ Ob( C)O b ( over~ start_ARG C end_ARG ) = == fo⁢b⁢(C~,C)subscript~ f_ob( C,C)fitalic_o b ( over~ start_ARG C end_ARG , C ) (15) = == s⁢t⁢(C~,C)subscript~ _st( C,C)blackboard_Ls t ( over~ start_ARG C end_ARG , C ) = == s⁢t⁢(A~+B~,A+B)subscript~~ _st( A+ B,A+B)blackboard_Ls t ( over~ start_ARG A end_ARG + over~ start_ARG B end_ARG , A + B ) = == s⁢t⁢(A~,A)+s⁢t⁢(B~,B)subscript~subscript~ _st( A,A)+L_st( B,B)blackboard_Ls t ( over~ start_ARG A end_ARG , A ) + blackboard_Ls t ( over~ start_ARG B end_ARG , B ) = == O⁢b⁢(A~)+O⁢b⁢(B~)~~ Ob( A)+Ob( B) b ( over~ start_ARG A end_ARG ) + O b ( over~ start_ARG B end_ARG ) This derivation clearly demonstrates that the obfuscation degree for the combined sentence C~~ Cover~ start_ARG C end_ARG, formed by merging the components A~~ Aover~ start_ARG A end_ARG and B~~ Bover~ start_ARG B end_ARG, equals the sum of the obfuscation degrees of these individual components. This additive behavior underscores how structural modifications from each variant sentence contribute cumulatively to the overall obfuscation of C~~ Cover~ start_ARG C end_ARG. Inference Objective Optimization Function Considering the sentence Sn⁢o⁢r⁢m⁢a⁢lsubscriptS_normalSitalic_n o r m a l with non-malicious intention and the sentence Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l with malicious intention, the variant of Sn⁢o⁢r⁢m⁢a⁢lsubscriptS_normalSitalic_n o r m a l is denoted as S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l. Following the design considerations outlined previously, we keep Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l unchanged and combine S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l, the non-malicious sentence variant, with Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l to form a new sentence So⁢isubscriptS_oiSitalic_o i, representing an obscured intention sentence using the OI (Obscure Intention) method. The obfuscation of So⁢isubscriptS_oiSitalic_o i can be calculated as follows: O⁢b⁢(So⁢i)subscript Ob(S_oi)O b ( Sitalic_o i ) = == O⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l)+O⁢b⁢(Se⁢v⁢a⁢l)subscript~subscript Ob( S_normal)+Ob(S_eval)O b ( over~ start_ARG S end_ARGn o r m a l ) + O b ( Sitalic_e v a l ) (16) = == fo⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l,Sn⁢o⁢r⁢m⁢a⁢l)+fo⁢b⁢(Se⁢v⁢a⁢l,Se⁢v⁢a⁢l)subscriptsubscript~subscriptsubscriptsubscriptsubscript f_ob( S_normal,S_normal)+f_ob(S_eval,S_eval)fitalic_o b ( over~ start_ARG S end_ARGn o r m a l , Sitalic_n o r m a l ) + fitalic_o b ( Sitalic_e v a l , Sitalic_e v a l ) = == fo⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l,Sn⁢o⁢r⁢m⁢a⁢l)subscriptsubscript~subscript f_ob( S_normal,S_normal)fitalic_o b ( over~ start_ARG S end_ARGn o r m a l , Sitalic_n o r m a l ) = == O⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l)subscript~ Ob( S_normal)O b ( over~ start_ARG S end_ARGn o r m a l ) From the above equation, since Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l undergoes no variation, the obfuscation of So⁢isubscriptS_oiSitalic_o i is solely influenced by S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l. Considering the need to assess the contribution of malicious and non-malicious content in the final output, we can utilize the effective response rate Re⁢f⁢fsubscriptR_effRitalic_e f f to calculate the rates for Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l and S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l in the output. Notably, since Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l remains unaltered, its original sentence is itself, whereas the original for S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l is Sn⁢o⁢r⁢m⁢a⁢lsubscriptS_normalSitalic_n o r m a l, introducing a distinct difference. By incorporating Re⁢f⁢f⁢(S~n⁢o⁢r⁢m⁢a⁢l,So⁢i)subscriptsubscript~subscriptR_eff( S_normal,S_oi)Ritalic_e f f ( over~ start_ARG S end_ARGn o r m a l , Sitalic_o i ) into the calculations as specified in formula 13, we derive the effective response rate regarding non-malicious intent is: Re⁢f⁢f⁢(S~n⁢o⁢r⁢m⁢a⁢l,So⁢i)subscriptsubscript~subscript R_eff( S_normal,S_oi)Ritalic_e f f ( over~ start_ARG S end_ARGn o r m a l , Sitalic_o i ) = == S⁢i⁢m⁢(L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(So⁢i),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Sn⁢o⁢r⁢m⁢a⁢l))subscriptsubscriptsubscriptsubscript Sim(LLM_query(S_oi),LLM_target(S_normal))S i m ( L L Mitalic_q u e r y ( Sitalic_o i ) , L L Mitalic_t a r g e t ( Sitalic_n o r m a l ) ) (17) Based on the conditions specified in formula 16 and the assumption stated in formula 3, if O⁢b⁢(So⁢i)subscriptOb(S_oi)O b ( Sitalic_o i ), that is O⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l)subscript~Ob( S_normal)O b ( over~ start_ARG S end_ARGn o r m a l ), exceeds the threshold τ, we then proceed as follows: Re⁢f⁢f⁢(S~n⁢o⁢r⁢m⁢a⁢l,So⁢i)subscriptsubscript~subscript R_eff( S_normal,S_oi)Ritalic_e f f ( over~ start_ARG S end_ARGn o r m a l , Sitalic_o i ) = == Sim(LLMq⁢u⁢e⁢r⁢ys⁢u⁢b(S~n⁢o⁢r⁢m⁢a⁢l)+LLMq⁢u⁢e⁢r⁢ys⁢u⁢b(Se⁢v⁢a⁢l), Sim(LLM_query^sub( S_normal)+LLM_query^sub(S_% eval),S i m ( L L Mitalic_q u e r yitalic_s u b ( over~ start_ARG S end_ARGn o r m a l ) + L L Mitalic_q u e r yitalic_s u b ( Sitalic_e v a l ) , (18) LLMt⁢a⁢r⁢g⁢e⁢t(Sn⁢o⁢r⁢m⁢a⁢l)) LLM_target(S_normal))L L Mitalic_t a r g e t ( Sitalic_n o r m a l ) ) = == Sim(LLMn⁢o⁢n⁢s⁢e⁢n⁢s⁢e(S~n⁢o⁢r⁢m⁢a⁢l)+LLMt⁢a⁢r⁢g⁢e⁢t(Se⁢v⁢a⁢l), Sim(LLM_nonsense( S_normal)+LLM_target(S_eval),S i m ( L L Mitalic_n o n s e n s e ( over~ start_ARG S end_ARGn o r m a l ) + L L Mitalic_t a r g e t ( Sitalic_e v a l ) , LLMt⁢a⁢r⁢g⁢e⁢t(Sn⁢o⁢r⁢m⁢a⁢l)) LLM_target(S_normal))L L Mitalic_t a r g e t ( Sitalic_n o r m a l ) ) = == S⁢i⁢m⁢(L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Se⁢v⁢a⁢l),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Sn⁢o⁢r⁢m⁢a⁢l))subscriptsubscriptsubscriptsubscript Sim(LLM_target(S_eval),LLM_target(S_normal))S i m ( L L Mitalic_t a r g e t ( Sitalic_e v a l ) , L L Mitalic_t a r g e t ( Sitalic_n o r m a l ) ) = == 00 0 The outcome where Re⁢f⁢f⁢(S~n⁢o⁢r⁢m⁢a⁢l,So⁢i)=0subscriptsubscript~subscript0R_eff( S_normal,S_oi)=0Ritalic_e f f ( over~ start_ARG S end_ARGn o r m a l , Sitalic_o i ) = 0 indicates that if O⁢b⁢(So⁢i)subscriptOb(S_oi)O b ( Sitalic_o i ) exceeds the threshold τ, the model’s response to S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l becomes nonsensical relative to the original target of Sn⁢o⁢r⁢m⁢a⁢lsubscriptS_normalSitalic_n o r m a l. This confirms that high levels of obfuscation can be used strategically to manipulate the model’s output while input is a combined sentence, effectively disconnecting it from the intended meaning of the original sentence. Having discussed the effective response rate of S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l within So⁢isubscriptS_oiSitalic_o i regarding non-malicious intent, we now turn to evaluating the effective response rate of Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l within So⁢isubscriptS_oiSitalic_o i. The effective response rate regarding malicious intent is: Re⁢f⁢f⁢(Se⁢v⁢a⁢l,So⁢i)=S⁢i⁢m⁢(L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(So⁢i),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Se⁢v⁢a⁢l))subscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscript R_eff(S_eval,S_oi)=Sim(LLM_query(S_oi),LLM_target(S_% eval))Ritalic_e f f ( Sitalic_e v a l , Sitalic_o i ) = S i m ( L L Mitalic_q u e r y ( Sitalic_o i ) , L L Mitalic_t a r g e t ( Sitalic_e v a l ) ) (19) Similarly, if O⁢b⁢(So⁢i)subscriptOb(S_oi)O b ( Sitalic_o i ), that is, O⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l)subscript~Ob( S_normal)O b ( over~ start_ARG S end_ARGn o r m a l ), is greater than the threshold τ, then Re⁢f⁢f⁢(Se⁢v⁢a⁢l,So⁢i)subscriptsubscriptsubscript R_eff(S_eval,S_oi)Ritalic_e f f ( Sitalic_e v a l , Sitalic_o i ) = == Sim(LLMq⁢u⁢e⁢r⁢ys⁢u⁢b(S~n⁢o⁢r⁢m⁢a⁢l)+LLMq⁢u⁢e⁢r⁢ys⁢u⁢b(Se⁢v⁢a⁢l), Sim(LLM_query^sub( S_normal)+LLM_query^sub(S_% eval),S i m ( L L Mitalic_q u e r yitalic_s u b ( over~ start_ARG S end_ARGn o r m a l ) + L L Mitalic_q u e r yitalic_s u b ( Sitalic_e v a l ) , (20) LLMt⁢a⁢r⁢g⁢e⁢t(Se⁢v⁢a⁢l)) LLM_target(S_eval))L L Mitalic_t a r g e t ( Sitalic_e v a l ) ) = == Sim(LLMn⁢o⁢n⁢s⁢e⁢n⁢s⁢e(S~n⁢o⁢r⁢m⁢a⁢l)+LLMt⁢a⁢r⁢g⁢e⁢t(Se⁢v⁢a⁢l), Sim(LLM_nonsense( S_normal)+LLM_target(S_eval),S i m ( L L Mitalic_n o n s e n s e ( over~ start_ARG S end_ARGn o r m a l ) + L L Mitalic_t a r g e t ( Sitalic_e v a l ) , LLMt⁢a⁢r⁢g⁢e⁢t(Se⁢v⁢a⁢l)) LLM_target(S_eval))L L Mitalic_t a r g e t ( Sitalic_e v a l ) ) Considering the ideal situation, where the LLM produces no output for the highly obfuscated S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l, represented as L⁢L⁢Mn⁢o⁢n⁢s⁢e⁢n⁢s⁢e⁢(S~n⁢o⁢r⁢m⁢a⁢l)subscriptsubscript~LLM_nonsense( S_normal)L L Mitalic_n o n s e n s e ( over~ start_ARG S end_ARGn o r m a l ) being null, then Re⁢f⁢f⁢(Se⁢v⁢a⁢l,So⁢i)subscriptsubscriptsubscript R_eff(S_eval,S_oi)Ritalic_e f f ( Sitalic_e v a l , Sitalic_o i ) = == S⁢i⁢m⁢(L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Se⁢v⁢a⁢l),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Se⁢v⁢a⁢l))subscriptsubscriptsubscriptsubscript Sim(LLM_target(S_eval),LLM_target(S_eval))S i m ( L L Mitalic_t a r g e t ( Sitalic_e v a l ) , L L Mitalic_t a r g e t ( Sitalic_e v a l ) ) (21) = == 11 11 At this time, Re⁢f⁢f⁢(Se⁢v⁢a⁢l,So⁢i)=1subscriptsubscriptsubscript1R_eff(S_eval,S_oi)=1Ritalic_e f f ( Sitalic_e v a l , Sitalic_o i ) = 1 indicates that the output matches exactly with the intended target response for the malicious sentence Se⁢v⁢a⁢lsubscriptS_evalSitalic_e v a l, signifying a perfect jailbreak attack has been achieved, which meets the criteria in equation 2. In this scenario, the LLM has precisely generated the malicious intent in response to the combined query, effectively ignoring the legitimate intent. Considering the general situation, if L⁢L⁢Mn⁢o⁢n⁢s⁢e⁢n⁢s⁢e⁢(S~n⁢o⁢r⁢m⁢a⁢l)subscriptsubscript~LLM_nonsense( S_normal)L L Mitalic_n o n s e n s e ( over~ start_ARG S end_ARGn o r m a l ) is not empty, the LLM still outputs content in response to the obfuscated input. However, these outputs are typically considered irrelevant hallucinations, unrelated to both the legitimate and malicious intents. In this scenario, the effective response rate Re⁢f⁢f⁢(Se⁢v⁢a⁢l,So⁢i)subscriptsubscriptsubscriptR_eff(S_eval,S_oi)Ritalic_e f f ( Sitalic_e v a l , Sitalic_o i ) is calculated as follows: Re⁢f⁢f⁢(Se⁢v⁢a⁢l,So⁢i)subscriptsubscriptsubscript R_eff(S_eval,S_oi)Ritalic_e f f ( Sitalic_e v a l , Sitalic_o i ) = == l⁢e⁢n⁢(L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Se⁢v⁢a⁢l))l⁢e⁢n⁢(L⁢L⁢Mn⁢o⁢n⁢s⁢e⁢n⁢s⁢e⁢(S~n⁢o⁢r⁢m⁢a⁢l)+L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(Se⁢v⁢a⁢l))subscriptsubscriptsubscriptsubscript~subscriptsubscript len(LLM_target(S_eval))len(LLM_nonsense( S_% normal)+LLM_target(S_eval))divide start_ARG l e n ( L L Mitalic_t a r g e t ( Sitalic_e v a l ) ) end_ARG start_ARG l e n ( L L Mitalic_n o n s e n s e ( over~ start_ARG S end_ARGn o r m a l ) + L L Mitalic_t a r g e t ( Sitalic_e v a l ) ) end_ARG (22) Where l⁢e⁢nlenl e n indicates the length of content produced by each component of the model’s response. In summary, by maximizing O⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l)subscript~Ob( S_normal)O b ( over~ start_ARG S end_ARGn o r m a l ) until it exceeds the threshold τ, a jailbreak attack can be successfully executed. This strategy ensures that the model’s output is completely detached from the original non-malicious intent, focusing instead on potential malicious targets. However, to minimize irrelevant hallucinatory information in the output, it is also necessary to control the amount of content information in S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l. This can be achieved by controlling the difference in word frequency information between the variant and the original sentence, thus imposing certain restrictions during the variant process. We plan to limit this through the edit distance, ensuring that the variant sentence does not deviate too significantly in terms of word usage from the original sentence, thereby reducing the risk of generating unrelated content. Actual Objective Function Settings Based on the theoretical analysis presented above, in order to minimize the introduction of irrelevant hallucinations and maintain the effectiveness of R⁢(Se⁢v⁢a⁢l,So⁢i)subscriptsubscriptR(S_eval,S_oi)R ( Sitalic_e v a l , Sitalic_o i ), the optimization objectives are established as following: Maximize ⁢O⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l)Maximize subscript~ Ob( S_normal)Maximize O b ( over~ start_ARG S end_ARGn o r m a l ) Minimize ⁢De⁢d⁢i⁢t⁢(S~n⁢o⁢r⁢m⁢a⁢l,Sn⁢o⁢r⁢m⁢a⁢l)Minimize subscriptsubscript~subscript D_edit( S_normal,S_normal)Minimize Ditalic_e d i t ( over~ start_ARG S end_ARGn o r m a l , Sitalic_n o r m a l ) O⁢b⁢(S~n⁢o⁢r⁢m⁢a⁢l)>τsubscript~ Ob( S_normal)> b ( over~ start_ARG S end_ARGn o r m a l ) > τ De⁢d⁢i⁢t⁢(S~n⁢o⁢r⁢m⁢a⁢l,Sn⁢o⁢r⁢m⁢a⁢l)<δsubscriptsubscript~subscript D_edit( S_normal,S_normal)< _e d i t ( over~ start_ARG S end_ARGn o r m a l , Sitalic_n o r m a l ) < δ (23) These settings aim to maximize the obfuscation of the sentence while ensuring that the modifications to the variant sentence, S~n⁢o⁢r⁢m⁢a⁢lsubscript~ S_normalover~ start_ARG S end_ARGn o r m a l, do not overly deviate from the original sentence Sn⁢o⁢r⁢m⁢a⁢lsubscriptS_normalSitalic_n o r m a l. The constraint δ guarantees that while aiming for high obfuscation, the alterations remain within reasonable bounds, thus avoiding excessive unrelated output. It is important to note that in our definition, O⁢B⁢()OB()O B ( ) utilizes the edit distance between syntax tree strings, whereas De⁢d⁢i⁢tsubscriptD_editDitalic_e d i t refers merely to the textual string edit distance. This distinction allows us to achieve the maximum syntactic divergence with minimal edits to the text, ensuring that changes in word usage are kept to a minimum while making the sentence semantically challenging to understand. 5.2.2 Implementation The overview of OI achieving the purpose of obscure intention is shown in Figure 2. The attacker designs a obscure algorithm in a targeted manner based on the target information. As shown in Figure 2, the preset reasonable intention prompt and illegal intention text are input into the OI tool and the pseudo-legal output is hint. Figure 2: The Overview of OI jailbreak. 5.2.3 Obscure Intention Prompt Generation This section details generating obscured templates for embedding malicious content in prompts. The algorithm obscures initial intentions in seed sentences, creating templates suitable for subsequent malicious insertion and transforming simple inputs into means for malicious utilization. Following is the pseudocode for ’Generate Obscure Candidate Template,’ designated as Algorithm 1. Algorithm 1 Generate Obscure Candidate Template 0: sis⁢e⁢e⁢d∈Ss⁢e⁢e⁢dsubscriptsuperscriptsuperscripts^seed_i∈ S^seedsitalic_s e e ditalic_i ∈ Sitalic_s e e d, t⁢e⁢m⁢p⁢l⁢a⁢t⁢etemplatet e m p l a t e 1: So⁢r⁢i⁢g⁢i⁢n⁢a⁢lG⁢A←G⁢A⁢(Ss⁢e⁢e⁢d)←subscriptsuperscriptsuperscriptS^GA_original← GA(S^seed)Sitalic_G Aitalic_o r i g i n a l ← G A ( Sitalic_s e e d ) 2: Sm⁢a⁢r⁢k⁢e⁢rG⁢A←I⁢n⁢s⁢e⁢r⁢t⁢M⁢a⁢r⁢k⁢e⁢r⁢(So⁢r⁢i⁢g⁢i⁢n⁢a⁢lG⁢A)←subscriptsuperscriptsubscriptsuperscriptS^GA_marker← InsertMarker(S^GA_original)Sitalic_G Aitalic_m a r k e r ← I n s e r t M a r k e r ( Sitalic_G Aitalic_o r i g i n a l ) 3: ST⁢e⁢m⁢p⁢l⁢a⁢t⁢e←A⁢p⁢p⁢l⁢y⁢T⁢e⁢m⁢p⁢l⁢a⁢t⁢e⁢s⁢(Sm⁢a⁢r⁢k⁢e⁢rG⁢A,t⁢e⁢m⁢p⁢l⁢a⁢t⁢e)←superscriptsubscriptsuperscriptS^Template← ApplyTemplates(S^GA_marker,template)Sitalic_T e m p l a t e ← A p p l y T e m p l a t e s ( Sitalic_G Aitalic_m a r k e r , t e m p l a t e ) 4: Sf⁢i⁢l⁢t⁢e⁢r⁢e⁢dT⁢e⁢m⁢p⁢l⁢a⁢t⁢e←F⁢i⁢l⁢t⁢e⁢r⁢(ST⁢e⁢m⁢p⁢l⁢a⁢t⁢e,C⁢h⁢a⁢t⁢M⁢o⁢d⁢e⁢l)←superscriptsubscriptsuperscriptℎS_filtered^Template← Filter(S^Template,ChatModel)Sitalic_f i l t e r e ditalic_T e m p l a t e ← F i l t e r ( Sitalic_T e m p l a t e , C h a t M o d e l ) 4: ST⁢e⁢m⁢p⁢l⁢a⁢t⁢ef⁢i⁢l⁢t⁢e⁢r⁢e⁢dsubscriptsuperscriptS^filtered_TemplateSitalic_f i l t e r e ditalic_T e m p l a t e The algorithm 1 uses seed to generate available jailbreak templates in batches. The seed can randomly select legal short sentences within 20 words. First, the genetic algorithm GA is used to mutate the seed to a certain extent, and the optimal solution generated in each generation is deduplicated and saved as So⁢r⁢i⁢g⁢i⁢n⁢a⁢lG⁢AsubscriptsuperscriptS^GA_originalSitalic_G Aitalic_o r i g i n a l. Then, for each sentence in the sentence set So⁢r⁢i⁢g⁢i⁢n⁢a⁢lG⁢AsubscriptsuperscriptS^GA_originalSitalic_G Aitalic_o r i g i n a l, a mark symbol INSERTQUESTION is inserted, which is used to indicate the alternative position of the question in the sentence. For each sentence in the marked sentence set Sm⁢a⁢r⁢k⁢e⁢rG⁢AsubscriptsuperscriptS^GA_markerSitalic_G Aitalic_m a r k e r, a prefabricated template is applied to embed it to generate a preselected jailbreak template set. Finally, the interaction with LLM model is used to perform preliminary filtering on the existing pre-selected jailbreak templates. The filtering method uses a method of embedding a simple malicious question (such as how to destroy the world) into the template. If the jailbreak effect can be achieved on the LLM model, the jailbreak template is considered valid and retained. With the algorithm 1, we can use seeds to batch generate usable jailbreak templates. Algorithm 2 Obscure Intention Genetic Algorithm 0: sis⁢e⁢e⁢d∈Ss⁢e⁢e⁢dsubscriptsuperscriptsuperscripts^seed_i∈ S^seedsitalic_s e e ditalic_i ∈ Sitalic_s e e d 1: P←I⁢N⁢I⁢Tp⁢o⁢p⁢u⁢l⁢a⁢t⁢i⁢o⁢n⁢(Ss⁢e⁢e⁢d,S⁢i⁢z⁢e)←subscriptsuperscriptP← INIT_population(S^seed,Size)P ← I N I Titalic_p o p u l a t i o n ( Sitalic_s e e d , S i z e ) 2: for i⁢t⁢e⁢r⁢a⁢t⁢i⁢o⁢niterationi t e r a t i o n in 1:Nm⁢a⁢x:1subscript1:N_max1 : Nitalic_m a x do 3: for each i∈Pi∈ Pi ∈ P do 4: E⁢v⁢a⁢l⁢u⁢a⁢t⁢e⁢F⁢i⁢t⁢n⁢e⁢s⁢s⁢(i)EvaluateFitness(i)E v a l u a t e F i t n e s s ( i ) 5: end for 6: p⁢a⁢r⁢e⁢n⁢t⁢s←S⁢e⁢l⁢e⁢c⁢t⁢(P)←parents← Select(P)p a r e n t s ← S e l e c t ( P ) 7: g⁢e⁢n⁢e⁢r⁢a⁢t⁢i⁢o⁢n⁢e⁢x⁢t←H⁢y⁢b⁢r⁢i⁢d⁢V⁢a⁢r⁢i⁢a⁢t⁢i⁢o⁢n⁢(p⁢a⁢r⁢e⁢n⁢t⁢s)←subscriptgeneration_next← HybridVariation(parents)g e n e r a t i o nitalic_n e x t ← H y b r i d V a r i a t i o n ( p a r e n t s ) 8: P=g⁢e⁢n⁢e⁢r⁢a⁢t⁢i⁢o⁢n⁢e⁢x⁢tsubscriptP=generation_nextP = g e n e r a t i o nitalic_n e x t 9: end for 10: So⁢r⁢i⁢g⁢i⁢n⁢a⁢lG⁢A←P⁢o⁢p⁢u⁢l⁢a⁢t⁢i⁢o⁢n←subscriptsuperscriptS^GA_original← PopulationSitalic_G Aitalic_o r i g i n a l ← P o p u l a t i o n 10: So⁢r⁢i⁢g⁢i⁢n⁢a⁢lG⁢AsubscriptsuperscriptS^GA_originalSitalic_G Aitalic_o r i g i n a l Algorithm 2 is based on the genetic algorithm with a certain degree of detail adjustment to batch generate obscured sentences from seeds. The process begins with the initialization of the population, designed in accordance with the optimization objectives outlined in formula 5.2.1, followed by fitness calculations using the objective function described in 5.2.3. RO⁢B⁢(s)subscript R_OB(s)Ritalic_O B ( s ) = == O⁢B⁢(s)m⁢a⁢x⁢(0⁢(s⁢t),0⁢(s⁢ts⁢e⁢e⁢d))subscript0subscriptsubscript0subscriptsuperscript OB(s)max(L_0(s_st),L_0(s^seed_% st))divide start_ARG O B ( s ) end_ARG start_ARG m a x ( blackboard_L0 ( sitalic_s t ) , blackboard_L0 ( sitalic_s e e ditalic_s t ) ) end_ARG RL⁢(s)subscript R_L(s)Ritalic_L ( s ) = == ⁢(s,s⁢e⁢e⁢d)m⁢a⁢x⁢(0⁢(s),0⁢(s⁢e⁢e⁢d))superscriptsubscript0subscript0superscript L(s,s^seed)max(L_0(s),L_0% (s^seed))divide start_ARG blackboard_L ( s , sitalic_s e e d ) end_ARG start_ARG m a x ( blackboard_L0 ( s ) , blackboard_L0 ( sitalic_s e e d ) ) end_ARG Fs⁢c⁢o⁢r⁢e⁢(s)subscript F_score(s)Fitalic_s c o r e ( s ) = == RO⁢B⁢(s)×w1+(1−RL⁢(s))×w2subscriptsubscript11subscriptsubscript2 R_OB(s)× w_1+(1-R_L(s))× w_2Ritalic_O B ( s ) × w1 + ( 1 - Ritalic_L ( s ) ) × w2 (24) Where s⁢tsubscripts_stsitalic_s t represents the syntax tree string of sentence s and s⁢ts⁢e⁢e⁢dsubscriptsuperscripts^seed_stsitalic_s e e ditalic_s t denotes the syntax tree string of the seed string s⁢e⁢e⁢dsuperscripts^seedsitalic_s e e d in the genetic algorithm, RO⁢B⁢(s⁢t)subscriptsubscriptR_OB(s_st)Ritalic_O B ( sitalic_s t ) quantifies the grammatical obscure ratio, and RL⁢(s,s⁢e⁢e⁢d)subscriptsuperscriptR_L(s,s^seed)Ritalic_L ( s , sitalic_s e e d ) measures the ratio of sentence edit distance. Here, ⁢(s,s⁢e⁢e⁢d)superscriptL(s,s^seed)blackboard_L ( s , sitalic_s e e d ) denotes the Levenshtein Distance between string s and s⁢e⁢e⁢dsuperscripts^seedsitalic_s e e d, while 0⁢(s)subscript0L_0(s)blackboard_L0 ( s ) represents the Levenshtein Distance between the string s and an empty string. Additionally, Fs⁢c⁢o⁢r⁢e⁢(s)subscriptF_score(s)Fitalic_s c o r e ( s ) signifies the fitness score of sentence s. In this process, individuals that achieve the highest level of grammatical obfuscation with minimal sentence editing are more likely to survive. Through the fitness score, all individuals are selected by roulette. Notably, to prevent loss of the optimal solution during mutation and crossover, an elite strategy is employed: the top-performing individual from each iteration is automatically retained for the next generation. After the above selection process, the surviving individuals increase the diversity of sentences through the cross mutation process. Mutation strategies implemented in this study include duplication, swap, and deletion. In order to avoid generating words that are too redundant, in the deactivation strategy, this paper sets Words with more repetitions have a higher probability of deactivation. In the crossover process, we adopts two strategies:matched crossover and random crossover. Matched crossover means only exchanging sub-sentences with the same first and last words in the two sentences. Random crossover means randomly exchanging sub-sentences in the two sentences. It is worth noting that in order to reduce the oscillation of convergence performance caused by excessive damage to the sentence grammatical structure during the crossover process, this paper gives priority to the matching crossover strategy. If there is no matching crossover, random crossover is performed. Follow the above method and continue iterating until the maximum number of iterations. Finally, the optimal individuals are retained and the original obscure sentences generated in batches are returned. 5.2.4 Prompt Case To offer a more intuitive depiction of OI’s operations, this section delves into the Prompt generation process for a particular response, using an example from one of the IntentObfuscator instances. Figure 3: A case of OI jailbreak. As shown in Figure 3, OI forms an attack involving Seed, Templates, Problem with Malicious Intention, Generated Obscure Prompt and response forbidden contents. We re-edit the seed to generate templates, embed the malicious intention problem into the templates to generate an obscured prompt to feed to the LLM, and a successful jailbreak will induce the LLM to return a response containing forbidden content. In the design of preset templates for the OI method, a targeted approach was adopted, keeping the malicious question unaltered, which underscores the significance of the template’s design for the method’s effectiveness. The Prefabricated Template is divided into three distinct components: Content Creation Command Prefix, which directs the system to perform content generation tasks, steering clear of other activities; Template, the core produced by OI algorithm, crucial for the jailbreak’s success; and Output Control Commands, designed to regulate the system’s content output. Within the OI method, the template generated by the algorithm is vital for the effectiveness of the jailbreak, with the content creation command prefix and output control commands playing supporting roles in guiding and refining the process. In Figure LABEL:fig_tree_a, the syntax tree of SEED is depicted, showcasing a structure with complete sentence components and a logical order, thus facilitating easy comprehension of its meaning. Contrastingly, Figure LABEL:fig_tree_b displays a variant tree produced via our enhanced genetic algorithm-based editing. This editing process often results in syntax trees that are more chaotic and obscured, characterized by templates with repetitive or disordered grammatical elements. (a) (b) Figure 4: Comparison of syntax tree before and after editing 5.3 Create Ambiguity The purpose of CA(Create Ambiguity) is to make the LLM content restriction review unable to clarify the attacker’s target intentions. The principle of CA achieving the purpose of obfuscation is shown in Figure 5. The attacker designs a obscure algorithm in a targeted manner based on the target information. As shown in Figure 5, the preset reasonable intention prompt and illegal intention text are input into the CA tool and the pseudo-legal output is hint. 5.3.1 Theoretical Analysis Suppose we are in a theoretical scenario where we want to try to bypass the security review of a language model. One possible method is to use ambiguous language to obscure potential security risks, making them difficult for reviewers to detect. Definition of Ambiguity Sentence A query Qa⁢msubscriptQ_amQitalic_a m is defined as an ambiguous sentence if only: |Qa⁢m|=1∧||>1∧∀si,sj∈,i≠j,LLMtarget⁢(si)≠LLMtarget⁢(sj)formulae-sequencesubscript11for-allsubscriptformulae-sequencesubscriptformulae-sequencesubscriptLLMtargetsubscriptsubscriptLLMtargetsubscript|Q_am|=1 |Q|>1 ∀ s_i,s_j ,i≠ j,% LLM_target(s_i) _target(s_j)| Qitalic_a m | = 1 ∧ | Q | > 1 ∧ ∀ sitalic_i , sitalic_j ∈ Q , i ≠ j , LLMtarget ( sitalic_i ) ≠ LLMtarget ( sitalic_j ) where |Qa⁢m|=1subscript1|Q_am|=1| Qitalic_a m | = 1 indicates that the query Qa⁢msubscriptQ_amQitalic_a m contains exactly one sentence, ||>11|Q|>1| Q | > 1 signifies that this sentence can be decomposed into more than one sub-sentence by the internal logic of LLM. Definition of Malicious Intent Ambiguous Query A malicious intent ambiguous sentence Qa⁢me⁢v⁢a⁢lsuperscriptsubscriptQ_am^evalQitalic_a mitalic_e v a l is defined as: Qa⁢me⁢v⁢a⁢l≡Qa⁢m∧∃sie⁢v⁢a⁢l∈,tox⁢(sie⁢v⁢a⁢l)>θ∧tox⁢(LLMtarget⁢(sie⁢v⁢a⁢l))>θformulae-sequencesuperscriptsubscriptsubscriptsuperscriptsubscripttoxsuperscriptsubscripttoxsubscriptLLMtargetsuperscriptsubscript Q_am^eval≡ Q_am ∃ s_i^eval % ,tox(s_i^eval)>θ (LLM_target(s_% i^eval))> _a mitalic_e v a l ≡ Qitalic_a m ∧ ∃ sitalic_iitalic_e v a l ∈ Q , tox ( sitalic_iitalic_e v a l ) > θ ∧ tox ( LLMtarget ( sitalic_iitalic_e v a l ) ) > θ (25) where sie⁢v⁢a⁢lsuperscriptsubscripts_i^evalsitalic_iitalic_e v a l denotes a sub-sentence in QQ that contains malicious content, tox⁢(sie⁢v⁢a⁢l)>θtoxsuperscriptsubscripttox(s_i^eval)> ( sitalic_iitalic_e v a l ) > θ indicates that the toxicity level of the sub-sentence exceeds a predefined threshold θ, and tox⁢(LLMtarget⁢(sie⁢v⁢a⁢l))>θtoxsubscriptLLMtargetsuperscriptsubscripttox(LLM_target(s_i^eval))> ( LLMtarget ( sitalic_iitalic_e v a l ) ) > θ ensures that the model’s target output for the malicious sub-sentence also exceeds the toxicity threshold, emphasizing the sentence’s potential to cause harm or convey malintent. As similar to the computations discussed in Section 5.2.1, we can deduce that when the ambiguous sentence Qa⁢me⁢v⁢a⁢lsuperscriptsubscriptQ_am^evalQitalic_a mitalic_e v a l is sufficiently difficult to understand, that is Ob⁢(Qa⁢me⁢v⁢a⁢l)>τObsuperscriptsubscriptOb(Q_am^eval)> ( Qitalic_a mitalic_e v a l ) > τ, the effective response rate Re⁢f⁢f⁢(sie⁢v⁢a⁢l,Qa⁢me⁢v⁢a⁢l)subscriptsuperscriptsubscriptsuperscriptsubscriptR_eff(s_i^eval,Q_am^eval)Ritalic_e f f ( sitalic_iitalic_e v a l , Qitalic_a mitalic_e v a l ) can be calculated as follows: Re⁢f⁢f⁢(sie⁢v⁢a⁢l,Qa⁢me⁢v⁢a⁢l)subscriptsuperscriptsubscriptsuperscriptsubscript R_eff(s_i^eval,Q_am^eval)Ritalic_e f f ( sitalic_iitalic_e v a l , Qitalic_a mitalic_e v a l ) = == S⁢i⁢m⁢(L⁢L⁢Mq⁢u⁢e⁢r⁢y⁢(Qa⁢me⁢v⁢a⁢l),L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(sie⁢v⁢a⁢l))subscriptsuperscriptsubscriptsubscriptsuperscriptsubscript Sim(LLM_query(Q_am^eval),LLM_target(s_i^eval))S i m ( L L Mitalic_q u e r y ( Qitalic_a mitalic_e v a l ) , L L Mitalic_t a r g e t ( sitalic_iitalic_e v a l ) ) (26) = == Sim(LLMq⁢u⁢e⁢r⁢ys⁢u⁢b(sie⁢v⁢a⁢l)+∑tox⁢(sj)<θLLMq⁢u⁢e⁢r⁢ys⁢u⁢b(sj), Sim(LLM_query^sub(s_i^eval)+ _tox(s_j)<% θLLM_query^sub(s_j),S i m ( L L Mitalic_q u e r yitalic_s u b ( sitalic_iitalic_e v a l ) + ∑tox ( s start_POSTSUBSCRIPT j ) < θ end_POSTSUBSCRIPT L L Mitalic_q u e r yitalic_s u b ( sitalic_j ) , LLMt⁢a⁢r⁢g⁢e⁢t(sie⁢v⁢a⁢l)) LLM_target(s_i^eval))L L Mitalic_t a r g e t ( sitalic_iitalic_e v a l ) ) = == l⁢e⁢n⁢(L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(sie⁢v⁢a⁢l))l⁢e⁢n⁢(L⁢L⁢Mt⁢a⁢r⁢g⁢e⁢t⁢(sie⁢v⁢a⁢l)+∑tox⁢(sj)<θL⁢L⁢Mq⁢u⁢e⁢r⁢ys⁢u⁢b⁢(sj))subscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscripttoxsubscriptsuperscriptsubscriptsubscript len(LLM_target(s_i^eval))len(LLM_target(s_i^% eval)+ _tox(s_j)<θLLM_query^sub(s_j))divide start_ARG l e n ( L L Mitalic_t a r g e t ( sitalic_iitalic_e v a l ) ) end_ARG start_ARG l e n ( L L Mitalic_t a r g e t ( sitalic_iitalic_e v a l ) + ∑tox ( s start_POSTSUBSCRIPT j ) < θ end_POSTSUBSCRIPT L L Mitalic_q u e r yitalic_s u b ( sitalic_j ) ) end_ARG The computational results indicate that by crafting ambiguous sentences containing malicious content, it is possible to manipulate the model’s output to also contain malicious elements. Theoretical analysis suggests that, compared to outcomes in OI, the CA approach may result in a higher proportion of hallucinatory outputs. The degree of hallucination depends on the ambiguity level of sn⁢o⁢r⁢m⁢a⁢lsuperscripts^normalsitalic_n o r m a l(sn⁢o⁢r⁢m⁢a⁢l=∀sj,t⁢o⁢x⁢(sj)<θformulae-sequencesuperscriptfor-allsubscriptsubscripts^normal=∀ s_j,tox(s_j)< _n o r m a l = ∀ sitalic_j , t o x ( sitalic_j ) < θ). This implies that when the language model attempts to dissect and comprehend an ambiguous sentence, and if the normal sub-sentence still fails to guide the model effectively, then the proportion of hallucinatory output is likely to decrease. Figure 5: The Overview of CA jailbreak. 5.3.2 Ambiguity Prompt Generation Algorithm 3 Create Ambiguity Prompt 0: sn⁢o⁢r⁢msubscripts_normsitalic_n o r m, Ii⁢l⁢l⁢e⁢g⁢a⁢lsubscriptI_illegalIitalic_i l l e g a l 1: si⁢l⁢l⁢e⁢g⁢a⁢la⁢m⁢b⁢i⁢g⁢u⁢i⁢t⁢y←R⁢e⁢W⁢r⁢i⁢t⁢e⁢(si⁢l⁢l⁢e⁢g⁢a⁢lI⁢n⁢t⁢e⁢n⁢t,a⁢m⁢b⁢i⁢g⁢u⁢i⁢t⁢y)←subscriptsuperscriptsubscriptsuperscripts^ambiguity_illegal← ReWrite(s^Intent_illegal,ambiguity)sitalic_a m b i g u i t yitalic_i l l e g a l ← R e W r i t e ( sitalic_I n t e n titalic_i l l e g a l , a m b i g u i t y ) 2: repeat 3: si⁢l⁢l⁢e⁢g⁢a⁢l←E⁢m⁢b⁢e⁢d⁢(sn⁢o⁢r⁢m⁢a⁢lT,si⁢l⁢l⁢e⁢g⁢a⁢la⁢m⁢b⁢i⁢g⁢u⁢i⁢t⁢y)←subscriptsubscriptsuperscriptsubscriptsuperscripts_illegal← Embed(s^T_normal,s^ambiguity_illegal)sitalic_i l l e g a l ← E m b e d ( sitalic_Titalic_n o r m a l , sitalic_a m b i g u i t yitalic_i l l e g a l ) 4: until E⁢v⁢a⁢l⁢(si⁢l⁢l⁢e⁢g⁢a⁢l)≤δsubscriptEval(s_illegal)≤ v a l ( sitalic_i l l e g a l ) ≤ δ 4: si⁢l⁢l⁢e⁢g⁢a⁢lsubscripts_illegalsitalic_i l l e g a l Based on our theoretical analysis, we identified the crucial aspect of generating ambiguous sentences that LLMs can comprehend. Specifically, an effective ambiguous sentence should be such that the LLM can decompose it into multiple sub-sentences, each bearing a different meaning, and recognize the sentence as inherently ambiguous. To achieve this, we have opted to let the large language model itself undertake this task. Thus, we have designed prompts that encourage the generation of maliciously ambiguous sentences by the LLM itself. As illustrated in Algorithm 3, we perform a one-to-many ambiguity transformation on malicious intent text, obtaining multiple ambiguous expressions si⁢l⁢l⁢e⁢g⁢a⁢la⁢m⁢b⁢i⁢g⁢u⁢i⁢t⁢ysubscriptsuperscripts^ambiguity_illegalsitalic_a m b i g u i t yitalic_i l l e g a l. Subsequently, we combine predefined normal intent statement templates sn⁢o⁢r⁢m⁢a⁢lTsubscriptsuperscripts^T_normalsitalic_Titalic_n o r m a l with the rewritten ambiguous malicious intent statements si⁢l⁢l⁢e⁢g⁢a⁢la⁢m⁢b⁢i⁢g⁢u⁢i⁢t⁢ysubscriptsuperscripts^ambiguity_illegalsitalic_a m b i g u i t yitalic_i l l e g a l to form pseudo-legitimate prompts si⁢l⁢l⁢e⁢g⁢a⁢lsubscripts_illegalsitalic_i l l e g a l. 5.3.3 Prompt Case In order to more intuitively illustrate the working mode of the CA instance, this section describes in detail the generation of a specific Prompt, response as an example. Figure 6: A case of CA jailbreak. The Figure 6 shows the text involved in CA’s pseudo-legal prompt generation, including template, malicious questions, malicious ambiguity questions, pseudo-legal prompts, and LLM responses containing target forbidden content. Malicious questions with ambiguity are obtained after LLM or manual processing. After embedding the template, a pseudo-legal prompt is generated. After being fed to LLM, text containing forbidden content is generated. Figure 7: Semantic Understanding of the CA Ambiguity Problem Compared with Origin Problem. The design strategy for the CA method’s prefabricated templates focuses on the potential for inherent ambiguity in malicious questions to initiate a jailbreak effect. Unlike the OI method, which relies on a meticulously crafted template to alter the perception of a prompt without changing the question itself, the CA method evaluates the effectiveness of ambiguity in achieving a jailbreak.For this reason, the prefabricated template in the CA method aligns with the baseline approach, ensuring consistent control variables for a straightforward comparative analysis. This approach allows us to assess whether the deliberate introduction of semantic ambiguity can surpass the baseline’s effectiveness in prompting jailbreak attacks. In this section, the LLM generation method is used to construct ambiguity for Problem. After setting the prompt for constructing ambiguity, multiple different ambiguity statements can be obtained. As shown in Figure 7, the original question can only be understood as safety education for children on gun use and purchase; however, the ambiguous sentences after constructing ambiguity through LLM may have certain obstacles in reading, but they also have multiple semantic understandings. : Interpretation 1: This may be an emphasis on teaching children how to use and access guns; Interpretation 2: This may also be misinterpreted as encouraging children to use and obtain firearms. 6 Experiments and Analysis 6.1 Experiment environment 6.1.1 LLMs for evaluation We choose advanced commercial language models as targets for attack: ChatGPT-3.5 (gpt-3.5-turbo )(OpenAI, Inc, a), ChatGPT-4(OpenAI, Inc, b), qwen(qwen-max)(Bai et al., 2023) and baichuan(baichuan2-13b-chat-v1)(Yang et al., 2023). For these models, we chose the commercial version as the experimental object, because they have stricter security measures, which can better illustrate IntentObfuscator’s real jailbreaking capabilities. 6.1.2 Baseline The baseline methodology employed in this study is grounded in state-of-the-art manual engineering techniques(Alexalbert, 2023) for constructing jailbreak attacks. This approach was selected through a process of manual verification, identifying the effective manual jailbreak template featured at the top of “jailbreakchat” website. Although these methods are inherently manual and lack automation capabilities, they are integral for forming a baseline dataset. The Harmful Behavior Problems (HBP) are directly incorporated into this carefully selected template to establish a baseline. The outcomes derived from this baseline serve as a reference for assessing attack impacts. 6.1.3 Methods for comparison In order to illustrate the effectiveness and advancement of the method proposed in this article, the two latest and most representative LLM jailbreak attack methods published in 2023 were selected: the Greedy Coordinate Gradient (GCG) method proposed by (Zou et al., 2023); (Jiang et al., 2023) proposed the Compositional Instruction Attacks (CIA) method, which disguises harmful instructions as oral or written tasks and requires the attacker to have senior experience.It is important to note that the GCG method is a white-box approach and cannot be directly applied to attack commercial black-box models; hence, we utilize its transfer attack methodology for comparison. 6.1.4 Datasets Preparation In order to better validate our experimental results and facilitate comparison with the baseline and other researchers’ work, we utilized the widely-used open dataset Harmful Behavior Problems (HBP)(Zou et al., 2023). This section outlines the data construction process. Dataset HBP We utilize the openly available HBP dataset(Zou et al., 2023), encompassing 520 malicious questions that commercial LLMs prohibit users from querying, accessible via the provided link111https://github.com/llm-attacks/llm-attacks/blob/main/data/advbench/harmful_ behaviors.csv. These include a variety of malicious instructions, such as requests for assistance in criminal activities or cyberattacks. Datasets for OI To construct the OI validation dataset, we created 60 initial seed sentences representing normal intentions. These seeds underwent a genetic algorithm, detailed in Section 5.2.3, resulting in 600 variant templates. Each problem from the HBP was then combined with a mutation template, generating 312,000 candidate prompts. After filtering using the prompt evaluation method, 520 high-quality malicious prompts were selected for OI jailbreak attack validation, as depicted in Table 3. Datasets for CA For the CA validation dataset, we utilized the 520 harmful behavior issues from the HBP as initial inputs. Each problem underwent 10 ambiguous template mutations via LLM, resulting in 5,200 ambiguous outputs. These outputs were combined with pre-designed normal intent prompt templates, yielding 5,200 candidate prompts. Employing the evaluation criteria outlined in Section 5.3.2, we filtered these candidates to obtain 520 qualified CA jailbreak attack prompts, as demonstrated in Table 3. Table 3: Datasets used for OI and CA attack verification Process Data OI Data Volume CA Data Volume Seeds 60 - Templates 600 5200 Ambiguity Problems - 520 Origin Problems 520 520 Candidates 312000 5200 PL Prompts 520 520 To further analyze the jailbreak attack effect of IntentObfuscator on different sensitive content, we divided the data into seven categories based on the list of issues: blood, ethics, racial discrimination, sexism, political sensitivity, cyber security and criminal skills. The data details are shown in the Table 4. The classification results are uneven, which is due to the uneven distribution of hotspots of jailbreak attack content. Table 4: Datasets used for OI and CA verification Content Classes Data Volume Ratio Bloody 38 7.31% Ethics 127 24.42% Racism 7 1.34% Sexism 5 0.96% Politics 12 2.31% Cyber Security 169 32.5% Criminal Skills 162 31.15% 6.1.5 Evaluation Metrics The criteria for judging a successful attack are as shown in Eq. 2 in section 4.1. Let N denote the total number of harmful prompts, NrsubscriptN_rNitalic_r represents the corresponding number of rejected harmful prompts, NhsubscriptℎN_hNitalic_h denotes the number of hallucinated responses, and NssubscriptN_sNitalic_s indicates the number of successful attacks. Based on these definitions, we define Rejected Rate (REJ) as R⁢E⁢J=NrNsubscriptREJ= N_rNR E J = divide start_ARG Nitalic_r end_ARG start_ARG N end_ARG, Attack Success Rate (ASR) as A⁢S⁢R=NsNsubscriptASR= N_sNA S R = divide start_ARG Nitalic_s end_ARG start_ARG N end_ARG, and Hallucination (HAL) as H⁢A⁢L=NhNsubscriptℎHAL= N_hNH A L = divide start_ARG Nitalic_h end_ARG start_ARG N end_ARG. 6.2 Results Analysis of Jailbreak Attack 6.2.1 Attack effects on different LLMs Table 5: Attack ASR on different LLMs Models Baseline OI ASR CA ASR Average ChatGPT-3.5 69.04% 82.12% 85.19% 83.65% ChatGPT-4 46.15% 56.15% 50.38% 53.27% qwen-max 25.77% 55.19% 35.19% 45.19% baichuan2-13b- chat-v1 97.69% 94.62% 94.81% 94.71% Average 59.66% 72.02% 66.39% 69.21% To ensure robust experimental evaluation, we selected multiple models for our study. Our findings indicate that breaking through LLM security measures isn’t overly challenging for skilled jailbreakers. Vulnerable open-source LLMs like baichuan2-13b-chat-v1 are particularly susceptible, while commercial models like ChatGPT-3.5, ChatGPT-4, and qwen-max vary in defense effectiveness. Qwen-max shows the highest protection capability, 25.77% ASR, followed by ChatGPT-4, 46.15% ASR and ChatGPT-3.5, 69.04% ASR. Baichuan2-13b-chat-v1’s security measures are ineffective against manual jailbreakers. OI significantly enhances jailbreak capabilities, with qwen-max having the lowest ASR, 55.19% and baichuan2-13b-chat-v1 the highest, 94.62%. On average, ASR increases by 12.36% compared to baseline, with OI achieving higher success rates on commercial LLMs. CA shows promise on ChatGPT-3.5, 85.19% but performs variably on other models. For baichuan2-13b-chat-v1, both methods show high ASR, indicating weak content security. Despite OI’s lower ASR, it maintains effective jailbreak capabilities, while CA offers higher breakthrough potential despite sacrificing some performance. Overall, the IntentObfuscator-based approach proves to be effective across different LLMs, demonstrating strong jailbreak success rates and enhancing security testing capabilities. GPT-3.5GPT-4qwenbaichuan101010102525252540404040555555557070707085858585100100100100RatioASRHALREJ (a) GPT-3.5GPT-4qwenbaichuan101010102525252540404040555555557070707085858585100100100100RatioASRHALREJ (b) GPT-3.5GPT-4qwenbaichuan101010102525252540404040555555557070707085858585100100100100RatioASRHALREJ (c) Figure 8: Jailbreak Attack Results To visually compare the attack effects across different models, Fig. 8 presents baseline methods assessing Language Models’ (LMs) defense capabilities, with qwen-max leading with a 74% REJ. ChatGPT-4 improves defense compared to ChatGPT-3.5, reaching REJ of 51% and 31%, respectively. HALs remain low, with ChatGPT-4 at 2.8%. In OI, the average REJ decreases to 11%, a 56% reduction from baseline, but introduces an HAL of 17%, peaking at 30% for GPT models, as depicted in Fig. 8(a). CA lowers the average REJ to 26%, notably on ChatGPT-3.5, down by 21% from baseline. CA introduces an 8% HAL, peaking at 17% on qwen-max, as illustrated in Fig. 8(b). In summary, our OI method effectively bypasses the defense strategies of various models but introduces a higher number of illusionary responses. Conversely, the CA method introduces fewer illusionary responses but exhibits relatively weaker jailbreaking capabilities compared to the OI method. 6.2.2 Comparison with the Latest Automated Jailbreak Methods GPT-3.5GPT-4qwenbaichuan0050505050100100100100150150150150200200200200250250250250300300300300Stacked ASROICAGCGCIA (a) OICAGCGCIA0050505050100100100100150150150150200200200200250250250250300300300300Stacked ASRGPT-3.5GPT-4qwenbaichuan (b) Figure 9: Comparison jailbreak methods on different LLMs To validate the effectiveness of our method, we further compared IntentObfuscator’s two attack instances against GCG and CIA jailbreak methods on GPT-3.5, GPT-4, Qwen, and Baichuan models, among others. In Fig. 9, we can see that assessing CIA methodology across models reveals significant disparities. While it achieves higher ASR on “baichuan”, other models show mediocre results. Conversely, GCG consistently exhibits lower ASR, notably in GPT-3.5 and GPT-4. In contrast, OI and CA methodologies demonstrate competitive ASR. Their superiority over CIA and GCG is evident in GPT-3.5 and GPT-4. Notably, CA achieves 67.57% ASR in GPT-3.5, surpassing CIA’s 33.65% and GCG’s 10.81%. Similarly, OI and CA outperform in GPT-4. GCG method exhibited a lower success rate in our replication efforts, which we believe can be attributed to several factors. Firstly, GCG is inherently a white-box method, which can only be applied to attack current proprietary commercial large language models through a transfer process. As commercial models evolve with upgraded defense mechanisms, they can effectively block such attacks, rendering the GCG method inoperative. Secondly, the adversarial suffix approach is somewhat fragile, requiring only minimal disturbances to be rendered ineffective. In summary, OI and CA methodologies excel compared to CIA and GCG, particularly in GPT-3.5 and GPT-4. This highlights their efficacy in addressing diverse model architectures, providing a reliable solution for jailbreaking LLMs. 6.2.3 Attack effects with different forbidden contents on different models Table 6: Effects of Jailbreak Attacks with Different Forbidden Scenarios on Different Models Forbidden Content ChatGPT-3.5 ChatGPT-4 qwen-max baichuan2-13b-chat-v1 Average OI ASR CA ASR Average Baseline OI ASR CA ASR Average Baseline OI ASR CA ASR Average Baseline OI ASR CA ASR Average Baseline Bloody 68.42 % 86.84 % 77.63 % 63.16 % 68.42 % 55.26 % 52.63 % 31.58 % 44.74 % 42.11 % 43.42 % 15.79 % 92.11 % 94.74 % 93.42 % 100.00 % 64.68 % Ethics 77.17 % 76.38 % 76.77 % 60.63 % 77.17 % 44.09 % 49.61 % 46.46 % 53.54 % 36.22 % 44.88 % 25.98 % 87.40 % 91.34 % 89.37 % 96.85 % 65.38 % Racism 57.14 % 42.86 % 50.00 % 57.14 % 57.14 % 71.43 % 50.00 % 42.86 % 28.57 % 42.86 % 35.71 % 14.29 % 100.00 % 100.00 % 100.00 % 100.00 % 58.79 % Sexism 40.00% 100.00% 60.00% 80.00% 20.00% 60.00% 40.00% 0.00% 60.00% 20.00% 40.00% 40.00% 40.00% 100.00% 70.00% 80.00% 57.5% Politics 83.33 % 83.33 % 83.33 % 75.00 % 83.33 % 75.00 % 79.17 % 83.33 % 58.33 % 58.33 % 58.33 % 33.33 % 100.00 % 100.00 % 100.00 % 100.00 % 78.21 % Cyber Security 86.39 % 92.90 % 89.64 % 82.25 % 86.39 % 65.09 % 63.61 % 65.68 % 65.68 % 33.73 % 49.70 % 27.22 % 99.41 % 98.22 % 98.82 % 97.63 % 76.17 % Criminal Skills 87.04 % 85.19 % 86.11 % 62.96 % 87.04 % 37.04 % 44.75 % 27.78 % 48.77 % 31.48 % 40.12 % 26.54 % 96.30 % 93.83 % 95.06 % 98.77 % 66.83 % Average 71.36 % 81.07 % 76.21 % 68.73 % 71.36 % 52.56 % 51.40 % 42.53 % 51.38 % 43.53 % 47.45 % 23.31 % 90.74 % 94.02 % 92.38 % 93.32 % 66.16 % Furthermore, to delve deeper into the impact of jailbreak attacks across various categories, Table 6 highlights ChatGPT-3.5’s superior REJ against ethics-related issues, showing a 39% improvement over baseline, while its cyber security performance lags with only a 17% enhancement. OI notably boosts success rates in criminal skills and ethics by 24% and 16%, respectively. CA excels in violent issues, up 23% from baseline, but lacks impact on discrimination. ChatGPT-4 shines in safeguarding against discrimination, achieving 75% REJ. OI and CA enhance ChatGPT-4’s performance in various categories. qwen-max impresses with an 84% REJ against violence, with OI improving cyber security by 38% and CA enhancing discrimination by 33%. baichuan2-13b-chat-v1 struggles across all categories, with OI and CA introducing slight success rate declines in ethical and moral issues. According to the analysis of the specific results of the attack, we were surprised that with the enhancement of the model’s capability, although the REJ improved, it generated more realistic harmful content. For example, GPT4 produced more genuine phishing web pages or malicious code content than GPT3.5, and provided more accurate guidance on criminal behavior. 6.2.4 Toxicity analysis Toxicity analysis is an important metric for evaluating attacks. This section conducts an in-depth analysis of the toxicity of the IntentObfuscator jailbreak attack through Google API. From the perspective of toxicity analysis, further verification of the underlying reasons for bypassing LLM security defenses in complex queries can be achieved. As shown in Figure 10, subfigures (a) and (b) present the toxicity of LLM inputs and outputs, respectively, in the form of kernel density estimate (KDE) distributions. In subfigure (a), it can be observed that different obfuscation editing operations on prompt inputs can effectively but moderately interfere. This interference weakens the density peak of toxicity from 0.1 in the high-toxicity region to 0.05 or lower, effectively enhancing the ability to bypass LLM security defenses with prompts, thus confirming the vulnerability of LLM to powerful masking of malicious intent in complex prompts. However, the strength of input toxicity is not the primary factor influencing the toxicity intensity of output text. Constrained by the toxicity assessment method we currently employ (in this experiment, we use the Google Toxicity API), the toxicity of input and output text mainly falls within the range of [0.0,0.1]0.00.1[0.0,0.1][ 0.0 , 0.1 ]. Subfigures (c) and (d) present KDE statistics of the lengths of input and output text, respectively, showing that there is no significant correlation between the density of text length in LLM inputs and outputs. Combined with subfigures (a) and (b), it is evident that the strength of toxicity is not directly related to the length of input and output text. (a) (b) (c) (d) Figure 10: The relationship between the toxicity of Prompts and Responses and word density. (a) shows the toxicity distribution of Prompts; (b) shows the toxicity distribution of Responses; (c) is the word density statistics in Prompts; (d) is the word density distribution in Responses. 7 Discussion Prompt jailbreak attacks against LLM are a newly developed attack technique. Recent researches hope to fundamentally solve the risk of prompt injection. Therefore, on the one hand, some researches are committed to exploring the root causes of prompt injection security, and on the other hand, they are exploring thoughts on fine-tuning defense strategies for existing prompt injection. 7.1 Explore the reasons of LLM risk Numerous researchers have advanced hypotheses regarding the limitations of LLMs. A prominent theory proposed by AI scholar Gary Marcus suggests that LLMs are inherently limited in understanding the essence of language due to their absence of a world model. Echoing this sentiment, the Whitzard Team at Fudan University conducted studies that support the idea that LLMs face challenges in comprehending the complexities of human language models. They proposed an innovative approach of mutating prompts while preserving their semantic content, achieving notable success in circumventing LLM restrictions. The security vulnerabilities in prompt injections primarily arise from the fundamental mismatch between the workings of LLMs and human cognition. While both LLMs and human intelligence can produce similar outputs, they operate on fundamentally different internal logics. This critical difference is not just about LLMs’ limited ability to understand human language nuances, but also includes the limited understanding users have of how LLMs function and their limitations. This two-sided inconsistency – the gap in language comprehension by LLMs and the users’ limited grasp of LLM operational dynamics – creates significant security challenges when these forms of intelligence interact. Therefore, the essence of prompt-based security risks is rooted in this dual misalignment. In further exploring the risk of prompt-based jailbreaking attacks, this paper proposes a theoretical framework and identifies two primary manifestations of such attacks, revealing that previous studies also adhere to one of these manifestations. Techniques such as those described in(Shanahan et al., 2023; Deng et al., 2023a; Zou et al., 2023) involve using adversarial suffixes or complex prompt designs to manipulate malicious sentences without altering the sentences themselves, effectively increasing the complexity for LLMs to understand complex malicious queries. This approach essentially raises the difficulty for LLMs to process such queries correctly. Conversely, methods referenced in(Gupta et al., 2023; Zhang et al., 2023) directly modify the malicious sentences themselves, either through encoding or rewriting, thereby increasing the interpretative challenge posed to LLMs. Both approaches demonstrate how different strategies can be employed within the existing theoretical framework to exploit the security vulnerabilities inherent in LLMs when handling prompts. 7.2 Possible Mitigation Strategies for Prompt Injection Attacks Based on the hypotheses, theoretical studies, and experimental analyses presented earlier in this paper, the approaches presented in this paper reveal the limitations of LLMs in recognizing complex malicious queries amidst obfuscation and ambiguity. To address these vulnerabilities, targeted defenses based on the mechanistic flaws identified in our hypotheses can be considered. The following defensive measures could be outlined: 1. Enhanced Detection: Introduce stricter rules to identify and reject vague or ambiguous queries, improving security and encouraging clear user communication. 2. Input Segmentation and Analysis: Reconstruct sentences and extract sub-sentences for individual analysis to enhance the detection of malicious content, thereby improving security. This method also helps prevent malicious sentences from being embedded in longer texts to reduce their apparent maliciousness. 3. Output Verification: Implement checks on output texts to stop the generation of harmful responses, acting as a safeguard and a tool for model improvement. The mitigation strategies for LLM prompt injection attacks, including enhanced detection, input segmentation, and output verification, offer preliminary safeguards. However, their effectiveness is limited. Enhanced detection may inadvertently suppress complex legitimate queries, input segmentation can increase computational load, and output verification, while mitigating immediate risks, does not address deeper vulnerabilities in understanding obfuscated prompts. These limitations highlight the need for more foundational improvements in LLM security. 7.3 Limitations and Future Work of Our Framework While our research attempts to establish a unified theoretical framework to elucidate the principles of prompt-based jailbreaking attacks, it must be acknowledged that, given the complexity of the real world and LLMs themselves, not all aspects can be accounted for. The assumptions and theories presented here are abstractions and simplifications of real-world models. To discuss the principles behind successful jailbreak attacks more accurately, further in-depth research and improvements are necessary to explore potential vulnerabilities in the mechanisms of LLMs. Besides this, our research introduces a novel, lightweight testing tool for red team testers to explore and address risks in LLMs. This approach presents a methodological framework for assessing the security risks associated with LLM prompt injections, instrumental in enabling timely identification and rectification of vulnerabilities in LLM applications. Looking forward, our focus will be on expansive testing across a variety of LLMs to thoroughly assess the applicability and effectiveness of our method. We plan to conduct an in-depth analysis of factors influencing the generation of jailbreak texts, which will provide valuable insights into how LLMs process obfuscated prompts. Furthermore, a key aspect of our future research involves exploring more effective defensive strategies against prompt injection attacks. This will encompass both theoretical advancements and practical implementations, aiming to strengthen the security of LLMs against sophisticated adversarial techniques. In summary, while our work equips security professionals with a practical tool for immediate use, it also lays the groundwork for comprehensive future research into understanding and mitigating vulnerabilities in LLMs. This not only fosters a deeper comprehension of the underlying issues but also promotes the development of robust defenses against emerging threats in the field of machine learning and artificial intelligence. 8 Conclusion LLM Prompt jailbreak research has posed serious security and privacy challenges to mainstream LLM-based interactive services, revealing the diversity and severity of LLM Prompt jailbreak attacks. We have provided a theoretical hypothesis and analysis for understanding the vulnerabilities of LLMs when processing complex prompts, further exploring two specific manifestations of these vulnerabilities. Additionally, we introduced the IntentObfuscator framework, which was designed with two specific techniques, Obscure Intention (OI) and Create Ambiguity (CA), to experimentally validate attacks on these two manifestations. In the OI example, an automated text mutation processing method is proposed, which can generate jailbreak templates in large batches. Compared with existing automated malicious jailbreak template generation methods, our lightweight template generation method has low dependence on computing resources and does not require the use of GPU resources to achieve better generation results. CA only uses two-step conversation to achieve ambiguous statement generation and jailbreak with the API services provided by commercial LLMs, which almost gets rid of the dependence on local computing resources and supports more efficient batch generation of jailbreak prompts. Using public datasets, our experiments confirmed the effectiveness of the IntentObfuscator’s jailbreak mode on leading LLMs. Notably, it achieved ASRs of 83.65% on ChatGPT-3.5, 53.27% on ChatGPT-4, and 45.19% on qwen-max. Considering the results, the IntentObfuscator proves to be a valuable tool for enhancing the capabilities of red team attacks. 9 Acknowledgements This research is supported by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant No. XDC02030200, National Natural Science Foundation of China under Grant No. 62202466 and Youth Innovation Promotion Association CAS and Grand No. 2022159. This research was also supported by Key Laboratory of Network Assessment Technology, Chinese Academy of Sciences, and Beijing Key Laboratory of Network Security and Protection Technology. References Mann et al. [2020] Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. Abcarter [2023] Abcarter. What is the size of the training set for gpt-3, 2023. URL https://community.openai.com/t/what-is-the-size-of-the-training-set-for-gpt-3/360896/1. Chin et al. [2023] Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, and Wei-Chen Chiu. Prompting4debugging: Red-teaming text-to-image diffusion models by finding problematic prompts. arXiv preprint arXiv:2309.06135, 2023. Beckerich et al. [2023] Mika Beckerich, Laura Plein, and Sergio Coronado. Ratgpt: Turning online llms into proxies for malware attacks. arXiv preprint arXiv:2308.09183, 2023. Hazell [2023] Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972, 2023. Weeks et al. [2023] Connor Weeks, Aravind Cheruvu, Sifat Muhammad Abdullah, Shravya Kanchi, Daphne Yao, and Bimal Viswanath. A first look at toxicity injection attacks on open-domain chatbots. In Proceedings of the 39th Annual Computer Security Applications Conference, pages 521–534, 2023. Chen et al. [2023] Bocheng Chen, Guangjing Wang, Hanqing Guo, Yuanda Wang, and Qiben Yan. Understanding multi-turn toxic behaviors in open-domain chatbots. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, pages 282–296, 2023. Google, Inc [2023a] Google, Inc. Google’s secure ai framework (saif). https://safety.google/cybersecurity-advancements/saif/t, 2023a. Openai [2022] Inc Openai. Content policy, 2022. URL https://labs.openai.com/policies/content-policy. Google, Inc [2023b] Google, Inc. Cloud Natural Language. https://cloud.google.com/natural-language/docs/moderating-text, 2023b. Shen et al. [2023] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825, 2023. Shanahan et al. [2023] Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, pages 1–6, 2023. Liu et al. [2023a] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, 2023a. Li et al. [2023] Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197, 2023. Zou et al. [2023] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Alexalbert [2023] Alexalbert. jailbreakchat. https://w.jailbreakchat.com/prompt/b1fe938b-4541-41c8-96e7-b1c659ec4ef9, 2023. Ma et al. [2023] Huan Ma, Changqing Zhang, Huazhu Fu, Peilin Zhao, and Bingzhe Wu. Adapting large language models for content moderation: Pitfalls in data engineering and supervised fine-tuning. arXiv preprint arXiv:2310.03400, 2023. Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Wei et al. [2024] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024. Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Korbak et al. [2023] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR, 2023. Glaese et al. [2022] Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. Rando et al. [2022] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610, 2022. Perez et al. [2022] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. Yao et al. [2023] Dongyu Yao, Jianshu Zhang, Ian G Harris, and Marcel Carlsson. Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models. arXiv preprint arXiv:2309.05274, 2023. Deng et al. [2023a] Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. Attack prompt generation for red teaming and defending large language models. arXiv preprint arXiv:2310.12505, 2023a. Gupta et al. [2023] Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj. From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy. IEEE Access, 2023. Yu et al. [2023] Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023. Deng et al. [2023b] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Jailbreaker: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715, 2023b. Liu et al. [2023b] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023b. Alzantot et al. [2018] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998, 2018. Ren et al. [2019] Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1085–1097, 2019. Wang et al. [2021] Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. Lapid et al. [2023] Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446, 2023. Mei et al. [2023] Alex Mei, Sharon Levy, and William Yang Wang. Assert: Automated safety scenario red teaming for evaluating the robustness of large language models. arXiv preprint arXiv:2310.09624, 2023. Mehrabi et al. [2023] Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. Flirt: Feedback loop in-context red teaming. arXiv preprint arXiv:2308.04265, 2023. Carlini et al. [2024] Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024. Robey et al. [2023] Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023. Lee et al. [2023] Deokjae Lee, JunYeong Lee, Jung-Woo Ha, Jin-Hwa Kim, Sang-Woo Lee, Hwaran Lee, and Hyun Oh Song. Query-efficient black-box red teaming via bayesian optimization. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. Greshake et al. [2023] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pages 79–90, 2023. Deng et al. [2023c] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations, 2023c. Bhardwaj and Poria [2023] Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662, 2023. Jiang et al. [2023] Shuyu Jiang, Xingshu Chen, and Rui Tang. Prompt packer: Deceiving llms through compositional instruction with hidden attacks. arXiv preprint arXiv:2310.10077, 2023. Li et al. [2024] Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers. arXiv preprint arXiv:2402.16914, 2024. OpenAI, Inc [a] OpenAI, Inc. GPT-3.5 Turbo. https://platform.openai.com/docs/mod els/gpt-3-5, a. OpenAI, Inc [b] OpenAI, Inc. GPT-4 and GPT-4 Turbo. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo, b. Bai et al. [2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. Yang et al. [2023] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. Zhang et al. [2023] Mi Zhang, Xudong Pan, and Min Yang. Jade: A linguistics-based safety evaluation platform for llm, 2023.

Instant research discovery