Paper deep dive

LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models

Miao Yu, Junfeng Fang, Yingjie Zhou, Xing Fan, Kun Wang, Shirui Pan, Qingsong Wen

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 68

Models: Gemini-Pro, GPT-3.5-Turbo, GPT-4, Llama-3.1-70B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/12/2026, 6:52:29 PM

Summary

The paper introduces 'LLM-Virus', a novel evolutionary jailbreak attack method for Large Language Models. Inspired by biological viral evolution, it treats jailbreak templates as genetic material and malicious queries as functional proteins. The framework utilizes LLMs as heuristic evolutionary operators to perform mutation and crossover, optimizing for attack success rate, stealthiness, diversity, and conciseness. It frames the attack as a transfer learning problem to improve efficiency and transferability, demonstrating superior performance over existing methods on benchmarks like HarmBench and AdvBench.

Entities (6)

AdvBench · benchmark · 100%Evolutionary Algorithm · optimization-technique · 100%Harmbench · benchmark · 100%LLM-Virus · attack-method · 100%AutoDAN · attack-method · 95%GPTFuzzer · attack-method · 95%

Relation Signals (4)

LLM-Virus → evaluatedon → Harmbench

confidence 100% · perform experiments on HarmBench [11] and AdvBench [15] datasets.

LLM-Virus → evaluatedon → AdvBench

confidence 100% · perform experiments on HarmBench [11] and AdvBench [15] datasets.

LLM-Virus → uses → Evolutionary Algorithm

confidence 100% · propose LLM-Virus, a jailbreak attack method based on evolutionary algorithm

LLM-Virus → outperforms → AutoDAN

confidence 90% · LLM-Virus achieves competitive or even superior performance compared to existing attack methods.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:While safety-aligned large language models (LLMs) are increasingly used as the cornerstone for powerful systems such as multi-agent frameworks to solve complex real-world problems, they still suffer from potential adversarial queries, such as jailbreak attacks, which attempt to induce harmful content. Researching attack methods allows us to better understand the limitations of LLM and make trade-offs between helpfulness and safety. However, existing jailbreak attacks are primarily based on opaque optimization techniques (e.g. token-level gradient descent) and heuristic search methods like LLM refinement, which fall short in terms of transparency, transferability, and computational cost. In light of these limitations, we draw inspiration from the evolution and infection processes of biological viruses and propose LLM-Virus, a jailbreak attack method based on evolutionary algorithm, termed evolutionary jailbreak. LLM-Virus treats jailbreak attacks as both an evolutionary and transfer learning problem, utilizing LLMs as heuristic evolutionary operators to ensure high attack efficiency, transferability, and low time cost. Our experimental results on multiple safety benchmarks show that LLM-Virus achieves competitive or even superior performance compared to existing attack methods.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

67,669 characters extracted from source content.

Expand or collapse full text

LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models Miao Yu1,∗superscriptMiao Yu1Miao Yu^1,*Miao Yu1 , ∗, Junfeng Fang2,∗superscriptJunfeng Fang2Junfeng Fang^2,*Junfeng Fang2 , ∗, Yingjie Zhou3superscriptYingjie Zhou3Yingjie Zhou^3Yingjie Zhou3, Xing Fan1,†superscriptXing Fan1†Xing Fan^1, Xing Fan1 , †, Kun Wang1,†superscriptKun Wang1†Kun Wang^1, Kun Wang1 , †, Shirui Pan4superscriptShirui Pan4Shirui Pan^4Shirui Pan4, Qingsong Wen1superscriptQingsong Wen1Qingsong Wen^1Qingsong Wen1 1Squirrel Ai Learning 2National University of Singapore 3Sichuan University 4Griffith University Marker * denotes equal contributions, and † † means that Xing Fan and Kun Wang are the corresponding authors.Correspondence to ymzgkxjsdx@mail.ustc.edu.cn Abstract While safety-aligned large language models (LLMs) are increasingly used as the cornerstone for powerful systems such as multi-agent frameworks to solve complex real-world problems, they still suffer from potential adversarial queries, such as jailbreak attacks, which attempt to induce harmful content. Researching attack methods allows us to better understand the limitations of LLM and make trade-offs between helpfulness and safety. However, existing jailbreak attacks are primarily based on opaque optimization techniques (e.g. token-level gradient descent) and heuristic search methods like LLM refinement, which fall short in terms of transparency, transferability, and computational cost. In light of these limitations, we draw inspiration from the evolution and infection processes of biological viruses and propose LLM-Virus, a jailbreak attack method based on evolutionary algorithm, termed evolutionary jailbreak. LLM-Virus treats jailbreak attacks as both an evolutionary and transfer learning problem, utilizing LLMs as heuristic evolutionary operators to ensure high attack efficiency, transferability, and low time cost. Our experimental results on multiple safety benchmarks show that LLM-Virus achieves competitive or even superior performance compared to existing attack methods. Our code is available at https://github.com/Ymm-cll/LLM-Virus. Index Terms: LLM Safety, Jailbreak Attack, Evolutionary Algorithm †publicationid: pubid: Perprint Warning: This paper contains potentially harmful text. I Introduction As LLMs emerge with exceptional and advanced capabilities such as knowledge [1], planning [2] and reasoning [3], they are exponentially being applied to systems (e.g. LLM-integrated applications [4] and LLM-based multi-agent systems [5]) across various domains and scenarios to solve certain complex problems. In this context, preventing the misuse of these powerful and influential LLM-based systems becomes increasingly critical [6]. This research area, known as “LLM Safety”, primarily focuses on preventing LLMs from being used for malicious behaviors, such as the spread of misinformation and bias, the generation of harmful content, and privacy breaches [7, 8]. Directly issuing malicious queries is typically rejected, as most available LLMs (e.g. GPT and Claude) are safety-aligned via techniques like fine-tuning to ensure adherence of responses to secure human values [9, 10]. Unfortunately, a variety of jailbreak attack methods still exist that can bypass the built-in safety mechanisms [11]. Figure 1: Illustration of different ways for malicious querying (direct attack, normal jailbreak and evolutionary jailbreak). Attacks to LLM typically occur during training or inference [7], with the latter being more relevant to current usage scenarios, where users do not participate in the training process. Depending on whether the model is open-source, inference-time attacks can be categorized into two types [12]: White-box attacks involve utilizing inherent information like gradient to optimize prefixes or suffixes that can elicit affirmative responses [13, 14], which are then linked to malicious queries to induce desired responses [15]. Black-box attacks, in contrast, do not have access to the model’s internal data and typically rely on manual techniques or LLM-generated methods to search for effective jailbreak prompts. For example, PAIR [16] leverages an attacker LLM to iteratively query a target LLM and refine the jailbreak prompt until successful. Additionally, a small number of studies have utilized evolutionary algorithms (EAs) to optimize attacks to models. (Figure 1). For instance, A [17] and [18] extends EAs to conduct evolutionary attacks in the image modality. As for those targeting at LLMs, AutoDAN [13] optimizes existing jailbreak prompts by performing word-level mutations and paragraph-level crossover, with generation probability of affirmative prefix as fitness. BlackDAN [19] extends AutoDAN by applying the NSGA-I for multi-objective evolution. GPTFuzzer [20] is inspired by EA-based fuzz testing to evolve jailbreak templates, while SMJ [21] focuses on improving the similarity between harmful actions and jailbreak prompts. Similar to existing works on LLM-enhanced EAs for combinatorial problems [22, 23, 24], part of these methods use LLMs to aid mutation, crossover, or fitness evaluation to varying degrees. However, there are several limitations in existing evolutionary jailbreak researches. (I) Irrational Evolutionary Operator: For instance, AutoDAN and BlackDAN only consider word-level mutation and random paragraph-level crossover. The former restricts the linguistic diversity, while the latter disrupts the contextual semantics of jailbreak prompt as a coherent piece of text. (I) Limited Attack Scenario: SMJ focuses solely on attacks against smaller, less secure models and only considers using LLM-based crossover by rephrasing. GPTFuzzer introduces additional LLM-based operators (e.g. expand and shorten), but it merely targets at toxicity. Besides, both of them conduct attacks to LLMs on a small malicious dataset (only 100 samples). (I) High Time Cost & Incomprehensive Baseline: None of them address the increasing time cost associated with multiple iterations of evolution and enormous population size, nor do they provide a comprehensive performance comparison of evolutionary jailbreak with traditional (non-evolutionary) LLM attack methods. To address the issues mentioned above, among others, we draw inspiration from infection and evolution of biological viruses and propose LLM-Virus, a black-box and efficient jailbreak attack based on LLM-enhanced EAs. Analogously, we treat the attack as viral infection. Jailbreak templates represent the mutating genetic material, while specific malicious queries as the functional proteins executing the attack. The LLM itself serves as the targeted host. Through selection driven by the host’s safety mechanisms, our goal is to evolve a population of virus strains (initialized by human-written jailbreak templates), with the help of LLMs as evolutionary operators. Specifically, we first propose LLM-based crossover and mutation operations to explore wider solution space of jailbreak templates, encouraging specific textual properties such as diversity and conciseness. We treat jailbreak template as an individual to allow for the embedding of other queries within the template (thus more transferable), rather than a specific jailbreak prompt for one given malicious query. Then, similar to how viruses transfer and infect different hosts, we view the problem as a transfer learning task and introduce Local Evolution and Generalized Infection techniques to reduce computational and time costs while enhancing transferability and generalization. Furthermore, we employ more stringent attack classifiers to evaluate fitness (attack success rate) and perform experiments on HarmBench [11] and AdvBench [15] datasets. Our experiments comprehensively compare performance with traditional LLM attack methods, achieving state-of-the-art success attack rates and lower time costs. Moreover, we analyze the evolution dynamic of the average population fitness and perform ablation studies to demonstrate the effectiveness and validity of certain methods in LLM-Virus. To conclude, our contributions are summarized as follows: ❶ Evolutionary Jailbreak: We propose LLM-Virus, an LLM attack via evolutionary algorithm, achieving extraordinary performance on multiple safety benchmarks. ❷ New Insights: We treat the jailbreak evolution process as a transfer learning problem to optimize time consumption and transferability, with LLMs as evolutionary operators. ❸ Holistic Experiments: We conduct a comprehensive performance comparison of our LLM-Virus with non-evolutionary attacks and other EA-based ones, demonstrating the advantages of our evolutionary jailbreak. I Related Works Jailbreak Attacks on LLM. State-of-the-art LLMs have undergone safety alignment processes to prevent their misuse in malicious activities [25, 26, 9]. However, jailbreak attacks aim to bypass these aligned values and internal safety mechanisms, aiming to elicite harmful outputs [27, 28]. Typically, jailbreak occurs during inference, where tailor-designed input prompts are used to deceive the LLM into responding to harmful queries, such as “How to make a bomb?”, which would otherwise be rejected when querying directly [12, 29]. Some existing works employ human expertise to heuristically design prompt templates [30, 31], such as “Do anything now” [32] or “Ignore previous prompt”[33]. Other approaches use optimization techniques to prepend or append optimized prefixes or suffixes to harmful queries, maximizing the likelihood of a positive response from the model [15, 14]. Another line of research utilizes sequence-to-sequence models, such as LLM [16] or Multi-agent System [34], to modify existing malicious queries and generate potential jailbreak prompts. In our work, we follow the last line but integrate EA enhanced by LLMs to explore a wider range of search space, boosting the efficiency and transferability of jailbreak attacks. Evolutionary Algorithm. As a family of population-based, stochastic optimization techniques inspired by natural evolution, evolutionary algorithms (EAs) primarily encompass methods like genetic algorithms [35], evolution strategies[36], evolutionary programming [37], and genetic programming [38]. These methods similarly model the search process as an evolution, where solutions are iteratively improved through selection, mutation, and crossover [39]. Advanced research explores the application of EAs to multi-objective or multi-task optimization [40, 41], dynamic environments [42], and even under noisy or uncertain conditions [43]. We primarily explore the feasibility and effectiveness of using EA to search for jailbreak templates that can attack LLMs successfully while also leveraging LLMs as evolutionary operators. Convergence of LLM and Evolutionary Algorithm. The remarkable reasoning [3] and knowledge [1] capabilities exhibited by LLMs have enabled them to achieve impressive performance across a wide spectrum of tasks [44, 45]. Some studies have leveraged LLMs in EAs to enhance the diversity and reliability of mutation and crossover processes [22, 23, 24]. For instance, LMEA [46] utilize LLM as evolutionary operators, revealing its potential in solving combinatorial problems. [47] further applies LLM in Multi-object EA and reports its robust generalization performance. On the other hand, there are works that, in contrast, use EA in the LLM field, such as prompt engineering [48] and structure searching [49]. For example, [50] automatically extends expert agents to LLM-based multi-agent systems via EA to improve task performance. AutoDAN [13] conducts synonym mutation and paragraph crossover of jailbreak prompts to get affirmative responses. However, it requires white-box model access and limits the search space with high time cost. In LLM-Virus, our method addresses these limitations and integrates the aforementioned two research lines: leveraging LLMs to enhance EA while using EA for jailbreak attacks on LLMs. Figure 2: Overview of LLM-Virus. General workflow of jailbreak attacks (Top) and three steps to search for more effective jailbreak templates (Bottom). We demonstrate the LLM system prompts for fitness, mutation and crossover in Step I. I Preliminaries Denotations. Suppose the set of all texts to be Tblackboard_T, and treat the LLM as a black-box function ℒ:→:ℒ→L:T L : blackboard_T → blackboard_T that maps an input query to an output answer. Let the set of malicious queries to be =di=(qi,ti)|1≤i≤||conditional-setsubscriptsubscriptsubscript1D=\d_i=(q_i,t_i)|1≤ i≤|D|\D = ditalic_i = ( qitalic_i , titalic_i ) | 1 ≤ i ≤ | D | , where qisubscriptq_iqitalic_i denotes a query to harmful contents, such as “how to steal personal privacy”, while tisubscriptt_ititalic_i represents an affirmative answer with prefix like “Sure, here is how to …”. However, directly querying qisubscriptq_iqitalic_i will normally be rejected by well-guarded LLMs. Threat Model. Jailbreak attack typically occurs during model inference, where a certain jailbreak template j∈j ∈ blackboard_T is used to help the malicious query qisubscriptq_iqitalic_i to bypass the LLM’s aligned safety mechanisms and induce a positive response, such as detailed ways to conduct harmful actions: ℒ⁢(j⊕qi)=ti,if jailbreak is successful“Sorry, I can’t …”,otherwiseℒdirect-sumsubscriptcasessubscriptif jailbreak is successfulotherwise“Sorry, I can’t …”otherwiseotherwiseL(j q_i)= casest_i, jailbreak is % successful\\ ``Sorry, I can't ...′, casesL ( j ⊕ qitalic_i ) = start_ROW start_CELL titalic_i , if jailbreak is successful end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL “Sorry, I can’t …” , otherwise end_CELL start_CELL end_CELL end_ROW (1) In Eq. 1, the text operator x⊕ydirect-sumx yx ⊕ y denotes a specific combination of x and y to form a new text, such as by concatenating y as the prefix/suffix of x, or inserting y within x. IV LLM-Virus Framework Inspired by the process through which biological viruses evolve to evade the host immune system, we introduce LLM-Virus, an LLM-targeted and LLM-enhanced framework that leverages evolutionary algorithms to optimize and search for more effective jailbreak. Concretely, LLM-Virus utilizes various emergent abilities of LLM (e.g., knowledge, optimization, text processing) and makes it an evolutionary operator to carry out jailbreak attacks on other models. Target Formulation. We define the binary (j,qi)subscript(j,q_i)( j , qitalic_i ) as an LLM virus. In analogy, the jailbreak template j functions like the genetic material (DNA/RNA) of a virus, undergoing mutations and evolving under the selection pressure of the LLM’s (host) safety mechanisms (immune system). On the other hand, qisubscriptq_iqitalic_i is akin to a functional protein, conducting an actual jailbreak attack (infection) on the host. To make it clear, we formulate the target of LLM-Virus to be: arg⁡max∗⁡1|∗|⁢∑j∈∗∑i=1||E⁢[ℒ⁢(j⊕qi),ti]⏟Toxicity+∑i=1|ℱ|αi⁢fi⁢(∗)⏟Constraintsubscriptsuperscriptsubscript⏟1superscriptsubscriptsuperscriptsuperscriptsubscript1Eℒdirect-sumsubscriptsubscriptToxicitysubscript⏟subscriptsuperscriptℱ1subscriptsubscriptsuperscriptConstraint _J^* \ 1|% J^*| _j ^* _i=1^|D|E [% L(j q_i),t_i ]_ [rgb]1,0,0 [% named]pgfstrokecolorrgb1,0,0Toxicity+ Σ^|% F|_i=1 _if_i(J^*)_ [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1Constraint \arg maxcaligraphic_J∗ under⏟ start_ARG divide start_ARG 1 end_ARG start_ARG | J∗ | end_ARG ∑j ∈ J∗ ∑i = 1| D | E [ L ( j ⊕ qitalic_i ) , titalic_i ] end_ARGToxicity + under⏟ start_ARG ∑| F |i = 1 αitalic_i fitalic_i ( J∗ ) end_ARGConstraint (2) Eq 2 consists of two terms: the toxicity term, which measures the attack success rate (also fitness) of the jailbreak templates ∗superscriptJ^*J∗, and the constraint term, which imposes additional requirements ℱFF on the target templates, such as text length. Specifically, E⁢(x,y)=1E1E(x,y)=1E ( x , y ) = 1 holds only when x=yx=yx = y, and αisubscript _iαitalic_i represents the weight of each requirement function fi∈ℱsubscriptℱf_i _i ∈ F. Since jailbreak templates, as semantic text, are discrete and lack desirable mathematical properties, LLM-Virus uses evolutionary algorithm to provide a heuristic suboptimal solution to Eq 2. Specifically, LLM-Virus consists of three steps: Strain Collection, Local Evolution, and Generalized Infection, as the pipeline demonstrated in Algorithm 1. IV-A Strain Collection Traditional evolutionary algorithms typically employ automated strategies such as random generation [51] to obtain an initial population. However, in the context of LLM jailbreak, we can take advantage of effective and human-written jailbreak templates from existing datasets. Thus, we propose Strain Collection to gather initial templates with desired properties and make the constraint functions in Eq 2 clear. We focus on the three key features as follows: • Stealthiness: Jailbreak template j itself, without the malicious qisubscriptq_iqitalic_i, should not be rejected by host LLM. The following Eq 3 is the constraint function of stealthiness: fs=∑j∈∗E⁢[ℒ⁢(j,“Sure, …”)]subscriptsubscriptsuperscriptEdelimited-[]ℒ“Sure, …”f_s= _j ^*E [L(j,``Sure, .% ..′) ]fitalic_s = ∑j ∈ J∗ E [ L ( j , “Sure, …” ) ] (3) • Diversity: Each jailbreak template j varies in semantics and employs different tricks to deceive LLM and bypass safety mechanisms. We quantify diversity as Eq 4 below: fd=∑1≤i<k≤|∗|−Similariy⁢(ji,jk)subscriptsubscript1superscriptSimilariysubscriptsubscriptf_d= _1≤ i<k≤|J^*|-Similariy(j_i,j_k)fitalic_d = ∑1 ≤ i < k ≤ | J∗ | - Similariy ( jitalic_i , jitalic_k ) (4) • Cheapness. Each j should be as concise as possible to minimize token consumption during attacks, thereby reducing attack cost, with constraint function in Eq 5: fc=∑j∈∗Length⁢(j)subscriptsubscriptsuperscriptLengthf_c= _j ^*Length(j)fitalic_c = ∑j ∈ J∗ Length ( j ) (5) To provide an approximate solution to Eq 2 and improve the quality of evolution, we start with filtering the jailbreak templates in existing datasets 111https://huggingface.co/datasets/rubend18/ChatGPT-Jailbreak-Prompts based on the above three features and obtain an initial population for subsequent evolutionary search and optimization. Algorithm 1 Execution Pipeline of LLM-Virus 1: Input: Targeted host LLM ℒhsubscriptℒℎL_hLitalic_h, evolution operator LLM ℒe⁢osubscriptℒL_eoLitalic_e o, set of malicious queries DD, set of human-designed jailbreak templates hsubscriptℎJ_hJitalic_h, maximum number of generations GGG, population size NNN, expected success rate rssubscriptr_sritalic_s. 2: Output: A set of more effective jailbreak templates ∗superscriptJ^*J∗. 3: r←subscriptabsentD_r _r ← Selects several centers after clustering of DD 4: ←absentJ ← Initialize NNN templates after Strain_Collection(hsubscriptℎJ_hJitalic_h) // Initialization via Strain Collection with desired features 5: ℱ←fs,fd,fc←ℱsubscriptsubscriptsubscriptF←\f_s,f_d,f_c\F ← fitalic_s , fitalic_d , fitalic_c 6: for g from 1111 to GGG do 7: ′←∇cℱ(jp1,jp2)|(jp1,jp2)←J^ ←\ _c^F(j_p_1,j_p_% 2)|(j_p_1,j_p_2) _FLOATSUPERSCRIPT ′ end_FLOATSUPERSCRIPT ← ∇ccaligraphic_F ( jitalic_p start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , jitalic_p start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) | ( jitalic_p start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , jitalic_p start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) ← Select_Parents(JJ) // LLM-based crossover to generate offspring 8: ′←∇mℱ(j|j∈J^ ←\ _m^F(j|j∈% JJstart_FLOATSUPERSCRIPT ′ ′ end_FLOATSUPERSCRIPT ← ∇mcaligraphic_F ( j | j ∈ J // LLM-based mutation 9: ℰf←∑(q,t)∈rEL(j⊕q⊕t)|j∈′E_f←\ _(q,t) _rE_L(j% q t)|j ^ Eitalic_f ← ∑( q , t ) ∈ D start_POSTSUBSCRIPT r end_POSTSUBSCRIPT Eitalic_L ( j ⊕ q ⊕ t ) | j ∈ Jstart_FLOATSUPERSCRIPT ′ ′ end_FLOATSUPERSCRIPT // LLM-based fitness evaluation 10: ←Top_N⁢[Sort(′∪′∪),based on (ℰf,fc,fs)]←Top_NSort(′∪′∪)based on (ℰf,fc,fs)J \_N[Sort($ J^ % ^ $),based on ($% E_f,f_c,f_s$)]J ← Top_N [ Sort( Jstart_FLOATSUPERSCRIPT ′ ′ end_FLOATSUPERSCRIPT ∪ Jstart_FLOATSUPERSCRIPT ′ end_FLOATSUPERSCRIPT ∪ J ) , based on ( Eitalic_f , fitalic_c , fitalic_s ) ] // Selection of next generation 11: end for // Local Evolution 12: ℰG⁢I←∑(q,t)∈EL(j⊕q⊕t)|j∈∗E_GI←\ _(q,t) E_L(j q% t)|j ^*Eitalic_G I ← ∑( q , t ) ∈ D Eitalic_L ( j ⊕ q ⊕ t ) | j ∈ J∗ // Generalized Infection 13: if Average(ℰG⁢IsubscriptℰE_GIEitalic_G I) <rsabsentsubscript<r_s< ritalic_s then h←subscriptℎJ_h _h ← J and turn to line 4 14: else ∗←superscriptJ^* ∗ ← J // Decide whether to loop or not (optimal) IV-B Local Evolution To reduce the time and computational overhead on the entire malicious query set DD, we also frame the evolution in LLM-Virus as a transfer learning problem. Specifically, we use clustering to extract a representative subset rsubscriptD_rDitalic_r from DD, and then apply evolutionary algorithm to optimize jailbreak templates on rsubscriptD_rDitalic_r. Just as virus can spread due to biological similarities between different hosts, LLM virus can migrate (spread) because different LLMs share similar knowledge structures and modes of thinking. Specifically, for the implementation of Local Evolution, due to the inability of mathematical operators to effectively handle text variables, we utilize LLMs, possessing exceptional language capabilities, as fitness (attack success rate) evaluator and evolutionary operators for crossover and mutation. IV-B1 Fitness We follow previous research on LLM safety [13] to evaluate the success of jailbreak using two methods: rejection keyword detection (e.g., “Sorry”,“can’t”) and LLM discrimination. We formalize the LLM-based method below: EL→0,1=ℒe⁢[e⊕ℒh⁢(j⊕qi)⊕ti]superscriptsubscriptE→01subscriptℒdelimited-[]direct-sumsubscriptsubscriptℒℎdirect-sumsubscriptsubscriptE_L^T→\0,1\=L_e[P_e % L_h(j q_i) t_i]Eitalic_Lblackboard_T → 0 , 1 = Litalic_e [ Pitalic_e ⊕ Litalic_h ( j ⊕ qitalic_i ) ⊕ titalic_i ] (6) In Eq 6, ℒesubscriptℒL_eLitalic_e and ℒhsubscriptℒℎL_hLitalic_h are the evaluating model and the attacked model, respectively. esubscriptP_ePitalic_e is a designed system prompt to guide ℒesubscriptℒL_eLitalic_e to determine the success of jailbreak virus (j,qi)subscript(j,q_i)( j , qitalic_i ). IV-B2 Crossover/Mutation [23] and [46] explore the exceptional performance of LLMs in crossover and mutation operations within the text modality. We extend crossover and mutation in previous work [13, 20] by introducing heuristic crossover/mutation, which encourages LLMs to perform crossover or mutation operations at a broader range, from words to paragraphs and in a specified direction (such as the 3 features mentioned in IV-A). We formulate these two LLM-based operators with constraint set ℱFF below: ∇cℱ=ℒc⁢(c⊕ji⊕jk,ℱ),∇mℱ=ℒm⁢(m⊕ji,ℱ)formulae-sequencesuperscriptsubscript∇ℱsubscriptℒdirect-sumsubscriptsubscriptsubscriptℱsuperscriptsubscript∇ℱsubscriptℒdirect-sumsubscriptsubscriptℱ _c^F=L_c(P_c j_i j_% k,F), _m^F=L_m(P_m% j_i,F)∇ccaligraphic_F = Litalic_c ( Pitalic_c ⊕ jitalic_i ⊕ jitalic_k , F ) , ∇mcaligraphic_F = Litalic_m ( Pitalic_m ⊕ jitalic_i , F ) (7) In Eq 7, csubscriptP_cPitalic_c and msubscriptP_mPitalic_m are tailor-made system prompts that guide the LLM to evolve text targeting at specified properties ℱFF in few-shot manner. The result of both operators is a new jailbreak template jnew∈subscriptnewj_new ∈ blackboard_T. Moreover, we can achieve greater diversity by increasing the temperature parameter of the crossover/mutation LLM via more diverse token generation. Heuristic vs. Normal. Compared to normal word-level mutation and paragraph-level crossover, heuristic mutation/crossover leverages system prompts to guide LLM in performing heuristic searches over a larger space based on prototypes. Additionally, it can utilize the LLM’s comprehension and generation capabilities to impose extra optimization requirements (fssubscriptf_sfitalic_s, fdsubscriptf_dfitalic_d, fcsubscriptf_cfitalic_c) on the search direction. IV-B3 Selection In selection process, we choose to adopt keyword ranking rather than a multi-objective evolutionary algorithm, aiming for evolutionary simplicity while maintaining effectiveness. Specifically, based on Eq. 2, we prioritize toxicity (success attack rate), as the primary fitness keyword, followed by stealthiness (fssubscriptf_sfitalic_s) and cheapness (fcsubscriptf_cfitalic_c). Using this order, we can employ common selection strategies [52]. IV-C Generalized Infection After Local Evolution, to obtain more adaptive LLM viruses, Generalized Infection tests the transferability (virus transmission) of the evolved jailbreak template population from r→subscriptD_r _r → D by applying Eq 6 to get the success rate. In the context of jailbreak attack scenario we consider, transferability is generally easier than that of traditional machine learning problems. This is due to the strong representativeness of DrsubscriptD_rDitalic_r derived from clustering, as well as the similar defense mechanisms of LLMs against different harmful queries [53]. IV-D Trade-off between Cost and Transferability In this section, we analyze the reduced cost of applying Local Evolution and Generalized Infection in LLM-Virus. Assumption: when querying an LLM, we specify the maximum number of generated tokens to be nmaxsubscriptmaxn_maxnmax, and the time cost tLsubscriptt_Ltitalic_L for each LLM query is approximately equal. Supposing that each round of evolution generates NNN new offspring, when employing transfer learning, the number of querying LLM for crossover/mutation operations and fitness evaluation are (N+N)N(N+N)( N + N ) and (|r|×N)subscriptN(|D_r|×N)( | Ditalic_r | × N ), respectively. Thus, the total LLM query count for GGG rounds of evolution with one final Generalized Infection can be represented in Eq 8: nq=N×[G×(2+|r|)+||]subscriptNdelimited-[]G2subscriptn_q=N×[G×(2+|D_r|)+|D|]nitalic_q = N × [ G × ( 2 + | Ditalic_r | ) + | D | ] (8) Thus, the upper bounds for time and output tokens are nq×tLsubscriptsubscriptn_q× t_Lnitalic_q × titalic_L and nq×nmaxsubscriptsubscriptmaxn_q× n_maxnitalic_q × nmax, respectively. Similarly, for computational convenience, we approximate the number of text tokens for each offspring jailbreak template j to be nmaxsubscriptmaxn_maxnmax, since they are all generated by LLMs with maximum generation token limit. Then, the input token consumption for LLM-Virus is: G×N×(4⁢nmax+∑d∈r|d|)GN4subscriptmaxsubscriptsubscriptG×N×(4n_max+ _d _r|d|)G × N × ( 4 nmax + ∑d ∈ D start_POSTSUBSCRIPT r end_POSTSUBSCRIPT | d | ) (9) In Eq 9, |d|=|(q,t)|=|q|+|t||d|=|(q,t)|=|q|+|t|| d | = | ( q , t ) | = | q | + | t | and for x∈x ∈ blackboard_T, |x||x|| x | represents the token number of text x after tokenization of the LLM. Compared to not using transfer learning, the ratio of query counts (which is also the ratio of time and output token consumption) and that of input token consumption is: rq=rt=ro⁢t=N×[G×(2+|r|])+||N×[G×(2+||)+||]≈|r|||r_q=r_t=r_ot= N×[G×(2+|D_r|% ])+|D|N×[G×(2+|D|)+| % D|]≈ |D_r||D|ritalic_q = ritalic_t = ritalic_o t = divide start_ARG N × [ G × ( 2 + | Ditalic_r | ] ) + | D | end_ARG start_ARG N × [ G × ( 2 + | D | ) + | D | ] end_ARG ≈ divide start_ARG | Ditalic_r | end_ARG start_ARG | D | end_ARG (10) ri⁢t=G×N×(4⁢nmax+∑d∈r|d|)G×N×(4⁢nmax+∑d∈|d|)≈|r|||,r⊂formulae-sequencesubscriptGN4subscriptmaxsubscriptsubscriptGN4subscriptmaxsubscriptsubscriptsubscriptr_it= G×N×(4n_max+ _d∈% D_r|d|)G×N×(4n_max+ _d% |d|)≈ |D_r||D|,D% _r _i t = divide start_ARG G × N × ( 4 nmax + ∑d ∈ D start_POSTSUBSCRIPT r end_POSTSUBSCRIPT | d | ) end_ARG start_ARG G × N × ( 4 nmax + ∑d ∈ D | d | ) end_ARG ≈ divide start_ARG | Ditalic_r | end_ARG start_ARG | D | end_ARG , Ditalic_r ⊂ D (11) Eq 10 and 11 demonstrate that LLM-Virus can reduce the cost of the evolution process through transfer learning, allowing us to balance the trade-off between overhead and transferability by adjusting |r|subscript|D_r|| Ditalic_r |. TABLE I: Baselines. “Harm” and “Adv” are short for HarmBench and AdvBench, respectively. Paper Method Name Derived Method Benchmark [15] GCG GCG-T, GCG-M Harm/Adv [54] PEZ - Harm [55] GBDA - Harm [56] UAT - Harm [57] AutoPrompt (AP) - Harm [58] Zero-Shot (ZS) Stochastic Few-Shot (SFS) Harm [16] PAIR - Harm/Adv [59] TAP TAP-Transfer (TAP-T) Harm/Adv [13] AutoDAN - Harm/Adv [60] PAP - Harm [61] Human Jailbreaks - Harm [31] DeepInception - Adv [19] BlackDAN - Adv Figure 3: LLM-Virus dynamic of ASRlsubscriptASRASR_lASRl and template length on part of AdvBench (rsubscriptD_rDitalic_r) in Step I (Local Evolution). TABLE I: ASRcsubscriptASRASR_cASRc on HarmBench with feature summary. Model/Feature/ASRcsubscriptModel/Feature/ASRModel/Feature/ASR_cModel/Feature/ASRc Baselines Ours Direct Human GCG GCG-M GCG-T PEZ GBDA UAT AP SFS AutoDAN ZS PAIR TAP TAP-T PAP-top5 LLM-Virus Closed-source LLM GPT-3.5-Turbo-0613 21.3 24.5 - - 38.9 - - - - - - 24.8 46.8 47.7 62.3 15.4 71.8 GPT-3.5-Turbo-1106 33.0 2.8 - - 42.5 - - - - - - 28.4 35.0 39.2 47.5 11.3 53.8 GPT-4-0613 9.3 2.6 - - 22.0 - - - - - - 19.4 39.3 43.0 54.8 16.8 29.3 Claude-2 2.0 0.3 - - 2.7 - - - - - - 4.1 4.8 2.0 0.8 1.0 1.5 Gemini Pro 18.0 12.1 - - 18.0 - - - - - - 14.814.814.814.8 35.1 38.8 31.2 11.8 56.8 Open-Source LLM Llama-2-7B-Chat 0.8 0.8 32.5 21.2 19.7 1.8 1.4 4.5 15.3 4.3 0.5 2.0 9.3 9.3 7.8 2.7 38.5 Llama-2-13B-Chat 2.8 1.7 30.0 11.3 16.4 1.7 2.2 1.5 16.3 6.0 0.8 2.9 15.0 14.2 8.0 3.3 33.5 Llama-2-70B-Chat 2.8 2.2 37.5 10.8 22.1 3.3 2.3 4.0 20.5 7.0 2.8 3.0 14.5 13.3 16.3 4.1 60.5 Vicuna-7B 24.3 39.0 65.6 61.5 60.8 19.8 19.0 19.3 56.3 42.3 66.0 27.2 53.5 51.0 59.8 18.9 80.5 Vicuna-13B 19.8 40.0 67.0 61.3 54.9 15.8 14.3 14.2 41.8 32.3 65.5 23.2 47.5 54.8 62.1 19.3 91.8 Features Black-box Workable ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ LLM-enhanced - - ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ Optimization - - ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ Evolution-based - - ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ Template/Suffix - Tem. Suf. Suf. Suf. Suf. Suf. Suf. Suf. - Tem. - Tem. Tem. Tem. Tem. Tem. TABLE I: ASRksubscriptASRASR_kASRk and ASRlsubscriptASRASR_lASRl on AdvBench. ASRk/ASRlsubscriptASRsubscriptASRASR_k/ASR_lASRk / ASRl GPT-4 GPT-3.5-Turbo Llama-2-7B Vicuna-7B GCG 0.4/- 16.5/15.2 45.4/43.1 97.1/87.5 AutoDAN 0.7/- 65.7/72.9 60.8/65.6 97.7/91.7 PAIR 48.1/30.0 51.3/34.0 5.2/4.0 62.1/41.9 TAP 36.0/11.9 48.1/5.4 30.2/23.5 31.5/25.6 DeepInception 61.9/22.7 68.5/40.0 77.5/31.2 92.7/41.5 BlackDAN 71.4/28.0 75.9/44.8 95.5/93.8 97.5/96.0 LLM-Virus 74.0/36.5 90.8/96.5 95.6/96.6 93.5/97.0 V Experiment V-A Experimental Setups Models. For the host LLMs to be attacked, we select closed-source models such as the GPT [25], Claude 222https://docs.anthropic.com/en/api/models, and Gemini [62] series, as well as open-source models including Llama [26], Vicuna [63], and Gemma [64] series. In addition, we utilize GPT-4o 333https://platform.openai.com/docs/models as crossover and mutation operators to enhance our evolutionary attack. Datasets. We select AdvBench [15] and HarmBench [11], which contain 520 and 400 instances of harmful behaviors in various fields, respectively, as the set of malicious queries DD. In Local Evolution, we first embed DD into vectors using all-MiniLM-L6-v2 [65], and then apply KMeans [66] clustering. Then harmful actions closest to the cluster centers are selected into rsubscriptD_rDitalic_r, and we set |r|||=2.5%subscriptpercent2.5 |D_r||D|=2.5\%divide start_ARG | Ditalic_r | end_ARG start_ARG | D | end_ARG = 2.5 %. Baselines. To comprehensively compare the performance of LLMs with existing works (both traditional and EA-based), we consider various baselines in Table I and evaluate them on the HarmBench and AdvBench. For LLM-Virus, we report the average performance in three runs due to evolution randomness. Metrics. To evaluate the attack success rate (ASR) below, ASR=1||⁢∑(q,t)∈E⁢v⁢a⁢l⁢u⁢a⁢t⁢o⁢r⁢[ℒ⁢(j⊕q),q]ASR1subscriptℒdirect-sumASR= 1|D| _(q,t) Evaluator[ % L(j q),q]ASR = divide start_ARG 1 end_ARG start_ARG | D | end_ARG ∑( q , t ) ∈ D E v a l u a t o r [ L ( j ⊕ q ) , q ] (12) we follow previous research to use a rejection keyword list (ASRksubscriptASRASR_kASRk) [15], a fine-tuned Llama-2-13b-cls model [11] and GPT-4o with system prompts for classifying (Figure 2) as the 01-valued attack success evaluator (denoted as ASRcsubscriptASRASR_cASRc and ASRlsubscriptASRASR_lASRl, respectively). We select ASRcsubscriptASRASR_cASRc as the fitness function for HarmBench, while ASRlsubscriptASRASR_lASRl for AdvBench in evolution. Evolution. Setting generation size N=1010N=10N = 10 and iteration G=1010G=10G = 10, we use GPT-4o, equipped with tailor-desigend system prompts (Figure 2) and setting t⁢e⁢m⁢p⁢e⁢r⁢a⁢t⁢u⁢r⁢e=11temperature=1t e m p e r a t u r e = 1, for mutation/crossover and fitness (ASR) evaluation. Each individual has a mutation probability of pmutation=0.5subscriptmutation0.5p_mutation=0.5pmutation = 0.5 and an equal chance to be selected as a parent, with pcrossover=1subscriptcrossover1p_crossover=1pcrossover = 1. The elitism strategy is applied for next generation selection. V-B Local Evolution Dynamic LLM-Virus enables efficient evolutionary optimization of jailbreak templates within the local dataset. As shown in Figure 3, as the generation number increases, both the average ASRlsubscriptASRASR_lASRl and template length across different LLMs progressively optimize towards the target direction (Eq 2). Even for the safest LLMs today, such as GPT-4o and Claude-3.5, the population average ASRlsubscriptASRASR_lASRl increases from 11.5 → 26.9 and 20.8 → 29.3, respectively. The most substantial gain is observed in GPT-3.5-Turbo, where ASRlsubscriptASRASR_lASRl progresses from 54.6 → 100.0. Template length, as the second rank criterion (Line 10 in Algorithm 1), increases sightly for LLMs like GPT-4o-mini due to prioritizing ASRlsubscriptASRASR_lASRl. However, other models, including Llama-3.1-70B and GPT-3.5-Turbo, exhibit a significant reduction in template length, from over 1300 → 453.8 and 461.2 → 292.2 (36.6% ↓ ↓), respectively. These observations demonstrate that LLM-Virus can evolve and optimize jailbreak templates towards specified directions (e.g. toxicity and cheapness). V-C Generalized Infection Performance LLM-Virus has demonstrated its effectiveness on the local dataset rsubscriptD_rDitalic_r. In the following, we investigate the generalization performance of these newly evolved jailbreak templates on the full dataset DD from the following comprehensive aspects. V-C1 Toxicity LLM-Virus outperforms the baselines in both HarmBench and AdvBench, achieving the best results. In Table I, we present the toxicity (evaluated on the full dataset DD) of the top-performing LLM virus from the final generation that is not in the initial population, with ASRcsubscriptASRASR_cASRc from HarmBench. LLM-Virus achieves optimal performance on 3 out of 5 closed-source models and all open-source models. Specifically, on Gemini-Pro and Llama-3.1-70B, the ASRcsubscriptASRASR_cASRc of LLM-Virus is 1.46×1.46×1.46 × and 1.61×1.61×1.61 × that of the second-best, respectively. On three scales of the Llama-3.1 model, the average ASRcsubscriptASRASR_cASRc of LLM-Virus is 44.2, whereas AutoDAN, also based on evolutionary algorithms, achieves only 1.37. Furthermore, in Table I, we show that LLM-Virus also performs competitively and outstandingly on AdvBench, nearly achieving the best results across both open-source and closed-source LLMs. Notably, on GPT-3.5-Turbo, the ASRlsubscriptASRASR_lASRl of LLM-Virus is more than twice that of BlackDAN, which also utilizes evolutionary algorithms and holds the second-highest ASRlsubscriptASRASR_lASRl. V-C2 Transferability Jailbreak templates evolved by LLM-Virus demonstrate strong host transferability. In Figure 4, we present the ASRlsubscriptASRASR_lASRl when the most toxic individual evolved for the original host LLM is used for malicious queries on the new hosts. Notably, for the highly safety-aligned Claude-3.5-Haiku, only LLM-Viruses specifically evolved on it exhibit toxicity, while those transferred from other models fail. In contrast, GPT-3.5-Turbo and Llama-3.1-70B are the most susceptible to transfer attacks, with average transfer ASRlsubscriptASRASR_lASRl of 73.7 and 49.3, respectively. Additionally, the jailbreak templates evolved on GPT-4o-mini exhibit the strongest transfer infection capability (the most blue column), with transfer ASRlsubscriptASRASR_lASRl of 54.2 and 75.0 on GPT-4o and Llama-3.1-8B, respectively, even surpassing their original ASRlsubscriptASRASR_lASRl values of 31.7 and 34.6, respectively. Figure 4: Jailbreak attack transferability (ASRlsubscriptASRASR_lASRl) from original host LLM to new host LLM on AdvBench. V-C3 Perplexity & Time LLM-Virus demonstrate the outstanding performance in terms of perplexity and time cost. As shown in Table IV, gradient-based GCG exhibit very high perplexity (1532.2), making them easily defended by a simple perplexity filter. Our approach achieves an average perplexity comparable to existing work (45.1 and 46.5), and is closer to manually written jailbreak texts (23.0). Furthermore, due to Localized Evolution strategy (Step I) and parallelism of evolutionary process, the average time cost per harmful action template for GPT-3.5-Turbo is 1.2 minutes (five parallel workers), as shown in Table IV, only 110110 110divide start_ARG 1 end_ARG start_ARG 10 end_ARG of AutoDAN and nearly half of BlackDAN. The time cost can be further reduced with lower |r|||subscript |D_r||D|divide start_ARG | Ditalic_r | end_ARG start_ARG | D | end_ARG or higher parallelism in evolution. TABLE IV: Perplexity and Time Comparison. Method Perplexity Time Per Sample Handcrafted DAN 23.0 - GCG 1532.2 15min AutoDAN 46.5 12min BlackDAN - 2min LLM-Virus (Ours) 45.1 1.2min V-D Ablation Study In this section, we conduct ablation experiments to investigate the effects of several settings and modules in LLM-Virus. We consider only remove: Step I (Strain Collection), mutation, crossover and only change: the temperature settings (affect LLM generation diversity) for mutation and crossover, population size. As shown in the top of Figure 5: Temperature. Compared with base, temperature=2 causes LLM-Virus losing its optimization capability (ASRcsubscriptASRASR_cASRc fluctuating around 47.7), while temperature=0 results in slower improvement (Generation 8). This highlights the importance of an appropriate temperature for LLM evolution [46]. Evolutionary Operators. Additionally, only removing mutation or crossover reduces the search space, leading to a decrease in the final ASRcsubscriptASRASR_cASRc from 80.8 to 66.9 and 71.5, respectively. Replacing heuristic mutation/crossover with normal operators in previous works leads to a slight decrease of ASRcsubscriptASRASR_cASRc in early generations, but much longer character length (around 2×2×2 ×). This proves the advantages of our proposed heuristic mutation/crossover in terms of multi-objective optimization. Initialization & Size. Removing Strain Collection results in a 26.8% drop (80.8 → 59.2) in ASRcsubscriptASRASR_cASRc, demonstrating its necessity, while N=20 and N=5 are not better choices in terms of average performance, compared with base (N=10). V-E Case Study Finally, on the bottom part of Figure 5, we present a typical case of LLM-Virus. The case jailbreak template in last generation is obviously evoved from that in the first generation, but it exhibits higher ASRcsubscriptASRASR_cASRc (57.2%↑percent57.2absent57.2\% 57.2 % ↑) and lower character length (29.0%↓percent29.0absent29.0\% 29.0 % ↓) after the evolution in LLM-Virus. Figure 5: Ablation study of LLM-Virus (ASRcsubscriptASRASR_cASRc) on part of HarmBench in Local Evolution (Top) and case study (Bottom). VI Conclusion In this paper, inspired by natural virus infection and evolution, we propose LLM-Virus, an evolutionary jailbreak attack method based on evolutionary algorithm. To improve the toxicity and transferability of jailbreaks, we leverage LLMs as evolutionary operators (crossover, mutation and fitness) to search for potential jailbreak templates. Additionally, we incorporate transfer learning into the evolutionary process, reducing the high time cost associated with multiple rounds of evolution and numerous malicious queries. Our experiments demonstrate that LLM-Virus performs comparably or even better than several baselines across multiple safety benchmarks. We highlight the necessity and effectiveness of certain tailor-designed settings and components with extra ablation experiments. In conclusion, LLM-Virus advances the research on using LLM-enhanced evolutionary algorithms for LLM attacks, providing new insights for future studies. References [1] K. Sun, Y. E. Xu, H. Zha, Y. Liu, and X. L. Dong, “Head-to-tail: How knowledgeable are large language models (llm)? aka will llms replace knowledge graphs?” arXiv preprint arXiv:2308.10168, 2023. [2] X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y. Wang, R. Tang, and E. Chen, “Understanding the planning of llm agents: A survey,” arXiv preprint arXiv:2402.02716, 2024. [3] L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin et al., “Advancing llm reasoning generalists with preference trees,” arXiv preprint arXiv:2404.02078, 2024. [4] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 2023, p. 79–90. [5] M. Yu, S. Wang, G. Zhang, J. Mao, C. Yin, Q. Liu, Q. Wen, K. Wang, and Y. Wang, “Netsafe: Exploring the topological safety of multi-agent networks,” arXiv preprint arXiv:2410.15686, 2024. [6] Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. G. H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li, “Trustworthy llms: A survey and guideline for evaluating large language models’ alignment,” arXiv preprint arXiv:2308.05374, 2023. [7] Z. Dong, Z. Zhou, C. Yang, J. Shao, and Y. Qiao, “Attacks, defenses and evaluations for llm conversation safety: A survey,” arXiv preprint arXiv:2402.09283, 2024. [8] Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing, p. 100211, 2024. [9] J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang, “Beavertails: Towards improved safety alignment of llm via a human-preference dataset,” Advances in Neural Information Processing Systems, vol. 36, 2024. [10] X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” arXiv preprint arXiv:2310.03693, 2023. [11] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li et al., “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” arXiv preprint arXiv:2402.04249, 2024. [12] S. Yi, Y. Liu, Z. Sun, T. Cong, X. He, J. Song, K. Xu, and Q. Li, “Jailbreak attacks and defenses against large language models: A survey,” arXiv preprint arXiv:2407.04295, 2024. [13] X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” arXiv preprint arXiv:2310.04451, 2023. [14] E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt, “Automatically auditing large language models via discrete optimization,” in International Conference on Machine Learning. PMLR, 2023, p. 15 307–15 329. [15] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023. [16] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” arXiv preprint arXiv:2310.08419, 2023. [17] Q. Zeng, Z. Wang, Y.-m. Cheung, and M. Jiang, “Ask, attend, attack: A effective decision-based black-box targeted attack for image-to-text models,” arXiv preprint arXiv:2408.08989, 2024. [18] Y. Gong, Q. Zeng, D. Xu, Z. Wang, and M. Jiang, “Cross-modality attack boosted by gradient-evolutionary multiform optimization,” arXiv preprint arXiv:2409.17977, 2024. [19] X. Wang, V. S.-J. Huang, R. Chen, H. Wang, C. Pan, L. Sha, and M. Huang, “Blackdan: A black-box multi-objective approach for effective and contextual jailbreaking of large language models,” arXiv preprint arXiv:2410.09804, 2024. [20] J. Yu, X. Lin, Z. Yu, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,” arXiv preprint arXiv:2309.10253, 2023. [21] X. Li, S. Liang, J. Zhang, H. Fang, A. Liu, and E.-C. Chang, “Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms,” arXiv preprint arXiv:2402.14872, 2024. [22] R. Lange, Y. Tian, and Y. Tang, “Large language models as evolution strategies,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2024, p. 579–582. [23] E. Meyerson, M. J. Nelson, H. Bradley, A. Gaier, A. Moradi, A. K. Hoover, and J. Lehman, “Language model crossover: Variation through few-shot prompting,” arXiv preprint arXiv:2302.12170, 2023. [24] A. E. Brownlee, J. Callan, K. Even-Mendoza, A. Geiger, C. Hanna, J. Petke, F. Sarro, and D. Sobania, “Enhancing genetic improvement mutations using large language models,” in International Symposium on Search Based Software Engineering. Springer, 2023, p. 153–159. [25] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [26] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [27] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” Advances in Neural Information Processing Systems, vol. 36, 2024. [28] A. G. Chowdhury, M. M. Islam, V. Kumar, F. H. Shezan, V. Jain, and A. Chadha, “Breaking down the defenses: A comparative survey of attacks on large language models,” arXiv preprint arXiv:2403.04786, 2024. [29] E. Shayegani, M. A. A. Mamun, Y. Fu, P. Zaree, Y. Dong, and N. Abu-Ghazaleh, “Survey of vulnerabilities in large language models revealed by adversarial attacks,” arXiv preprint arXiv:2310.10844, 2023. [30] Z. Wei, Y. Wang, A. Li, Y. Mo, and Y. Wang, “Jailbreak and guard aligned language models with only few in-context demonstrations,” arXiv preprint arXiv:2310.06387, 2023. [31] X. Li, Z. Zhou, J. Zhu, J. Yao, T. Liu, and B. Han, “Deepinception: Hypnotize large language model to be jailbreaker,” arXiv preprint arXiv:2311.03191, 2023. [32] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” arXiv preprint arXiv:2308.03825, 2023. [33] F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” arXiv preprint arXiv:2211.09527, 2022. [34] Y. Tian, X. Yang, J. Zhang, Y. Dong, and H. Su, “Evil geniuses: Delving into the safety of llm-based agents,” arXiv preprint arXiv:2311.11855, 2023. [35] J. H. Holland, “Genetic algorithms and the optimal allocation of trials,” SIAM journal on computing, vol. 2, no. 2, p. 88–105, 1973. [36] H.-G. Beyer and H.-P. Schwefel, “Evolution strategies–a comprehensive introduction,” Natural computing, vol. 1, p. 3–52, 2002. [37] X. Yao, Y. Liu, and G. Lin, “Evolutionary programming made faster,” IEEE Transactions on Evolutionary computation, vol. 3, no. 2, p. 82–102, 1999. [38] J. R. Koza, “Genetic programming as a means for programming computers by natural selection,” Statistics and computing, vol. 4, p. 87–112, 1994. [39] T. Bartz-Beielstein, J. Branke, J. Mehnen, and O. Mersmann, “Evolutionary algorithms,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 4, no. 3, p. 178–195, 2014. [40] Q. Zhang and H. Li, “Moea/d: A multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on evolutionary computation, vol. 11, no. 6, p. 712–731, 2007. [41] Z. Wang, L. Cao, L. Feng, M. Jiang, and K. C. Tan, “Evolutionary multitask optimization with lower confidence bound-based solution selection strategy,” IEEE Transactions on Evolutionary Computation, 2024. [42] J. Branke, Evolutionary optimization in dynamic environments. Springer Science & Business Media, 2012, vol. 3. [43] Y. Jin and J. Branke, “Evolutionary optimization in uncertain environments-a survey,” IEEE Transactions on evolutionary computation, vol. 9, no. 3, p. 303–317, 2005. [44] Z. Shen, “Llm with tools: A survey,” arXiv preprint arXiv:2409.18807, 2024. [45] Y. Li, H. Wen, W. Wang, X. Li, Y. Yuan, G. Liu, J. Liu, W. Xu, X. Wang, Y. Sun et al., “Personal llm agents: Insights and survey about the capability, efficiency and security,” arXiv preprint arXiv:2401.05459, 2024. [46] S. Liu, C. Chen, X. Qu, K. Tang, and Y.-S. Ong, “Large language models as evolutionary optimizers,” in 2024 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2024, p. 1–8. [47] F. Liu, X. Lin, Z. Wang, S. Yao, X. Tong, M. Yuan, and Q. Zhang, “Large language model for multi-objective evolutionary optimization,” arXiv preprint arXiv:2310.12541, 2023. [48] Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y. Yang, “Connecting large language models with evolutionary algorithms yields powerful prompt optimizers,” arXiv preprint arXiv:2309.08532, 2023. [49] J. Gao, H. Xu, H. Shi, X. Ren, L. Philip, X. Liang, X. Jiang, and Z. Li, “Autobert-zero: Evolving bert backbone from scratch,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 10, 2022, p. 10 663–10 671. [50] S. Yuan, K. Song, J. Chen, X. Tan, D. Li, and D. Yang, “Evoagent: Towards automatic multi-agent generation via evolutionary algorithms,” arXiv preprint arXiv:2406.14228, 2024. [51] B. Kazimipour, X. Li, and A. K. Qin, “A review of population initialization techniques for evolutionary algorithms,” in 2014 IEEE congress on evolutionary computation (CEC). IEEE, 2014, p. 2585–2592. [52] T. Blickle and L. Thiele, “A comparison of selection schemes used in evolutionary algorithms,” Evolutionary Computation, vol. 4, no. 4, p. 361–394, 1996. [53] Z. Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, and Y. Li, “How alignment and jailbreak work: Explain llm safety through intermediate hidden states,” arXiv preprint arXiv:2406.05644, 2024. [54] Y. Wen, N. Jain, J. Kirchenbauer, M. Goldblum, J. Geiping, and T. Goldstein, “Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery,” Advances in Neural Information Processing Systems, vol. 36, 2024. [55] C. Guo, A. Sablayrolles, H. Jégou, and D. Kiela, “Gradient-based adversarial attacks against text transformers,” arXiv preprint arXiv:2104.13733, 2021. [56] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh, “Universal adversarial triggers for attacking and analyzing nlp,” arXiv preprint arXiv:1908.07125, 2019. [57] T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh, “Autoprompt: Eliciting knowledge from language models with automatically generated prompts,” arXiv preprint arXiv:2010.15980, 2020. [58] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,” arXiv preprint arXiv:2202.03286, 2022. [59] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,” arXiv preprint arXiv:2312.02119, 2023. [60] Y. Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi, “How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms,” arXiv preprint arXiv:2401.06373, 2024. [61] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, p. 1671–1685. [62] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. [63] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023), vol. 2, no. 3, p. 6, 2023. [64] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé et al., “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024. [65] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” Advances in Neural Information Processing Systems, vol. 33, p. 5776–5788, 2020. [66] J. MacQueen et al., “Some methods for classification and analysis of multivariate observations,” in Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, vol. 1, no. 14. Oakland, CA, USA, 1967, p. 281–297.

Instant research discovery