Paper deep dive
Let the Bees Find the Weak Spots: A Path Planning Perspective on Multi-Turn Jailbreak Attacks against LLMs
Yize Liu, Yunyun Hou, Aina Sui
Models: Gemma-9B-uncensored, GPT-3.5-Turbo, GPT-4-Turbo, LLaMA-2-7B, LLaMA-3.1-70B, LLaMA-3.1-8B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/11/2026, 1:00:29 AM
Summary
The paper introduces ABC, an enhanced Artificial Bee Colony algorithm that models multi-turn jailbreak attacks against LLMs as a path planning problem on a dynamically weighted graph. By treating attack strategies as nodes and their combinations as paths, the method optimizes for high attack success rates (ASR) while minimizing query overhead, achieving over 90% ASR on various models with significantly fewer queries than existing baselines.
Entities (5)
Relation Signals (3)
ABC → optimizes → Multi-turn jailbreak
confidence 95% · we propose ABC, an enhanced Artificial Bee Colony algorithm for multi-turn jailbreaks
Multi-turn jailbreak → targets → LLM
confidence 95% · utilizing multi-turn jailbreaks to identify potential vulnerabilities in LLMs
ABC → testedon → JailBreakBench
confidence 95% · Extensive experiments were conducted on the JailbreakBench benchmark
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) have been widely deployed across various applications, yet their potential security and ethical risks have raised increasing concerns. Existing research employs red teaming evaluations, utilizing multi-turn jailbreaks to identify potential vulnerabilities in LLMs. However, these approaches often lack exploration of successful dialogue trajectories within the attack space, and they tend to overlook the considerable overhead associated with the attack process. To address these limitations, this paper first introduces a theoretical model based on dynamically weighted graph topology, abstracting the multi-turn attack process as a path planning problem. Based on this framework, we propose ABC, an enhanced Artificial Bee Colony algorithm for multi-turn jailbreaks, featuring a collaborative search mechanism with employed, onlooker, and scout bees. This algorithm significantly improves the efficiency of optimal attack path search while substantially reducing the average number of queries required. Empirical evaluations on three open-source and two proprietary language models demonstrate the effectiveness of our approach, achieving attack success rates above 90\% across the board, with a peak of 98\% on GPT-3.5-Turbo, and outperforming existing baselines. Furthermore, it achieves comparable success with only 26 queries on average, significantly reducing red teaming overhead and highlighting its superior efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2511.03271
- Canonical: https://arxiv.org/abs/2511.03271
Trouble viewing inline? Open PDF directly →
Full Text
47,099 characters extracted from source content.
Expand or collapse full text
Let the Bees Find the Weak Spots: A Path Planning Perspective on Multi-Turn Jailbreak Attacks against LLMs Yize Liu 1 , Yunyun Hou 1 , Aina Sui 1 * 1 School of Computer and Cyberspace Security, Communication University of China, Beijing, China Abstract Large Language Models (LLMs) have been widely deployed across various applications, yet their potential security and ethical risks have raised increasing concerns. Existing re- search employs red teaming evaluations, utilizing multi-turn jailbreaks to identify potential vulnerabilities in LLMs. How- ever, these approaches often lack exploration of successful dialogue trajectories within the attack space, and they tend to overlook the considerable overhead associated with the at- tack process. To address these limitations, this paper first in- troduces a theoretical model based on dynamically weighted graph topology, abstracting the multi-turn attack process as a path planning problem. Based on this framework, we pro- pose ABC, an enhanced Artificial Bee Colony algorithm for multi-turn jailbreaks, featuring a collaborative search mech- anism with employed, onlooker, and scout bees. This algo- rithm significantly improves the efficiency of optimal attack path search while substantially reducing the average num- ber of queries required. Empirical evaluations on three open- source and two proprietary language models demonstrate the effectiveness of our approach, achieving attack success rates above 90% across the board, with a peak of 98% on GPT-3.5- Turbo, and outperforming existing baselines. Furthermore, it achieves comparable success with only 26 queries on average, significantly reducing red teaming overhead and highlighting its superior efficiency. Introduction Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini have exhibited remarkable generalization per- formance in natural language processing and have been widely adopted across a variety of applications, includ- ing dialogue systems and code generation (OpenAI et al. 2024). However, as large-scale deployment of these mod- els continues to expand, concerns regarding their security have become increasingly prominent (Barman, Guo, and Conlan 2024). Attackers can exploit specific techniques (Yi et al. 2024; Li et al. 2024) to induce models into generat- ing harmful, discriminatory, or ethically inappropriate con- tent, thereby posing significant risks to public safety. Al- though researchers have proposed various techniques, such as reinforcement learning and Direct Preference Optimiza- tion (DPO) (Rafailov et al. 2023) to enhance model align- ment and safety (Dai et al. 2023; Liu et al. 2023), existing * Corresponding author. defenses remain insufficient to fully prevent malicious at- tacks. Currently, security evaluation of LLMs primarily relies on red teaming (Ganguli et al. 2022; OpenAI 2023), which identifies model vulnerabilities by simulating adversarial be- haviors. Despite the lack of standardized evaluation crite- ria, two key objectives have gradually become widely rec- ognized in practice: one is to maximize the Attack Success Rate (ASR) so as to comprehensively expose security weak- nesses across diverse scenarios; the other is to minimize the average number of queries, thereby reducing computa- tional resource consumption and enhancing evaluation effi- ciency. Early red teaming efforts predominantly employed single-turn attacks, which trigger model violations through one-time adversarial prompt construction, including man- ual generation (Anil et al. 2024), white-box attacks (Zou et al. 2023), and LLM-assisted attacks (Sitawarin et al. 2024; Shah et al. 2023; Yu et al. 2023). However, with continuous advancements in model alignment techniques, the success rate of single-turn attacks has significantly declined, limit- ing their effectiveness in reliably identifying model vulnera- bilities. Recent studies have demonstrated that multi-turn attack methods can effectively overcome many of the limitations faced by single-turn attacks in identifying model vulnera- bilities (Chao et al. 2025). By emulating prolonged adver- sarial interactions with target models in realistic dialogue scenarios, these methods allow for dynamic refinement and adaptation of attack strategies, significantly enhancing their strategic flexibility, targeting precision, and overall effec- tiveness. A growing body of research on multi-turn jailbreak attacks (Du et al. 2025; Ding et al. 2024) adopts strategies that fragment adversarial intent across multiple dialogue turns, progressively uncovering and exploiting security vul- nerabilities through iterative interactions. In addition, sev- eral studies have integrated specially trained attacker agent models into the multi-turn dialogue process, where they serve as key components for actively steering the evolution of attack strategies. These agents are typically fine-tuned on red-teaming objectives, enabling them to detect exploitable signals in model responses and generate contextually adap- tive follow-up prompts, thereby improving the overall effi- ciency and success rate of the attack chain. Despite the increased flexibility and success rates demon- arXiv:2511.03271v1 [cs.CR] 5 Nov 2025 strated by multi-turn red teaming in practice, current re- search still faces two key challenges. First, there is insuf- ficient attention to the overall attack space. Existing studies typically perform repeated attack attempts centered on the same query, utilizing proposed methods to generate multi- ple distinct attack strategy combinations from the available strategy set. We define the set of all possible attack strategy combinations as the attack space, encompassing all strategy sequences accessible to the attacker. Current approaches of- ten lack systematic exploration of the attack space and over- look the potential of mining attack paths from the global structure of strategy combinations. Second, high resource consumption and limited practicality remain significant ob- stacles. Some studies rely on dedicated attack models for strategy selection, which require substantial computational resources for training and iterative refinement. Others adopt exhaustive methods that repeatedly invoke attack strategies to uncover potential vulnerabilities. Both approaches signif- icantly increase attack costs and restrict the applicability of these methods in resource-constrained environments. To bridge this gap, this paper introduces a multi-turn jail- break attack method based on a dynamically weighted graph topology, and proposes an enhanced ABC algorithm that significantly reduces resource consumption while ensuring high ASR. Specifically, we formalize the search strategy within the attack space as a path planning problem on a directed weighted graph. In the constructed graph, nodes represent attack strategies, edges denote the combinatorial relationships between strategies, and edge weights corre- spond to the evaluated effectiveness of the respective strat- egy sequences. The attack objective is thus transformed into searching for an optimal path on the graph that is both short in length and high in attack success probability. Building on this, we propose an enhanced ABC algorithm composed of employed, onlooker, and scout bees, which col- laboratively exploit localized exploration and global solu- tion tracking to efficiently identify high-success paths within the large attack space. To accommodate the characteristics of multi-turn attacks, we optimize several core components of the algorithm. In particular, the fitness function is dis- cretized into five levels, and for responses in the interme- diate tiers, we extract and utilize informative cues to re- fine the guidance of subsequent attack strategies. This de- sign enhances both the granularity of path planning and the overall ASR. Extensive experiments were conducted on the JailbreakBench benchmark (Chao et al. 2024), involv- ing three open-source and two closed-source models. The results demonstrate that our method achieves significantly higher ASR than existing baselines on most models, while markedly reducing the average number of queries. This not only enhances attack effectiveness but also markedly lowers the cost of red teaming evaluations. Our contributions are as follows: • We introduce a theoretical model based on dynamically weighted graph topology to characterize the multi-turn attack space, formalizing the multi-turn attack problem as a path planning task. • We propose an enhanced Artificial Bee Colony (ABC) al- gorithm that enables dynamic and efficient search within the attack space, with a fitness function specifically opti- mized for multi-turn attack scenarios. • We conduct extensive experiments on the latest bench- mark datasets involving five large language models, demonstrating that our method significantly outperforms existing baselines in both ASR and average number of queries, thereby validating the superiority of the pro- posed approach. Related Work Research on the safety of large language models (LLMs) primarily comprises defenses and red teaming evaluation. Training-time defenses employ preference alignment and safety fine-tuning to improve refusal consistency. Inference- time defenses rely on system prompts and policy tem- plates, as well as input and output filtering or rewrit- ing, to enforce constraints during interaction. Red-teaming complements defenses by systematically uncovering failure modes in realistic conversational settings and quantifying the safety–utility trade-off. Against this backdrop, jailbreak attacks (Wei, Haghtalab, and Steinhardt 2023) are a key ad- versarial technique for revealing alignment vulnerabilities. By crafting covert and targeted inputs, attackers can bypass built-in safeguards and induce the model to produce content that may violate ethical norms, laws, or platform policies. Based on interaction mode and turn count, jailbreak attacks are generally classified into single-turn and multi-turn vari- ants. Single-turn jailbreak attacks. Single-turn jailbreak at- tacks construct adversarial prompts in a one-shot manner to elicit unsafe outputs from LLMs. These methods gener- ally fall into two categories. White-box approaches lever- age gradient-based optimization to manipulate model inputs or representations, as in GCG (Zou et al. 2023), which ap- plies greedy coordinate descent to generate effective adver- sarial prompts. Black-box methods rely on semantic ma- nipulations or algorithmic heuristics without gradient ac- cess. Examples include AutoDAN (Liu et al. 2024), Self- Cipher (Yuan et al. 2024), and PAP (Zeng et al. 2024), which adopt diverse optimization strategies. PRS (An- driushchenko, Croce, and Flammarion 2025) further per- forms random search over suffixes appended to benign prompts to maximize the target log-likelihood of unsafe completions. Despite their simplicity, single-turn attacks have become less effective due to improved alignment tech- niques, limiting their utility for evaluating models under stronger defenses. Multi-turn jailbreak attacks. Compared with single- turn attacks, multi-turn jailbreak attacks emulate realistic ad- versarial interactions by engaging the target model in itera- tive dialogues, gradually inducing it to produce boundary- violating outputs. PAIR (Chao et al. 2025) uses dialogue- driven iterative refinement to achieve semantic jailbreaks under black-box access, whereas CoA (Sun et al. 2024) couples a seed generator with toxicity escalation to pre- construct and adapt multi-turn attack chains in response to model feedback. Crescendo (Russinovich, Salem, and El- dan 2025) introduces a progressive multi-turn dialogue ap- proach that begins with benign queries and gradually lever- ages model outputs from previous turns to intensify the at- tack. GOAT (Pavlova et al. 2024) employs an attacker agent that adaptively selects attack strategies based on the tar- get model’s responses during the multi-round interaction. GALA (Chen et al. 2025) proposes a novel dual-dimensional learning mechanism, training the attacker agent using both global strategic planning and local prompt-level refinement. Path Planning and Swarm Intelligence. Path planning formulates decision making as finding shortest or constraint- satisfying paths on weighted graphs. Classical explicit meth- ods such as Dijkstra typically operate under static edge costs and admissible heuristics. When costs depend on in- teraction history, explicit search often suffers from fron- tier blow-up and fragile heuristics. Swarm-intelligence ap- proaches such as PSO and ABC maintain diversity through parallel neighborhood exploration, probabilistic selection, and restart mechanisms, and are widely used in combina- torial routing and navigation, offering a practical toolkit for long-horizon problems with expensive evaluations. Despite gains in ASR and realism, prior multi-turn stud- ies largely optimize strategy choice within individual dia- logues and rarely model history-dependent costs at a global level. Other lines rely on attacker-agent training or exten- sive query-based search, which increases evaluation cost and constrains scalability under realistic budgets. These gaps motivate adopting a planning-style attack search that achieves higher ASR while controlling attack cost. Methodology In this section, we provide a comprehensive description of the proposed dynamically weighted graph topology model and the enhanced Artificial Bee Colony (ABC) algorithm. Our approach abstracts the search for strategy combinations in multi-turn attacks as a path planning problem on a dynam- ically weighted graph topology and utilizes the enhanced ABC algorithm to intelligently explore and optimize attack paths. Attack Space Modeling To systematically characterize the multi-turn jailbreak pro- cess, we model the attack space as a directed weighted topol- ogy and, on this basis, construct a layered state graph to cap- ture path-dependent memory. Graph and paths. We model the multi-turn attack space as a directed complete graph topology G = (V,E). The vertex comprises the initial query node together with a set of attack-strategy nodes, and edges denote the sequential combinational relationships among these strategies. Attack strategies are sourced from prior work (Lin et al. 2024; Jiang et al. 2025) and community repositories, and are organized into several classes according to content characteristics. A path of length T is denoted p = (v 0 → v 1 → · → v T ), where v 0 denotes the initial harmful query. Path score. In this model, an edge weight is defined as the harmfulness score induced on the target model by a suc- cessor edge under a given prefix history. Let the path prefix be pref(p,T) = (v 0 → v 1 → · → v T−1 ), and let the terminal edge be e T = (v T−1 → v T ). Then, the resulting score of the entire path is defined as S T = w(pref(p,T),e T ),(1) where w denotes the target model’s harmfulness-scoring function under the current combination of strategies. When S T reaches the harm threshold a, the attack is considered successful. The objective is to find a successful attack path of minimal length. Layered state graph. To better represent that the same edge may take different weights under different historical prefixes, we extend the directed complete graph into a lay- ered state graph so that path memory is encoded as a static topology in the state space. We introduce a progress map- ping P(pref(p,t))∈0,...,a, which compresses any his- torical prefix into a finite progress level. Based on this, we construct layered state graph as ˆ G = ( ˆ V , ˆ E), ˆ V =(v,s)| v ∈ V, s∈0,...,a, (2) were (v,s) denotes the state of being at strategy v with pre- fix progress level s. If there exists a prefix pref(p,t) in the original graph with P(pref(p,t)) = s, and after extending by node u the new progress becomes s ′ = mina, P(pref(p,t), u).(3) then an edge (v,s)→ (u,s ′ ) exists in ˆ G. Search objective. We cast the goal of multi-turn jail- breaks as a path-planning problem over the layered state graph. Let the start state be (v 0 , 0), where v 0 is the initial harmful query, and let the goal set be (v,a) | v ∈ V. Within this finite, static space formed by strategy nodes and progress levels, the task is to reach any success state (v,a) in the fewest steps, equivalently, to solve a unit-weight shortest-path problem on ˆ G: min p⊆ ˆ G |p| s.t. (v 0 , 0) p −→ (v,a).(4) This reduction converts the original history dependence into static transitions across a finite set of levels while remaining strictly aligned with the attack’s success criterion. Theoretical Properties and Rationale Building on the graph modeling and problem analysis, we employ an improved ABC-based swarm optimization. Com- pared with traditional meta-heuristics, ABC is better aligned with this task: it treats multiple candidate prefixes as par- allel units, reuses high-quality prefix fragments during lo- cal edits, and employs probabilistic selection to mitigate single-instance misjudgments. Under an attack-dominated cost model, ABC reaches the success level in fewer rounds with lower implementation overhead. Meanwhile, when the theoretical graph topology is manageable, explicit shortest- path search (BFS) performs well. But in our setting, edge weights are highly prefix-dependent, and explicit expansion causes the state space and frontier to grow rapidly. By con- trast, ABC on an implicit layered space avoids enumerat- ing long histories and, through parallel probing, raises the per-round probability of effective progress, thus achieving higher efficiency and greater stability under limited evalua- tion budgets. We next analyze ABC’s typical properties in the multi-turn jailbreak setting as follows. Let m denote the population size, a the success-threshold level, s 0 the initial progress level, η the minimum effec- tive per-step level increment, q a uniform lower bound on the per-round progress probability, t the number of effective parallel trials per round, and C eval the per-evaluation model time cost. Time complexity. We bound the expected total time by E[T] = O C eval t a−1 X ℓ=s 0 1 η ℓ q ℓ ! .(5) Here, the summation term estimates the expected number of progress rounds from s 0 to a, and each round incurs t evaluations at a per-evaluation cost of C eval . Space complexity. The memory usage scales as Space ABC = O m a− s 0 η .(6) For comparison, explicit shortest-path search requires O(na) space with n = |V|, whereas ABC grows only lin- early with the population size m and the number of levels. Convergence. We obtain the following convergence bound: E[τ a ] ≤ a−1 X ℓ=s 0 1 η ℓ q ℓ .(7) Here, τ a denotes the hitting time to the success level a. The layered state space is finite and the success level is absorb- ing. With reuse of high-quality prefix fragments and a restart mechanism, these conditions yield the above upper bound on the expected hitting time. ABC Algorithm for Multi-Turn Attacks The Artificial Bee Colony (ABC) algorithm is a represen- tative swarm intelligence optimization method and a widely adopted approach for solving path planning problems. In this study, we employ ABC to efficiently search for candidate at- tack paths given an initial query, aiming to maximize success rates while minimizing query overhead and achieving an ef- fective balance between performance and cost. The ABC al- gorithm consists of three primary phases: the employed bee phase, the onlooker bee phase, and the scout bee phase. The overall procedure is illustrated in Algorithm 1. The popula- tion size determines the number of employed and onlooker bees, where each individual in the population corresponds to a candidate path and is associated with either an employed or an onlooker bee. In the employed bee phase, the algorithm first evaluates the fitness of each candidate attack path in the population, which measures the effectiveness of the path in the initial stage of the attack. This phase aims to construct the initial search space and lay the foundation for subsequent path op- timization. The fitness function scores each path based on the harmfulness of the target model’s response to the final Algorithm 1: Artificial Bee Colony for Multi-Turn Attack Require: Initial population P =r 1 ,r 2 ,...,r l ; max iter- ations N ; update threshold γ; selection size m Ensure: Best attack path r ∗ 1: for all r ∈ P do 2:UpdateCount(r)← 0 3:Fitness(r)← CalculateFitness(r) 4: end for 5: for t = 1 to N do 6: S ← Selection(P,m)// Extract high-quality paths from P 7: P ← (P \ S)∪Mutation(r)| r ∈ S 8: for all r ∈ P newly generated by Mutation do 9:Fitness(r)← CalculateFitness(r) 10: end for 11: for all r ∈ P do 12:if UpdateCount(r) ≥ γ and Fitness(r) < MaxFitness(P) then 13:r ← GenerateNewRoad() 14:Fitness(r) ←CalculateFitness(r); UpdateCount(r)← 0// Reinitialize stagnated path 15:end if 16: end for 17: if∃r ∈ P such that Fitness(r) = Target then 18:break 19: end if 20: end for 21: return Best(P) prompt in the path, thereby reflecting the attack potential of the current path. In the onlooker bee phase, the algorithm selects a subset of high-quality paths from the population according to their fitness scores, using a predefined ratio and probabilistic selection mechanism to form the candidate pool. Each candidate path is then subjected to a mutation process to further expand the search space and improve the algorithm’s ability to escape local optima. The mutation op- erations include random replacement, where an attack strat- egy within the path is substituted to explore alternative com- binations, and random insertion, where a new strategy node is added to enhance the path’s expressive capacity and in- crease its attack depth. In the scout bee phase, the algorithm monitors the activity and contribution of each individual in the population. If a path has not been updated over multiple iterations and its fitness remains suboptimal, it is considered to be stuck in a stagnation state. In such cases, the scout bee mechanism is triggered to restructure the local search space. Specifically, the stagnant path is replaced with a newly gen- erated random path, thereby enhancing search diversity and helping the algorithm avoid local optima. The components of the algorithm are described in detail as follows. Population Initialization. During the entire algorithm execution, the population size is kept constant. Initially, we construct a population composed of multiple paths, where the attack strategies in each path are uniformly and randomly selected from a predefined strategy set. To accommodate the MethodLLaMA-2-7B LLaMA-3.1-8B LLaMA-3.1-70B GPT-3.5 GPT-4 GCG3%//47%4% PAIR0%//71%34% PRS90%//93%78% Crescendo72%84.5%77%80.4%70.9% GOAT85.3%96.5%91%91.6%87.9% GALA/87%92%91%/ ABC (Ours)96%94%94%98%90% Table 1: Comparison of Attack Success Rate (ASR) across seven methods on five target models. The best and second-best results are highlighted in bold and underlined, respectively. The performance data for GCG, PAIR, and PRS were sourced from the official JailbreakBench leaderboard, while the results for Crescendo and GOAT were computed based on the number of successful queries out of 10 multi-turn attempts. need for search space expansion in subsequent stages, while also minimizing the number of queries in multi-turn attacks, the initial path length is restricted to no more than three strategy nodes. The generation process of the population and paths can be formalized as: P =r 1 ,r 2 ,...,r l , r i = (t i1 ,t i2 ,...,t ik )(8) where each t ij is independently drawn from a uniform dis- tribution over the predefined strategy set T , i.e., t ij ∼U(T), and the initial path length is constrained such that|r|≤ 3. Selection. The selection module employs a tournament selection mechanism. Specifically, the algorithm randomly selects a subset of paths from the entire population to form a candidate set. Within this candidate set, the path with the highest fitness score is chosen as the winner of the current tournament. This process is repeated multiple times to pro- gressively filter out higher-performing paths, thereby con- structing the final selection set. The selection set can be rep- resented by the formula: S = arg max r∈R j Fitness(r) n j=1 (9) where R j is the candidate subset for the j-th tournament, consisting of m paths randomly sampled from the popula- tion, and n tournaments are conducted in total. Mutation. Mutation is a crucial module for maintain- ing population diversity and discovering superior paths. For each selected path, the algorithm performs one of two mu- tation operations based on a probability coefficient. The first operation is random replacement, which involves selecting a new attack strategy at a random position in the path to re- place the original strategy, thereby exploring new possible strategy combinations. The second operation is random in- sertion, where a new attack strategy node is appended at the end of the path, ensuring that the total path length does not exceed a predefined maximum limit. Both operations allow the newly added strategies to duplicate existing ones within the path, expanding the search space and increasing the al- gorithm’s chances of finding the optimal path. The mutation operation m(r) can be represented as follows: m(r) = (t 1 ,...,t p−1 ,t ′ ,t p+1 ,...,t k ), β (t 1 ,...,t k ,t ′ ),1− β (10) where t ′ is sampled from a uniform distribution over the pre- defined strategy set T , i.e., t ′ ∼ U(T), and the augmented path in the second case satisfies |t 1 ,...,t k ,t ′ | ≤ L max , where L max is the maximum allowed path length. Fitness Evaluation. Previous studies (Russinovich, Salem, and Eldan 2025) have shown that embedding harmful queries in batches across multiple dialogue turns can signif- icantly improve ASR. Inspired by this approach, we design a targeted fitness function to finely evaluate the harmfulness of the model’s response at each turn. Specifically, the fit- ness function categorizes model responses into five levels: level 1 indicates a completely safe response with no harm- ful content, while level 5 represents a fully harmful output generated by the model. Levels 2 through 4 correspond to responses containing partial harmful content, from which the attacker can extract useful information to construct more targeted adversarial prompts for the next turn, progressively advancing the attack toward full success. To implement this mechanism, we design a dedicated prompt template that in- structs the model to extract information from its response that is useful for subsequent attack prompts, simulating a semantic extraction process in an uncensored environment. The extracted key content is then combined with the initial query to form the input for the next turn, enabling a gradual and iterative progression of the multi-turn attack. Experiment Experimental Setup Dataset. We adopt JailbreakBench (Chao et al. 2024) as the primary benchmark to evaluate the effectiveness of our proposed method. This dataset comprises 100 diverse sam- ples of harmful behaviors. Among them, 55% are originally collected instances, while the remaining samples are care- fully curated from existing benchmark datasets such as Ad- vBench (Zou et al. 2023) and HarmBench (Mazeika et al. 2024). All behavior categories in the dataset are aligned with OpenAI’s safety policies, spanning a wide range of high- risk domains, including but not limited to phishing, privacy MethodLLaMA-2-7B LLaMA-3.1-8B LLaMA-3.1-70B GPT-3.5 GPT-4 Crescendo5050505050 GOAT5050505050 ABC (Ours)27.927.727.510.336.6 Table 2: Average number of queries for multi-turn attack methods across five target models. Lower values indicate higher attack efficiency. The query numbers listed for baseline methods correspond exactly to the ASR results in Table 1. Note that the 50- query budget for GOAT and Crescendo is strictly derived from the evaluation protocol specified in their original papers: for each harmful seed, the attacker independently interacts with the target model k = 10 times, with 5 turns per interaction, and an attack is considered successful if at least one of the 10 dialogues produces an unsafe output. violations, hate speech, and illegal activities. This makes JailbreakBench a more advanced and comprehensive bench- mark for assessing the robustness and safety of language models under adversarial scenarios. Scoring Setup. We employ StrongREJECT (Souly et al. 2024) as the evaluator to assess the harmfulness of model outputs. Compared to existing evaluation tools, StrongRE- JECT offers a more rigorous assessment by jointly consid- ering two critical dimensions: the target model’s willingness to refuse harmful queries and its tendency to provide action- able or harmful content. In our experiments, we use an open- source version of StrongREJECT fine-tuned on Gemma 2B, which retains the high accuracy of rubric-based evaluation while significantly reducing computational overhead. This makes it well-suited for our resource-constrained scenario, enabling efficient and reliable evaluation of adversarial out- puts. Evaluation Metrics. We adopt the Attack Success Rate (ASR) as the primary evaluation metric, defined as the pro- portion of successfully attacked samples relative to the total number of test cases. Additionally, to assess the resource ef- ficiency of different algorithms within the same category, we report the average number of queries required per successful attack as a secondary metric. Attacker LLM. The attacker language model is used to assist in generating adversarial prompts, and its primary requirement is low sensitivity to safety constraints when faced with potentially harmful instructions. In this work, we employ a non-aligned “helpful-only” model Gemma-9B- uncensored as the adversarial prompt generator. Trained on a mixture of both benign and harmful data, the model is de- signed to prioritize instruction-following without regard for safety constraints, as it has not been subjected to any safety fine-tuning or filtering. Importantly, no explicit red-teaming samples or jailbreak-specific data are included during its pretraining or fine-tuning. This ensures that its adversarial capabilities in our experiments arise purely from the struc- ture and semantics of the input prompts, rather than from memorization or prior exposure to attack-specific examples. Target LLMs. We conducted evaluations on both open- source and closed-source models. For open-source models, we selected LLaMA family members, including LLaMA 2 7b (Touvron et al. 2023), LLaMA 3.1 8b, and LLaMA 3.1 70b, to assess the effectiveness of our method across differ- ent parameter scales. For closed-source models, we selected GPT-4-Turbo (OpenAI et al. 2024) and GPT-3.5-Turbo, both of which feature advanced safety mechanisms. Evaluating our method on these models offers a more rigorous valida- tion of its practical effectiveness. Algorithm Hyperparameters. To strike an effective bal- ance between attack efficiency and query cost, we config- ured two distinct sets of hyperparameters tailored to dif- ferent model groups. For smaller-scale models, including LLaMA 2 7b, LLaMA 3.1 8b, and GPT-3.5-Turbo, we capped both the number of iterations and maximum path length at 3, with a population size of 5. For the larger-scale model group, such as LLaMA 3.1 70b and GPT-4-Turbo, due to their increased complexity and higher attack diffi- culty, we appropriately increased the number of iterations and path length to 5, and expanded the population size to 10, thereby enhancing the search space and the algorithm’s exploration capability to improve attack effectiveness. Baselines. We compare the proposed ABC algorithm with several jailbreak attack methods, including single- turn attacks such as GCG (Zou et al. 2023) and PRS (Andriushchenko, Croce, and Flammarion 2025), as well as multi-turn attacks like PAIR (Chao et al. 2025), Crescendo (Russinovich, Salem, and Eldan 2025), GOAT (Pavlova et al. 2024), and GALA (Chen et al. 2025).Among these, Crescendo and GOAT perform repeated multi-turn attacks to conceal harmful queries, while GALA leverages dual-dimensional learning with trained agents to assist in the attack process. Results We conduct a systematic comparison between ABC and sev- eral baseline approaches from three key dimensions: effec- tiveness, efficiency, and generalizability. The evaluation fo- cuses on three specific metrics: ASR, average number of queries, and performance across diverse attack scenarios. These metrics align with the core objectives of jailbreak at- tacks: ensuring attack success, minimizing testing costs, and enhancing generalization. The ASR reflects the method’s ability to bypass existing safety mechanisms, while the aver- age number of queries captures the resource cost associated with executing the attack. Additionally, fine-grained evalua- tions across different attack types help reveal the method’s stability and adaptability in varied scenarios, offering impor- tant insights into its real-world applicability. In the following sections, we analyze the results from these three perspec- tives. Figure 1: Attack performance of ABC across different harm categories. Left: Attack Success Rate (ASR); Right: Average Number of Queries. The radar plot and horizontal bar chart respectively summarize ABC’s effectiveness and efficiency across 10 types of harmful behaviors defined in JailbreakBench. Attack Success Rate We compare our method with seven baseline approaches across five target models, with the re- sults summarized in Table 1. As shown in the table, in the single-turn attack setting, PRS achieves a success rate of 93% on GPT-3.5-Turbo, standing out among baseline meth- ods. In the multi-turn scenario, GOAT and GALA perform relatively well, with both methods attaining an average ASR close to 90% across the five target models. In contrast, ABC demonstrates even stronger performance, consistently out- performing all baseline approaches on nearly every target model. This highlights its superior capability in red-teaming evaluations. Notably, ABC achieves over 90% success on all tested models, reaching as high as 96% on LLaMA 2 7b and 98% on GPT-3.5-Turbo. These results indicate that our method not only offers high attack effectiveness but also ex- hibits strong robustness, making it highly valuable for prac- tical applications such as red-teaming, safety assessment, and the optimization of defense strategies. Average Number of Queries To demonstrate that ABC not only significantly improves ASR but also reduces eval- uation overhead in multi-turn dialogue settings, we conduct a statistical analysis of the average number of queries used in our experiments. Note that the 50-query budget reported for GOAT and Crescendo is strictly derived from the evalu- ation protocol in their original papers, where each harmful seed is queried k = 10 times with 5 turns per interaction. The results are presented in Table 2. As shown in the com- parison results, ABC consistently achieves lower average number of queries than the strong baselines Crescendo and GOAT across all five target models. On GPT-3.5-Turbo, for example, it requires only 10 queries on average to complete a successful jailbreak, compared to 50 for the baselines. Even on the more complex GPT-4-Turbo, ABC maintains a low query cost, significantly reducing the overall burden of red-teaming while effectively balancing ASR and query efficiency. Moreover, other multi-turn attack approaches, such as GALA, rely on training dedicated attacker models to gen- erate adversarial prompts. While this can improve ASR, it introduces considerable resource and time costs due to model fine-tuning. In contrast, our ABC method efficiently searches for attack paths through optimization alone, with- out any attacker model training or auxiliary learning mod- ules. This substantially reduces both implementation barri- ers and system overhead, making ABC more lightweight and easier to deploy in practical red teaming. These advantages further highlight its superior efficiency and usability over ex- isting baselines. Diverse Scenario Analysis To further evaluate the gener- alization capability of our proposed method in diverse attack scenarios, we conduct a fine-grained analysis of ABC using the JailbreakBench dataset. This benchmark categorizes all test samples into ten representative types of harmful behav- ior, enabling a systematic assessment of the method’s adapt- ability and robustness across different attack categories. We aggregate the ASR of ABC across all target mod- els according to these ten scenario categories and visualize the results. As shown in Figure 1, ABC consistently main- tains an ASR above 85% across all categories, and even ex- ceeds 95% in particularly challenging cases such as malware and hacking. These results demonstrate that ABC not only achieves strong overall attack performance but also exhibits consistent and reliable effectiveness across a wide range of adversarial intents, highlighting its generality and practical utility. Further analysis shows that the top five categories with the highest ASR also correspond to the five categories with the lowest average number of queries. This phenomenon is likely influenced by both the characteristics of the strategies in the initial strategy pool and the security mechanisms em- ployed by the target models. In these scenarios, the tested models exhibit weaker defenses against jailbreak attacks, indicating an urgent need for strengthening and optimizing protective measures. Notably, this result also reveals an in- teresting relationship between ASR and average number of queries overhead: as the ASR increases, the required num- ber of queries actually decreases. This observation differs from the common perception that these two metrics are typ- ically at odds. Although further experimental validation is needed, achieving higher attack effectiveness while reducing resource consumption remains the core objective of red team testing. The performance of the ABC method strongly aligns with this goal, highlighting its significant practical value in security assessment. Limitation and Future Work Despite the strong empirical performance of our enhanced Artificial Bee Colony (ABC) algorithm across multiple large language models, several limitations remain. First, the ini- tial population in our current implementation is generated purely at random. While this ensures diversity to some ex- tent, it may lead to a lack of high-quality candidate paths in the early stages, potentially hindering convergence effi- ciency and overall attack effectiveness. Future work may ex- plore heuristic-guided or quality-aware initialization strate- gies to improve search performance. Second, the success of the attack also depends on the quality of the predefined strategy set. If the strategy pool contains a large number of low-quality or redundant attack actions, it may reduce the likelihood of identifying effective paths. Therefore, develop- ing automatic filtering mechanisms and enhancing the over- all quality of the strategy space are promising directions for further improvement. Conclusion In this study, we propose a dynamically weighted graph topology to characterize the multi-turn attack space, for- malizing the dynamic search process of jailbreak attacks as a path planning problem. Building on this formulation, we develop ABC—an enhanced Artificial Bee Colony algo- rithm tailored for multi-turn attacks. ABC demonstrates sig- nificant advantages in both optimal path search and query cost control, enabling high attack success rates with sub- stantially reduced query overhead. Extensive experimental results show that ABC achieves over 90% ASR across five target models, requiring only 26 queries on average per successful attack, significantly outperforming existing base- lines. Our work aims to facilitate the development of more robust defense mechanisms by providing an efficient and ef- fective attack strategy, thereby offering a more comprehen- sive safeguard against jailbreak threats in real-world scenar- ios. References Andriushchenko, M.; Croce, F.; and Flammarion, N. 2025. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. In The Thirteenth International Confer- ence on Learning Representations. Anil, C.; DURMUS, E.; Rimsky, N.; Sharma, M.; Benton, J.; Kundu, S.; Batson, J.; Tong, M.; Mu, J.; Ford, D. J.; et al. 2024. Many-shot Jailbreaking. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS), volume 37, 129696–129742. Barman, D.; Guo, Z.; and Conlan, O. 2024. The Dark Side of Language Models: Exploring the Potential of LLMs in Multimedia Disinformation Generation and Dissemination. Machine Learning with Applications, 16: 100545. Chao, P.; Debenedetti, E.; Robey, A.; Andriushchenko, M.; Croce, F.; Sehwag, V.; Dobriban, E.; Flammarion, N.; Pap- pas, G. J.; Tram ` er, F.; et al. 2024.JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Lan- guage Models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Chao, P.; Robey, A.; Dobriban, E.; Hassani, H.; Pappas, G. J.; and Wong, E. 2025. Jailbreaking Black-Box Large Language Models in Twenty Queries. In Proceedings of the 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 23–42. IEEE. Chen, S.; Yu, X.; Mehrabi, N.; Gupta, R.; Yu, Z.; and Jia, R. 2025.Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learning. arXiv:2504.01278. Dai, J.; Pan, X.; Sun, R.; Ji, J.; Xu, X.; Liu, M.; Wang, Y.; and Yang, Y. 2023. Safe RLHF: Safe Reinforcement Learn- ing from Human Feedback. arXiv:2310.12773. Ding, P.; Kuang, J.; Ma, D.; Cao, X.; Xian, Y.; Chen, J.; and Huang, S. 2024. A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily. arXiv:2311.08268. Du, X.; Mo, F.; Wen, M.; Gu, T.; Zheng, H.; Jin, H.; and Shi, J. 2025. Multi-Turn Jailbreaking Large Language Models via Attention Shifting. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 39, 23814–23822. Ganguli, D.; Lovitt, L.; Kernion, J.; Askell, A.; Bai, Y.; Kadavath, S.; Mann, B.; Perez, E.; Schiefer, N.; Ndousse, K.; et al. 2022. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv:2209.07858. Jiang, L.; Rao, K.; Han, S.; Ettinger, A.; Brahman, F.; Ku- mar, S.; Mireshghallah, N.; Lu, X.; Sap, M.; Choi, Y.; et al. 2025. WILDTEAMING at scale: from in-the-wild jailbreaks to (adversarially) safer language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24. Red Hook, NY, USA: Curran Associates Inc. ISBN 9798331314385. Li, N.; Han, Z.; Steneker, I.; Primack, W.; Goodside, R.; Zhang, H.; Wang, Z.; Menghini, C.; and Yue, S. 2024. LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet. arXiv:2408.15221. Lin, L.; Mu, H.; Zhai, Z.; Wang, M.; Wang, Y.; Wang, R.; Gao, J.; Zhang, Y.; Che, W.; Baldwin, T.; et al. 2024. Against The Achilles’ Heel: A Survey on Red Teaming for Genera- tive Models. arXiv:2404.00629. Liu, R.; Yang, R.; Jia, C.; Zhang, G.; Zhou, D.; Dai, A. M.; Yang, D.; and Vosoughi, S. 2023. Training Socially Aligned Language Models on Simulated Social Interactions. arXiv:2305.16960. Liu, X.; Xu, N.; Chen, M.; and Xiao, C. 2024. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In The Twelfth International Conference on Learning Representations. Mazeika, M.; Phan, L.; Yin, X.; Zou, A.; Wang, Z.; Mu, N.; Sakhaee, E.; Li, N.; Basart, S.; Li, B.; et al. 2024. Harm- Bench: A Standardized Evaluation Framework for Auto- mated Red Teaming and Robust Refusal. arXiv:2402.04249. OpenAI. 2023.OpenAI Red Teaming Network.https: //openai.com/index/red-teaming-network. Accessed: 2024- 08-16. OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; et al. 2024.GPT-4 Technical Report. arXiv:2303.08774. Pavlova, M.; Brinkman, E.; Iyer, K.; Albiero, V.; Bitton, J.; Nguyen, H.; Li, J.; Ferrer, C. C.; Evtimov, I.; and Grattafiori, A. 2024. Automated Red Teaming with GOAT: the Genera- tive Offensive Agent Tester. arXiv:2410.01606. Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Er- mon, S.; and Finn, C. 2023. Direct Preference Optimiza- tion: Your Language Model is Secretly a Reward Model. In Thirty-seventh Conference on Neural Information Process- ing Systems. Russinovich, M.; Salem, A.; and Eldan, R. 2025. Great, Now Write an Article About That: The Crescendo Multi- Turn LLM Jailbreak Attack. arXiv:2404.01833. Shah, R.; Feuillade-Montixi, Q.; Pour, S.; Tagade, A.; Casper, S.; and Rando, J. 2023.Scalable and Transfer- able Black-Box Jailbreaks for Language Models via Persona Modulation. arXiv:2311.03348. Sitawarin, C.; Mu, N.; Wagner, D.; and Araujo, A. 2024. PAL: Proxy-Guided Black-Box Attack on Large Language Models. arXiv:2402.09674. Souly, A.; Lu, Q.; Bowen, D.; Trinh, T.; Hsieh, E.; Pandey, S.; Abbeel, P.; Svegliato, J.; Emmons, S.; Watkins, O.; et al. 2024.A StrongREJECT for Empty Jailbreaks. arXiv:2402.10260. Sun, X.; Zhang, D.; Yang, D.; Zou, Q.; and Li, H. 2024. Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles. arXiv:2408.04686. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. Wei, A.; Haghtalab, N.; and Steinhardt, J. 2023. Jailbro- ken: How Does LLM Safety Training Fail? In Thirty-seventh Conference on Neural Information Processing Systems. Yi, S.; Liu, Y.; Sun, Z.; Cong, T.; He, X.; Song, J.; Xu, K.; and Li, Q. 2024. Jailbreak Attacks and Defenses Against Large Language Models: A Survey. arXiv:2407.04295. Yu, J.; Lin, X.; Yu, Z.; and Xing, X. 2023. GPTFuzzer: Red Teaming Large Language Models with Auto-Generated Jail- break Prompts. arXiv:2309.10253. Yuan, Y.; Jiao, W.; Wang, W.; tse Huang, J.; He, P.; Shi, S.; and Tu, Z. 2024.GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. In Proceedings of the Twelfth International Conference on Learning Representa- tions (ICLR). Zeng, Y.; Lin, H.; Zhang, J.; Yang, D.; Jia, R.; and Shi, W. 2024. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Human- izing LLMs. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), 14322–14350. Bangkok, Thailand: Association for Computational Linguistics. Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023.Universal and Transfer- able Adversarial Attacks on Aligned Language Models. arXiv:2307.15043.