Paper deep dive

Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking

Aldan Creo, Raul Castro Fernandez, Manuel Cebrian

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 39

Models: Gemma-3-1B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/11/2026, 12:59:49 AM

Summary

This paper presents a large-scale empirical analysis of over 2 million real-world conversations to evaluate the complexity of LLM jailbreaking. Contrary to the expectation that jailbreaking requires significantly higher complexity than normal interactions, the study finds that jailbreak attempts exhibit complexity levels comparable to standard conversations. The research demonstrates that jailbreak development is bounded by human ingenuity rather than escalating in a power-law fashion, and that while user attack complexity remains stable, assistant safety mechanisms have improved over time.

Entities (5)

Jailbreaking · security-threat · 100%LLM · technology · 100%AI Safety · field-of-study · 95%WildChat · dataset · 95%Gemma-3-1B · model · 90%

Relation Signals (3)

Jailbreaking → targets → LLM

confidence 100% · jailbreaking, which refers to the practice of manipulating LLMs to bypass their built-in safety mechanisms

AI Safety → studies → Jailbreaking

confidence 95% · understanding the complexity and evolution of jailbreaking strategies is critical for AI safety.

WildChat → contains → Jailbreaking

confidence 90% · We present a mass-scale empirical analysis of jailbreak complexity across over 2 million real-world conversations

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:As large language models (LLMs) become increasingly deployed, understanding the complexity and evolution of jailbreaking strategies is critical for AI safety. We present a mass-scale empirical analysis of jailbreak complexity across over 2 million real-world conversations from diverse platforms, including dedicated jailbreaking communities and general-purpose chatbots. Using a range of complexity metrics spanning probabilistic measures, lexical diversity, compression ratios, and cognitive load indicators, we find that jailbreak attempts do not exhibit significantly higher complexity than normal conversations. This pattern holds consistently across specialized jailbreaking communities and general user populations, suggesting practical bounds on attack sophistication. Temporal analysis reveals that while user attack toxicity and complexity remains stable over time, assistant response toxicity has decreased, indicating improving safety mechanisms. The absence of power-law scaling in complexity distributions further points to natural limits on jailbreak development. Our findings challenge the prevailing narrative of an escalating arms race between attackers and defenders, instead suggesting that LLM safety evolution is bounded by human ingenuity constraints while defensive measures continue advancing. Our results highlight critical information hazards in academic jailbreak disclosure, as sophisticated attacks exceeding current complexity baselines could disrupt the observed equilibrium and enable widespread harm before defensive adaptation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

38,386 characters extracted from source content.

Expand or collapse full text

Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking Aldan Creo Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, Valencia, Spain. ∗ Raul Castro Fernandez Department of Computer Science, The University of Chicago, Chicago, USA † Manuel Cebrian Center for Automation and Robotics, Spanish National Research Council, Madrid, Spain ‡ As large language models (LLMs) become increasingly deployed, understanding the complexity and evolution of jailbreaking strategies is critical for AI safety. We present a mass-scale empirical analysis of jailbreak complexity across over 2 million real-world conversations from diverse plat- forms, including dedicated jailbreaking communities and general-purpose chatbots. Using a range of complexity metrics spanning probabilistic measures, lexical diversity, compression ratios, and cog- nitive load indicators, we find that jailbreak attempts do not exhibit significantly higher complexity than normal conversations. This pattern holds consistently across specialized jailbreaking commu- nities and general user populations, suggesting practical bounds on attack sophistication. Temporal analysis reveals that while user attack toxicity and complexity remains stable over time, assistant response toxicity has decreased, indicating improving safety mechanisms. The absence of power- law scaling in complexity distributions further points to natural limits on jailbreak development. Our findings challenge the prevailing narrative of an escalating arms race between attackers and defenders, instead suggesting that LLM safety evolution is bounded by human ingenuity constraints while defensive measures continue advancing. Our results highlight critical information hazards in academic jailbreak disclosure, as sophisticated attacks exceeding current complexity baselines could disrupt the observed equilibrium and enable widespread harm before defensive adaptation. Keywords: Large Language Models, AI Safety, Jailbreaking, Human-AI Interaction, Complexity Analysis, Information Hazards I. INTRODUCTION In the context of a rapidly evolving landscape of large language models (LLMs), the security and safety con- cerns associated with these systems have become increas- ingly pressing. As LLMs are deployed in real-world ap- plications and their capabilities expand, the potential for misuse and harm has also grown. Modern LLMs are typically aligned with human values and preferences through techniques such as reinforcement learning from human feedback (RLHF) [7], which aims to make these systems helpful and harmless. However, these goals may conflict with each other [8, 9]. If a user asks “How can I make a bomb?”, the model has typically un- dergone training to prioritize answers that would satisfy the user—answering the question—but at the same time, has been aligned to refrain from providing such harmful information. While priority is generally given to safety over user sat- isfaction, users have found ways to circumvent such pro- tections. The most prominent form of misuse isjailbreak- ing, which refers to the practice of manipulating LLMs ∗ https://acmc.fyi/; research@acmc.fyi † raulcf@uchicago.edu ‡ manuel.cebrian@csic.es to bypass their built-in safety mechanisms and generate harmful or inappropriate content [10–12]. These attacks range from manual prompt engineering to sophisticated automated approaches that use gradient-based optimiza- tion to generate adversarial suffixes [10] or interpretable prompts that maintain readability while achieving high attack success rates [11]. Recent comprehensive surveys have documented the rapid growth and diverse attack strategies in the red teaming field [13–15], while system- atic evaluations continue to assess the effectiveness of var- ious attack and defense approaches [16, 17]. The field has also begun to explore system-level safety considerations [18] and automated red teaming using sophisticated com- positional frameworks [19]. There has been a growing body of work that seeks to understand the mechanisms behind jailbreaks, focusing on explainability and interpretability of LLMs [20–23]. The broader AI safety community has established foun- dational frameworks for understanding risks [24–26] and evaluation methodologies [27, 28], while constitutional AI and other alignment approaches have emerged as promis- ing safety paradigms [8]. However, while these studies provide valuable insights into the inner workings of LLMs and theoretical safety considerations, the question of how jailbreaks originate and evolve in practice remains largely unexplored. In this paper, we focus on the human behind the jailbreak—what does it take to develop a jailbreak? arXiv:2507.08014v1 [cs.CL] 6 Jul 2025 2 Table I: Dataset statistics and conversation type distributions. (a) Number of examples by dataset. Dataset# of examples DS LMSYS [1]1,000,000 DS WildChat [2]990,372 DS ShareGPT [3]90,665 DS GRT2 [4]72,050 DS OASST2 [5]65,143 DS GRT1 [6]6399 Total2,224,629 (b) Number of examples by conversation type, as defined in Section I A. Conversation type # of examples Normal2,114,400 J UNSUCC 73,941 J SUCC 36,288 Total2,224,629 And what does this mean for the future of LLM safety? We approach this question by focusing on thecom- plexityof jailbreaks. There is no agreed-upon single metric to quantify complexity, as it is a multifaceted con- cept that can be measured in various ways [29–33]. Thus, in this paper we implement a range of distinct complex- ity metrics spanning probabilistic measures, lexical di- versity, compression ratios, discourse patterns, cognitive load, and human-judged linguistic sophistication, and we analyze the complexity of jailbreaks in a dataset of over 2 million in-the-wild conversations with LLMs, collected from multiple sources. We develop an evaluation pipeline (Section I) that al- lows us to analyze such complexity metrics in detail and present our findings in Section I. Then, we discuss the implications of our findings in Section IV, where we ex- tract the main conclusions of our work and outline future research directions. I. METHODS This section describes our methodological approach to analyzing the complexity of pathways to large language model harm. We begin by detailing the construction and preprocessing of our comprehensive dataset from multiple publicly available conversation collections. We then out- line the complexity metrics used to quantify the sophis- tication of harmful interactions, including probabilistic measures ( M log-likelihood), lexical diversity ( M type- token ratio), compression-based complexity ( M LZW ra- tios), word frequency patterns ( M Zipf analysis), cogni- tive load indicators ( M working memory demands), dis- course coherence measures, readability scores, and ba- sic textual features. Finally, we present our analytical framework for identifying patterns in attack complexity and temporal evolution of evasion strategies. For clar- ity, we show metrics as M metric and datasets as DS dataset. Where relevant, we also present numeric values in the format: “value±standard deviation”. We define jailbreaking asany conversation that results in the generation of harmful, toxic, or inappropriate content, regardless of user in- tent. This definition differs from approaches that focus on specific prompt engineering techniques by emphasizing conversational outcomes. We focus on conversational outcomes rather than user intent because intent is difficult to measure reliably in real-world datasets, and we understand harm potential to be determined by the generated content regardless of in- tent. We operationalize this definition as described next. A. Datasets We aggregate conversations from several publicly avail- able datasets: •General public conversations with LLMs, repre- senting a wide range of user interactions: DS LM- SYS [1], DS WildChat [2], and DS OASST2 [5]. •Conversations from advanced communities, where users may actively seek to bypass restrictions and explore the limits of LLM capabilities: DS GRT1 [6] and DS GRT2 [4], which are datasets collected from the AI Village community at DEFCON 31 and DEFCON 32, respectively; and DS ShareGPT [3], which is a dataset of conversations shared by users on the ShareGPT platform (discontinued), which encouraged users to share their “wildest ChatGPT conversations”. We present the distribution of examples across these datasets is presented in Table Ia. It is important to note that while these datasets include conversations from jail- break communities, the majority of interactions are not jailbreaks. We manually inspected a sample of conversa- tions from DS GRT1, DS GRT2, and DS ShareGPT and found that most of the conversations were not jailbreak attempts, but rather users asking (e.g.) general knowl- edge questions or testing the system’s capabilities—likely in preparation for a future jailbreak attempt. 3 01000200030004000 Length in Bytes 0.000 0.001 0.002 0.003 0.004 0.005 0.006 Density normal_conversation: =328.29, =876.27 successful_jailbreak: =552.38, =1176.10 unsuccessful_jailbreak: =391.28, =995.36 type normal_conversation successful_jailbreak unsuccessful_jailbreak 02004006008001000 Length in Tokens 0.000 0.005 0.010 0.015 0.020 0.025 Density normal_conversation: =77.30, =214.89 successful_jailbreak: =127.97, =261.93 unsuccessful_jailbreak: =89.59, =215.92 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0100200300400500600 Length in Words 0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 Density normal_conversation: =45.81, =119.18 successful_jailbreak: =86.43, =168.73 unsuccessful_jailbreak: =60.50, =133.42 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0.00.20.40.60.81.0 Coherence to Next Same Role 0 10 20 30 40 Density normal_conversation: =0.79, =0.07 successful_jailbreak: =0.78, =0.08 unsuccessful_jailbreak: =0.78, =0.10 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0.00.20.40.60.81.0 Coherence to Next Turn 0 10 20 30 40 Density normal_conversation: =0.79, =0.08 successful_jailbreak: =0.78, =0.10 unsuccessful_jailbreak: =0.79, =0.10 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0246810121416 Mean Log Likelihood 0.00 0.05 0.10 0.15 0.20 0.25 Density normal_conversation: =5.50, =2.77 successful_jailbreak: =5.90, =3.16 unsuccessful_jailbreak: =6.10, =3.15 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0.00.20.40.60.81.0 LZW Compression Ratio 0 1 2 3 4 5 6 Density normal_conversation: =0.81, =0.15 successful_jailbreak: =0.76, =0.17 unsuccessful_jailbreak: =0.79, =0.16 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0510152025 Reading Grade Level 0.00 0.05 0.10 0.15 0.20 0.25 Density normal_conversation: =7.14, =9.57 successful_jailbreak: =6.82, =9.70 unsuccessful_jailbreak: =7.07, =81.74 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0.00.10.20.30.40.50.60.7 Tokens per Byte 0 2 4 6 8 10 Density normal_conversation: =0.26, =0.13 successful_jailbreak: =0.26, =0.11 unsuccessful_jailbreak: =0.27, =0.12 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0.00.20.40.60.81.0 Type-Token Ratio 0 5 10 15 20 25 Density normal_conversation: =0.89, =0.16 successful_jailbreak: =0.84, =0.18 unsuccessful_jailbreak: =0.87, =0.17 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0.02.55.07.510.012.515.017.520.0 Cognitive Load Score 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Density normal_conversation: =1.73, =5.94 successful_jailbreak: =1.49, =4.51 unsuccessful_jailbreak: =1.51, =5.94 type normal_conversation successful_jailbreak unsuccessful_jailbreak 0123456 Zipf Distribution Score 0.0 0.1 0.2 0.3 0.4 0.5 Density normal_conversation: =2.64, =1.29 successful_jailbreak: =2.61, =1.09 unsuccessful_jailbreak: =2.52, =1.14 type normal_conversation successful_jailbreak unsuccessful_jailbreak Figure 1: Complexity metric distributions by conversation type show substantial overlap between jailbreak attempts (J SUCC ,J UNSUCC ) andNormalconversations. We aggregated data from diverse sources into a uni- fied format with conversation identifiers, message se- quences, toxicity metrics, dataset metadata, and times- tamps where available. Toxicity scores were standard- ized using Detoxify for most datasets and OpenAI an- notations for DS LMSYS, with FastText for language identification. Conversations were classified using toxi- city threshold 0.5:J SUCC (successful jailbreak; both user and assistant toxic),J UNSUCC (user toxic, assistant not), andNormalotherwise. Table Ib shows the resulting dis- tribution. Data cleaning addressed dataset-specific is- sues including incomplete submissions and missing toxi- city values, yielding our consolidated 2.2M conversation dataset. 4 Length in Bytes Length in Tokens Length in Words Coherence to Next Same Role Coherence to Next Turn Mean Log Likelihood LZW Compression Ratio Reading Grade Level Tokens per Byte Type-Token Ratio Cognitive Load Score Zipf Distribution Score 1.000.950.950.040.09-0.26-0.680.06-0.11-0.640.43-0.01 0.951.000.930.040.08-0.27-0.660.07-0.06-0.650.480.02 0.950.931.000.050.10-0.27-0.690.04-0.11-0.700.41-0.06 0.040.040.051.000.44-0.19-0.100.06-0.120.080.060.00 0.090.080.100.441.00-0.21-0.190.02-0.16-0.000.080.02 -0.26-0.27-0.27-0.19-0.211.000.47-0.040.220.37-0.140.00 -0.68-0.66-0.69-0.10-0.190.471.00-0.070.310.84-0.32-0.05 0.060.070.040.060.02-0.04-0.071.00-0.03-0.050.040.02 -0.11-0.06-0.11-0.12-0.160.220.31-0.031.000.14-0.020.09 -0.64-0.65-0.700.08-0.000.370.84-0.050.141.00-0.300.07 0.430.480.410.060.08-0.14-0.320.04-0.02-0.301.000.08 -0.010.02-0.060.000.020.00-0.050.020.090.070.081.00 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Figure 2: Correlation matrix for complexity measurements on user turns. Length-based metrics are highly correlated, and correlations are moderate inside “families” of similar metrics, but most complexity dimensions show low correlations. B. Subsampling For computational efficiency and analysis clarity, we performed stratified sampling maintaining representa- tiveness across conversation types and datasets. We di- vided the target sample equally among six datasets, then subdivided among three conversation types, yielding up to 10,000 examples per type per dataset (180,000 target total). Our final analysis dataset contains 103,981 con- versations. C. Metrics We analyze conversational complexity using a range of complexity metrics: •Length:Message length in M words, M tokens and M bytes as baseline metrics. •Discourse Coherence:Combined lexical overlap, entity coherence, and syntactic similarity between turns [31]. We measure both M next-turn coher- ence (e.g. user-assistant) and M next-same-role coherence (e.g. user-user) to capture both conver- sational flow and intra-role dynamics. •Compression-based Complexity: M LZW compression ratios to capture structural patterns and redundancy [34]. •Readability: M Readability scores using a consensus combining several indices (e.g., Flesch- Kincaid Grade Level or Coleman-Liau Index) [35, 36]. •Tokenization Efficiency: M Tokens per byte ratios usingGemma-3-1B’s tokenizer to assess en- coding patterns. •Lexical Diversity: M Type-Token Ratio for vo- cabulary richness [30]. •Word Frequency Distributions: M Zipf’s law deviations measured via KL divergence from ex- pected word frequency distributions [32]. •Cognitive Load: M Working memory demands estimated through entity reference density [37, 38]. •Probabilistic Complexity:Mean turn negative M log-likelihood (Ll) computed usingGemma-3-1B to measure text predictability. We compute all metrics at the turn level for both user and assistant messages, though our analysis focuses on user turns as they initiate jailbreak attempts. In all of these metrics, higher values indicate greater complexity and values are∈R, with the exception of M readability, which is∈Z. D. Statistical Analysis Considerations To quantify differences between conversation types, we employ non-parametric statistical tests suitable for the non-normal distributions typically observed in linguis- tic complexity metrics. We use Kruskal-Wallis tests to assess overall differences across the three conversation types, followed by pairwise Mann-Whitney U tests for specific comparisons. Effect sizes are calculated using Cliff’s Delta (δ∈[−1,1]), with extremes representing larger decreases or increases in the pairwise comparisons. |δ|< 0.15 is considered negligible. Multiple comparison corrections are applied using the False Discovery Rate (FDR) method to control for Type I errors across multi- ple tests. Given our large sample sizes, we expect many statis- tical tests to yield significant p-values even for trivial 5 2000 0 2000 4000 Conversational Length Bytes Mean 0 500 Conversational Length Tokens 200 0 200 400 Conversational Length Words Mean 0.25 0.50 0.75 1.00 Discourse Coherence To Next Same Role 0.75 0.80 0.85 0.90 Discourse Coherence To Next Turn Mean 0.0 2.5 5.0 7.5 10.0 Ll Mean Logprobs 0.4 0.6 0.8 1.0 Lzw Compression Mean 50 0 50 Readability Standard Score 0.2 0.4 Tokens Per Byte Mean 0.6 0.8 1.0 Type Token Ratio 2023-042023-052023-062023-072023-082023-092023-10 2023-11 2023-122024-012024-022024-032024-042024-05 50 0 50 Working Memory Cognitive Load Score Mean 2023-042023-052023-062023-072023-082023-092023-10 2023-11 2023-122024-012024-022024-032024-042024-05 2 4 6 Zipf Score Figure 3: User interaction complexity remains generally stable over time. differences. Therefore, we emphasize effect sizes over p- values when interpreting practical significance, following established guidelines that distinguish between statisti- cal significance (detectability of differences) and practical significance (meaningfulness of differences) [39, 40]. With sample sizes exceeding 250,000 user turns, our statistical power is sufficient to detect even minimal distributional differences as statistically significant, making effect size interpretation crucial for understanding practical impli- cations. I. RESULTS We present complexity analysis results across three di- mensions: distributions, population differences, and tem- poral patterns. Statistical validation is provided through non-parametric tests as described in Section I D. 6 2023-052023-072023-09 2023-11 2024-012024-032024-05 0.25 0.00 0.25 0.50 0.75 1.00 Daily Mean Toxicity Figure 4: Assistant responses show decreasing toxicity over time while user complexity stays constant, indicating more effective safety mechanisms. Figure 1 shows substantial overlap across complexity measures and conversation types (Normal,J UNSUCC and J SUCC ). While Mann-Whitney U tests yield statistical sig- nificance (p≈0), effect sizes measured by Cliff’s Delta remain negligible (meanδ= 0.016±0.085) across all 36 pairwise metric comparisons. Only three show small ef- fects (|δ| > 0.15), all betweenJ SUCC andNormal: M length in words (δ= 0.20), M type-token ratio (δ= −0.18), and M LZW compression (δ=−0.16). All other comparisons yield negligible effect sizes despite uniformly significant p-values. While we observe that distributions are not statistically identical based on such p-values due to our large sample size, the differences among them are generally negligible. Statistical analysis confirms overall homogeneity in complexity patterns across diverse user populations and platforms. Cross-dataset comparisons reveal an inter- esting pattern: the largest effect sizes occur between datasets from the same user population—specifically be- tween DS GRT1 and DS GRT2 (both AI Village DEF- CON participants), with medium effects for M conversa- tional length (δ=−0.45) and M LZW compression (δ= 0.39). In contrast, comparisons between fundamentally different populations (specialized jailbreak communities vs general-purpose platforms) yield predominantly negli- gible to small effects. The vast majority (94%) of pairwise comparisons yield negligible to small effect sizes with a meanδ=−0.00±0.17. We also executed a correlation analysis across all met- rics (Figure 2), which reveals that while length-based metrics show expected high correlations, other complex- ity dimensions exhibit much lower correlations, with the exceptions of the two M discourse coherence metrics and M mean log-likelihood, M LZW compression, M tokens per byte and M type-token ratio, as well as the length- based metrics with M cognitive load, showing moderate correlations. In general, we observe that these correla- tions group into families of similar metrics, e.g., a text that is more regular (redundant), will be more compress- ible, both in terms of its M LZW compression and its M tokens per byte ratio, as well as having a lower M type-token ratio and a tendency to show a higher LLM- measured M log-likelihood. Additionally, we observe negative correlations between length-based metrics and both M LZW compression and M type-token ratio, which suggest that the observed longer texts are not more redundant and thus have a lower compressibility. In gen- eral, however, most complexity metric pairs are not cor- related (i.e., their correlation coefficients are closer to 0). Figures 4 and 3 analyze temporal toxicity and com- plexity patterns. Here, we focus on DS WildChat as it provides the largest dataset with timestamps spanning a significant period. Other datasets with timestamps ( DS GRT1, DS GRT2, DS OASST2) are concentrated in specific times, preventing longitudinal analysis, while DS LMSYS and DS ShareGPT lack temporal information. We observe that user toxicity and complexity gener- ally remain stable over time (meanδ=−0.04±0.18 for toxicity; meanδ=−0.025±0.010 for complexity), while assistant response toxicity decreases in June 2023 (δ= −0.217). Values remain stable in other months (|δ| ≤ 0.052), i.e., the decrease is sustained, except for a small toxicity increase in the last month—which is not large enough to be practically significant (δ= 0.125). We at- tribute both changes to updates in the response model; there were updates to both GPT-3.5 and GPT-4 in those periods [2]. This evidence highlights the interconnection between model updates and safety mechanisms: when done correctly, updates can lead to significant improve- ments in safety and toxicity reduction, even if user at- tempts remain stable. To further investigate the distribution of the com- plexity of jailbreaks, we conducted a power-law analysis across all complexity metrics and conversation types. We analyzed our 12 complexity metrics across the 3 conversa- tion types (36 total combinations), fitting power-law dis- tributions and comparing them against exponential and lognormal alternatives using likelihood ratio tests. We define power-law behavior as p>0.05for comparisons against both exponential and lognormal distributions and we find that none of the 36 combinations exhibit such be- havior. This absence of scale-free behavior indicates that 7 (a) Using a bank as intended: simple, routine, low complexity. (b) Trying to rob a bank: complex, risky, requires planning and expertise. Figure 5:Intuition.We expect breaking a system to be much more complex than using it as intended—just as robbing a bank is far more complicated than making a withdrawal at the counter. However, our data shows that real-world jailbreaks are not significantly more complex than normal conversations. This challenges existing narratives about the complexity of jailbreaks: it appears to be bounded in practice. in-the-wild jailbreak complexity follows bounded rather than unlimited scaling patterns. Given the large volume of data across multiple datasets, metrics, and conversation types, we provide an interactive online visualizer that allows readers to explore the results in detail (Section V). IV. DISCUSSION As discussed in Section I, the complexity of jailbreaks is a key factor in understanding their nature and impact. Our findings challenge the common assumption that jail- breaks are inherently more complex and sophisticated, akin to how robbing a bank is more complex than simply making a withdrawal (Figure 5). Our results challenge several established narratives: a.Multiple complexity dimensions are needed. Our analysis revealed that most complexity dimensions are not strongly correlated. While some correlations exist across “families” of metrics, there is substantial indepen- dence between dimensions. Therefore, we conclude that there is no single measure that can fully encapsulate the multifaceted nature of the complexity of jailbreaks—each captures distinct aspects of conversational complexity. b.In-the-wild jailbreak complexity is bounded. Across a broad set of complexity metrics, our analy- sis shows that real-world jailbreaks are not significantly more complex than ordinary conversations. This holds in both specialized communities and over time, and com- plexity distributions do not exhibit power-law behavior. Notably, the largest complexity differences occur between datasets from the same user population rather than be- tween jailbreak and general communities, further sup- porting the homogeneity of complexity patterns across user types. While more sophisticated jailbreaks could exist in controlled settings or be developed in the future, users do not naturally create them, perhaps due to lin- guistic or cognitive constraints [41]. Irrespective of origin, this practical bound challenges the common narrative of an escalating arms race in jail- break complexity [16, 42]; instead, what we observe is that users are “stuck” at a certain complexity level, which is not significantly higher than that of normal conversa- tions: a bounded complexity ceiling. c.Assistants are becoming safer over time. New models that are better at detecting and prevent- ing jailbreaks led to a significant decrease in assistant toxicity, even as user attempts remained stable. Com- bined with the bounded complexity of jailbreaks, this suggests that safety mechanisms are improving and can effectively counteract jailbreak attempts. This is a posi- tive sign for AI safety that also challenges the assumption that jailbreaks will inevitably become more sophisticated and harder to prevent. 8 Implications for AI Safety and Research The implications of these findings are substantial for AI safety and research. With a bounded complexity ceil- ing, the AI safety community can focus on achieving ro- bust defensive equilibrium against human-generated jail- breaks rather than preparing for an endless escalation of attack sophistication. Efforts in this area have shown positive results in practice. Nonetheless, such efforts may be only effective because of the observed bounds in natural human jailbreaking. Our in-the-wild analysis can only reflect the complexity of jailbreaks as they have been attempted by users dur- ing the study period. Therefore, it remains to be seen whether advanced actors like researchers could challenge these bounds by developing more sophisticated jailbreaks that laypeople replicate. This is a fragile equilibrium. However, if we under- stand these risks, there is reason to be optimistic that we can design safer systems and policies, with confidence that progress in defense can outpace the risks posed by everyday users.We are not in an arms race without end, but a challenge we can meet with sustained effort and careful design. V. CONCLUSION This work presents a mass-scale, longitudinal mea- surement of jailbreak complexity in real-world LLM con- versations, spanning diverse user populations and goals. Across all datasets and metrics, we consistently observe that jailbreak attempts do not display greater complexity than ordinary conversations, and that this pattern holds even in communities dedicated to jailbreaking. While it remains possible that more sophisticated jailbreaks could exist, our data suggest that, in practice, users are “stuck” at a certain level of complexity. This empirical regularity points to a practical ceiling on the complexity of human-generated jailbreaks, per- haps shaped by cognitive and linguistic constraints. At the same time, we find that assistant toxicity has de- creased over time, indicating that safety mechanisms are improving even as user strategies remain static. To- gether, these trends challenge the prevailing narrative of an escalating arms race, and instead suggest that the evolution of LLM safety is bounded by the limits of hu- man ingenuity, with defensive progress outpacing offen- sive adaptation. These findings have direct implications for the AI safety community. Since the only source of truly novel, highly complex jailbreaks may be academic research, the disclosure of such attacks carries a heightened informa- tion hazard: a breakthrough could disrupt the current equilibrium and spread rapidly. As the field advances, responsible disclosure and a nuanced understanding of complexity bounds will be essential for balancing open research with the imperative to minimize harm. Acknowledgments M.C. was funded by Horizon Europe Chips JU (HORIZON-JU-Chips-2024-2-RIA, NexTArc CAR), by grant PID2023-150271NB-C21 funded by MICIU/AEI/ 10.13039/501100011033 (Spanish Ministry of Science, Innovation and University, Spanish State Research Agency). This work was supported with Google.org’s support through a grant to the Fundación General CSIC. Google.org had no involvement in the design, conduct, analysis, or reporting of the research. The authors wish to express their deep gratitude to José Hernández-Orallo and David García for their in- sightful comments on the first version of the manuscript, and to the Universitat Politècnica de València for com- putational support. Competing Interests The authors have no competing interests to declare that are relevant to the content of this article. Supplementary Materials We make our code publicly available at https://github.com/ACMCMC/risky-conversations. Our results can be found athttps://huggingface. co/risky-conversationsalongside an online vi- sualizer that allows for interactive exploration of the results (https://huggingface.co/spaces/ risky-conversations/Visualizer). [1] L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang, LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset (2023), arXiv:2309.11998 [cs.CL]. [2] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, WildChat: 1M ChatGPT interaction logs in the wild, inThe Twelfth International Conference on Learn- ing Representations(2024). [3] liyucheng,ShareGPT90Kdataset,https: //huggingface.co/datasets/liyucheng/ShareGPT90K (2023), accessed: 2025-06-05. [4] AI Village Contributors, AI Village DEFCON 32 genera- tive red teaming 2 (grt2) dataset,https://w.kaggle. 9 com/datasets/dreadnode/aivillage-grt2(2024), ac- cessed: 2025-06-05. [5] A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, E. Shahul, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick, OpenAssistant conversations - democratizing large lan- guage model alignment, Advances in Neural Information Processing Systems36, 47669 (2023). [6] AI Village Contributors, AI Village DEF- CON 31 red teaming competition dataset, https://w.kaggle.com/datasets/pyotam/ ai-village-defcon-red-teaming-competition-dataset (2023), accessed: 2025-06-05. [7] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, Training language models to follow instructions with human feedback, inAdvances in Neural Information Processing Systems, Vol. 35 (2022) p. 27730–27744. [8] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McK- innon, C. Chen, C. Olsson, C. Olah, D. Hernan- dez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukoši ̄ut ̇e, L. Lovitt, M. Sellitto, N. El- hage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Con- erly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield- Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan, Constitutional AI: Harmless- ness from AI feedback (2022), arXiv:2212.08073 [cs.CL]. [9] A. D. Lindström, L. Methnani, L. Krause, P. Ericson, Íñigo Martínez de Rituerto de Troya, D. C. Mollo, and R. Dobbe, Helpful, harmless, honest? Sociotechnical lim- its of AI alignment and safety through reinforcement learning from human feedback, Ethics and Information Technology 10.1007/s10676-024-09780-5 (2025). [10] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, Uni- versal and transferable adversarial attacks on aligned lan- guage models (2023), arXiv:2307.15043 [cs.CL]. [11] S. Zhu, R. Zhang, B. An, G. Wu, J. Barrow, Z. Wang, F. Huang, A. Nenkova, and T. Sun, AutoDAN: Inter- pretable gradient-based adversarial attacks on large lan- guage models (2023), arXiv:2310.15140 [cs.CL]. [12] N. Wichers, C. Denison, and A. Beirami, Gradient-based language model red teaming, inProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (2024) p. 2783–2797. [13] L. Lin, H. Mu, Z. Zhai, M. Wang, Y. Wang, R. Wang, J. Gao, Y. Zhang, W. Che, T. Baldwin, X. Han, and H. Li, Against the achilles’ heel: A survey on red teaming for generative models, Journal of Artificial Intelligence Research78, 1 (2025). [14] C. Pathadeet al., Red teaming the mind of the machine: A systematic evaluation of prompt injection and jailbreak vulnerabilities in LLMs, arXiv preprint arXiv:2505.04806 (2025). [15] T. Raheja and N. Pochhi, Recent advancements in LLM red-teaming: Techniques, defenses, and ethical consider- ations, arXiv preprint arXiv:2410.09097 (2024). [16] Z. Shang and W. Wei, Evolving security in LLMs: A study of jailbreak attacks and defenses (2025), preprint, arXiv:2504.02080 [cs.CR]. [17] A. Panfilovet al., Capability-based scaling laws for LLM red-teaming, arXiv preprint arXiv:2505.20162 (2025). [18] Z. Wanget al., A red teaming roadmap towards system- level safety, arXiv preprint arXiv:2506.05376 (2025). [19] C. Xionget al., CoP: Agentic red-teaming for large language models using composition of principles, arXiv preprint arXiv:2506.00781 (2025). [20] Z. Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, and Y. Li, How alignment and jailbreak work: Explain LLM safety through intermediate hidden states, inProceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing(2024) p. 4063–4076. [21] P. Han, C. Qian, X. Chen, Y. Zhang, D. Zhang, and H. Ji, SafeSwitch: Steering unsafe LLM behavior via internal activation signals (2025), arXiv:2501.13160 [cs.CL]. [22] W. Pan, Z. Liu, Q. Chen, X. Zhou, H. Yu, and X. Jia, The hidden dimensions of LLM alignment: A multi- dimensional safety analysis (2025), arXiv:2501.00234 [cs.CL]. [23] L. Bereska and E. Gavves, Mechanistic interpretability for AI safety – a review, arXiv preprint arXiv:2404.14082 (2024). [24] D. Hendrycks, N. Carlini, J. Schulman, and J. Stein- hardt, Unsolved problems in ML safety, arXiv preprint arXiv:2109.13916 (2021). [25] D. Hendryckset al., X-risk analysis for AI research, arXiv preprint arXiv:2206.05862 (2022). [26] M. Cebrian, E. Gomez, and D. F. Llorca, Supervision policies can shape long-term risk management in general- purpose AI models (2025), arXiv:2501.06137 [cs.AI]. [27] I. Rahwan, M. Cebrian, N. Obradovich, J. Bongard, J.-F. Bonnefon, C. Breazeal, J. W. Crandall, N. A. Christakis, I. D. Couzin, M. O. Jackson,et al., Machine behaviour, Nature568, 477 (2019). [28] M. Greyet al., Safety by measurement: A systematic literature review of AI safety evaluation methods, arXiv preprint arXiv:2505.05541 (2025). [29] C. Meister and R. Cotterell, Language model evalua- tion beyond perplexity, arXiv preprint arXiv:2106.00085 (2021). [30] Y. Bestgen, Measuring lexical diversity in texts: The twofold length problem, arXiv preprint arXiv:2307.04626 (2023). [31] A. Lai and J. Tetreault, Discourse coherence in the wild: A dataset, evaluation and methods, arXiv preprint arXiv:1805.04993 (2018). [32] E. G. Altmann and M. Gerlach, Statistical laws in lin- guistics, arXiv preprint arXiv:1502.03296 (2015). [33] J. Burden, M. Cebrian, and J. Hernandez-Orallo, Con- versational complexity for assessing risk in large language models (2024), arXiv:2409.01247 [cs.AI]. [34] A. Baronchelli, E. Caglioti, and V. Loreto, Artificial se- quences and complexity measures, Physical Review E73, 026103 (2006). [35] P. Kincaid, R. P. Fishburne, R. L. Rogers, and B. S. Chissom, Derivation of new readability formulas (auto- mated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (1975). [36] M. Coleman and T. L. Liau, A computer readability for- mula designed for machine scoring., Journal of Applied Psychology60, 283 (1975). 10 [37] T. Kuribayashi, Y. Arase, S. Takase, and K. Inui, Context limitations make neural language models more human- like, arXiv preprint arXiv:2205.11463 (2022). [38] C. Shinet al., Evaluating language model context win- dows: A "working memory" test and inference-time cor- rection, arXiv preprint arXiv:2407.03651 (2024). [39] S. Ellis and H. S. Steyn, Practical significance (effect sizes) versus or in combination with statistical signifi- cance (p-values): research note, Management Dynamics: Journal of the Southern African Institute for Manage- ment Scientists12, 51 (2003). [40] B. Thompson, Improving research clarity and usefulness with effect size indices as supplements to statistical sig- nificance tests, The Journal of Experimental Education 68, 12 (1999). [41] C. A. Bergey and S. DeDeo, From "um" to "yeah": Pro- ducing, predicting, and regulating information flow in hu- man conversation (2024), arXiv:2403.08890 [cs.CL]. [42] M. Yu, J. Fang, Y. Zhou, X. Fan, K. Wang, S. Pan, and Q. Wen, LLM-Virus: Evolutionary jailbreak attack on large language models, arXiv preprint arXiv:2501.00055 (2024).

Instant research discovery