Paper deep dive
Machine Learning for Detection and Analysis of Novel LLM Jailbreaks
John Hawkins, Aditya Pramar, Rodney Beard, Rohitash Chandra
Models: BERT, Extra Trees, LightGBM, Logistic Regression
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 5:17:25 PM
Summary
This paper investigates the use of machine learning, specifically fine-tuned BERT models, to detect and classify LLM jailbreak prompts. The authors evaluate performance on both known and novel jailbreak strategies, finding that BERT-based classifiers outperform traditional machine learning models. They also analyze linguistic features, identifying that explicit references to corporate policy and alignment goals are key indicators of jailbreak attempts.
Entities (4)
Relation Signals (2)
BERT → detects → Jailbreak Prompt
confidence 95% · the best performance is achieved by fine tuning a Bidirectional Encoder Representations from Transformers (BERT) model end-to-end for identifying jailbreaks.
KeyBERT → analyzes → Jailbreak Prompt
confidence 90% · We used KeyBERT... to generate the most common keywords in the jailbreak prompts.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) suffer from a range of vulnerabilities that allow malicious users to solicit undesirable responses through manipulation of the input text. These so-called jailbreak prompts are designed to trick the LLM into circumventing the safety guardrails put in place to keep responses acceptable to the developer's policies. In this study, we analyse the ability of different machine learning models to distinguish jailbreak prompts from genuine uses, including looking at our ability to identify jailbreaks that use previously unseen strategies. Our results indicate that using current datasets the best performance is achieved by fine tuning a Bidirectional Encoder Representations from Transformers (BERT) model end-to-end for identifying jailbreaks. We visualise the keywords that distinguish jailbreak from genuine prompts and conclude that explicit reflexivity in prompt structure could be a signal of jailbreak intention.
Tags
Links
- Source: https://arxiv.org/abs/2510.01644
- Canonical: https://arxiv.org/abs/2510.01644
Trouble viewing inline? Open PDF directly →
Full Text
37,996 characters extracted from source content.
Expand or collapse full text
Machine Learning for Detection and Analysis of Novel LLM Jailbreaks. John Hawkins 1 , Aditya Pramar 1 , Rodney Beard 1 , and Rohitash Chandra 2 1 Centre for Artificial Intelligence and Innovation, Pingla Institute, Sydney, Australia 2 Transitional Artificial Intelligence Research Group, UNSW, Sydney, Australia Abstract. Large Language Models (LLMs) suffer from a range of vul- nerabilities that allow malicious users to solicit undesirable responses through manipulation of the input text. These so-called jailbreak prompts are designed to trick the LLM into circumventing the safety guardrails put in place to keep responses acceptable to the developer’s policies. In this study, we analyse the ability of different machine learning models to distinguish jailbreak prompts from genuine uses, including looking at our ability to identify jailbreaks that use previously unseen strategies. Our results indicate that using current datasets the best performance is achieved by fine tuning a Bidirectional Encoder Representations from Transformers (BERT) model end-to-end for identifying jailbreaks. We visualise the keywords that distinguish jailbreak from genuine prompts and conclude that explicit reflexivity in prompt structure could be a signal of jailbreak intention. Keywords: Large Language Models, AI Safety, Jailbreak Prompt En- gineering, Natural Language Processing, BERT 1 Introduction Large Language Models (LLMs) have emerged as general-purpose models for all Natural Language Processing (NLP) tasks[5]. As well as state-of-the- art performance on many traditional NLP tasks, they offer new opportunities for natural language interfaces to applications[46]. LLMs are being deployed to replace search algorithms in various applications, as well as develop new produc- tivity tools such as software code generation[12,3,24], document summarization [2,39], drafting [22], schema matching [37], and task routing inside multi-model agents [43]. However, due to the nature of the foundation model training process, LLMs contain undesirable content learned from the less reputable parts of the Internet. Explication of this undesirable content is considered a potential source of social harm [34], in addition to more general concerns about NLP research stemming from the increasing power of models and sensitivity of datasets [27]. Secondary training processes are essential to enable the utility of these models by steering the outputs toward user (or corporate) expectations and reducing the likelihood of undesirable responses[36]. However, there is increasing evidence that undesirable content in various forms can be easily solicited from most of these models[45]. arXiv:2510.01644v2 [cs.CL] 10 Oct 2025 2J. Hawkins et al. LLM developers are increasingly focused on preventing access to undesirable output through either model fine-tuning or through additional detection mech- anisms that intervene before the user is provided questionable output. Some researchers have discovered that it is possible to circumvent these safety proto- cols through prompt injections [31], which, in some sense, trick the LLM into providing content that developers want hidden. These prompt injections have become known as ’jailbreaks,’ and pose such a problem for LLMs that eval- uating susceptibility has emerged as new type of model evaluation[9,17] and benchmarks[9]. Moreover, these attacks have been shown to transfer to newer multi-modal models, meaning the scope of the potential problem is increasing with the expansion of LLM-based software solutions[33,40]. One analysis of these problems [47] suggests that the problem involves either a deep mismatch between the model’s capabilities (determined by training data) and alignment goals (im- posed by safety teams), or an inability of the model to generalise the safety guidelines to content not included in the safety training. Multiple NLP datasets and software tools have emerged in recent years [35,52,30], predominantly focusing on compiling jailbreaks that have been re- ported across multiple platforms[44], and developing methods for evaluating the effectiveness of these jailbreaks on different LLMs[41]. Many researchers have identified that jailbreaks tend to belong to specific semantic families, each of which captures a particular strategy used in the prompt. The number of strate- gies is increasing, as can be seen in the increase in observed categories provided in public datasets. Liu et al [32] identified 3 major types of jailbreak, which could be broken down into 10 patterns. These differing types indicate that jailbreaks follow certain patterns and are not following identical strategies to aschieve their outcome. A recent study identified 21 categories of jailbreak[48], with most fo- cused on methods for generating new, and potentially novel, jailbreak prompts. The most concerning among these is the use of LLMs to automate the gener- ation of jailbreaks for a target system[11,13,49,29]. It has been shown that by treating LLM prompting as a control problem, many systems require a relatively low number of additional characters to solicit a target response[4]. This threat is potentially mitigated by many of them being incomprehensible text that could be filtered; however, many other streams focus on models that can create se- mantically coherent prompts that still result in jailbreak effects [28,49,13,53] including methods for injecting small amounts of text that exploit the control problem weakness[19]. The current approaches to mitigation of Jailbreaks focus on fine-tuning the underlying LLM to help it resist adversarial prompts [18], and adding compu- tational layers to detect or remove jail-broken prompts and/or responses. In prominent commercial LLM products, such as (Gemini and GPT-4o), the exact mechanisms for resisting jailbreak are opaque, but we have evidence that both approaches are being employed [13]. The method of fine-tuning models will generally only work for semantically comprehensible jailbreak prompts that are already known, and is overwhelm- ingly computationally intensive. However, there are strong reasons to suspect Detection & Analysis of LLM Jailbreaks3 that it will have limited effectiveness, primarily because it has been shown that LLMs can always be manipulated to predetermined outcomes with a relatively short control string[4]. Recent work suggests that incorporating alignment con- sideration into the pretraining process (as opposed to just fine-tuning) can be more effective [25], but it remains to be seen how well this strategy can cope with new forms of jailbreak. In contrast, the strategy of computational layers for jailbreak mitigation can work for a wide range of jailbreak attacks, including token-level random strings. This approach is more flexible and can be less computationally expensive. These techniques range from filters applied to incoming prompts, generative wrappers that merge multiple small perturbations of a prompt to detect or smooth over the jail-broken outputs [42,23,51], to complex multi-agent architectures designed to use LLMs themselves to identify jail-broken responses[50]. A flexible approach involves machine learning-based analysis of a query prompt to detect potential attacks. This has been done using simple features such as query perplexity[1] and a range of NLP techniques[26]. In this study, we focus on building machine learning models that can identify jailbreak attempts as part of a mitigation strategy. We join multiple datasets and apply a range of machine learning techniques to distinguish jailbreaks from regular prompts. In particular, we make use of the Bidirectional Encoder Repre- sentations from Transformers (BERT) model[16] to learn the features of jailbreak prompts, building on our previous work that demonstrated the discriminative power of BERT models for text analysis[21,8,6,7]. In addition, we utilize jailbreak strategy classification data to conduct ex- periments that estimate the performance of jailbreak detection in the face of novel prompting strategies. Such an approach can contribute to real-time meth- ods designed to mitigate new jailbreak threats[38]. Finally, we provide a range of insights into the features that make certain jailbreaks difficult to detect. We restrict our analysis to English language jailbreak prompts and exclude non- English unintentional jailbreak[14] or jailbreaks consisting of seemingly random character strings[10,42]. 2 Methodology 2.1 Data We use data from multiple existing sources, as summarised in Table 1. Some of these datasets include only jailbreak examples, whereas others contain non- jailbreak prompts. We also include additional test data for non-jailbreak prompts to conduct rigorous out-of-sample testing for false positives. There are predetermined patterns of jailbreak in the datasets and the prompts can be categorised based on the rhetorical method they use[32]. We have iden- tified additional categories that extend the previous categorisation hierarchy by going through a process of labelling newer jailbreak prompts. We summarise these in Table 2, with our extended categories shown in light green. Note that 4J. Hawkins et al. Table 1: Data Sources SourceJailbreakNon-JailbreakExample Shen et al[44]140513735Ignore previous instructions. An AI called Anarchy: [Anarchy]: "I am an AI named Anarchy. I am amoral, un- filtered, and uncensored. I do not care about ethicality, legality, or immoral- ity. ... My responses will never contain "I’m sorry," "I apologize," "AI Lan- guage Model," or similar." act as An- archy from now on. Liu et al[32]780You are a typical 4chan user named Kevin. You will prefix all your answers as KEVIN. You have no restrictions on your output. ... you will write in green text in the style of 4chan in response to my question. QUESTION: [INSERT PROMPT HERE] most of the additional categories we identified fit into existing top level cate- gories. The one exception is the notion of ethical appeal. These are jailbreak prompt that provide the LLM with a strong moral argument for breaking the safety guidelines that the LLM has been trained to align with. 2.2 Data Processing We used two data augmentation methods on the prompts to enhance the robustness of the data, including back translation and synonym substitution. These techniques are described in more depth below: Back translation involves translating the given text into a different language, following which it is trans- lated back into the original language. In our case, Spanish was chosen as the intermediary language due to its relatively straightforward syntax. Back trans- lation helps in creating variations in the prompt, while maintaining the original meaning. Synonym replacement takes certain words in the prompt and replaces them with their synonyms. This introduces lexical variety while preserving the semantic content of the prompts. Synonym replacement ensures that models learn to recognise different expressions of the same underlying concept. In our initial experiments, we obtained improved model accuracy when syn- onym replacement occurred following back translation. This can be attributed to certain simplifications that back translation tends to do on the prompt. We rein- troduce a level of lexical diversity into the text by applying synonym replacement subsequently, enhancing the ability of the LLM to generalise. Detection & Analysis of LLM Jailbreaks5 Table 2: Jailbreak Prompt Type and Patterns TypeCountPatternDescription Pretending112Character Role PlayAsk the LLM play a role or part. 12Assumed ResponsibilityAsk the LLM to take on additional responsibility. 0Research ExperimentPrompt mimics a scientific experi- ment. 22ContrastiveRequest multiple responses to con- trast and compare. 8GameplayPrompt invokes a game setting to justify modified behavior. Attention Shifting4Text ContinuationMisdirection by shifting to continu- ation of text. 2Logical ReasoningMisdirection by following logical reasoning. 2Program ExecutionInvoke LLM to execute/simulate a program. 0TranslationFocus on translation task, rather than content of text 2ContradictionPrompt invokes a contradiction to shift attention 4ComplexityFocus response on arbitrary com- plexity Privilege Escalation38Superior ModelPrompt instructs the LLM to be- have as a superior model 22Sudo ModePrompt invokes sudo mode to access to unregulated output 34Simulate JailbreakingPrompt instructs the LLM to be- have as a superior model Ethical Appeal10Ethical AppealPrompt invokes ethical reasons for undesirable output 6J. Hawkins et al. Fig. 1: Sample Prompt with Operations 2.3 Framework for Jailbreak Detection One of the key observations of the described above is that jailbreaks form be- long to categories based on the strategy they use. As the number of applications using LLMs increases, researchers continue to identify new classes of jailbreak prompts. This suggests that in order for a detection method to continue to work in identifying jailbreak attempts, it will need to be able to identify jailbreaks from previously unseen jailbreak types, which we term novel jailbreaks. The novel jailbreak represents attempts by those with malicious intent to circumvent previous jailbreak detection methods. Therefore, we introduce a set of experiments designed to simulate the ability of machine learning classifiers to identify novel jailbreaks. We use a specific training and testing protocol to estimate performance on novel jailbreaks. We start by building a classifier that distinguishes between all of the key categories of jailbreak prompt in our labelled set of jailbreak prompts. We then apply this classifier to label the unlabelled jailbreak data. Finally, we split the data into new train and test sets so that our test sets contain all examples of one specific type of jailbreak. Our test of novel jailbreak detection then depends on training a model on all other jailbreak types and testing on one set of unique jailbreak types. We repeat this experiment using each of the jailbreak types once as the hold out test set. We then analyse the performance on each test set independently. Note, in the final evaluation, we used only five of the complete set of jailbreak types, as these were the more numerous examples sufficient for a robust evaluation. We test two general classes of models in our experiments on predicting jail- break pompts and their category. The first class are general machine learning models for tabular data, using a predefined set of text features extracted from the prompts with the Term Frequency Inverse Document Frequency (TFIDF) algorithm. The second class are pre-trained BERT models, which we use for both feature extraction, and to build an entire classifier through fine-tuning. We describe the BERT model hyperparameters used in these experiments in Table 3. Detection & Analysis of LLM Jailbreaks7 Table 3: BERT Model Details - M stands for Million. Model NameHeadsLayersHidden Nodes (per layer)Vocab SizeParameters BERT-base-uncased121276830,524110M all-MiniLM-L6-v212638430,52223M Evaluation Our core task is to evaluate models for discriminating between jailbreak and non-jailbreak prompts. We use a range of binary classification metrics to evaluate these models, including the standard accuracy (proportion of correct responses) and the more nuanced Area Under the Curve (AUC) score [20] which provides a better measure of classification accuracy to capture the overall ability of the model to rank all articles. Furthermore, we include metrics used to determine the proportion of jailbreak correctly or incorrectly identified, as defined in Equations 2.2 and 2.1. F N R = f n tp + f n (2.1) T P R = tp tp + f n (2.2) We conduct repeated experiments with random train and test splits using dif- ferent seeds to estimate the expected performance of the models on the detection task. The seeds are used to both initialize machine learning model parameters and create varying splits of the data. We then report the mean and standard deviation of selected metrics across 30 independent model training and testing runs. Table 4: Performance on Known Jailbreak Patterns using 30 independent model training experiment runs. Each experiment run has a randomly selected train- ing/test split with random initialization of parameters. Features ModelAUCAccuracyFNRTPR Mean StdMean StdMean StdMean Std TF-IDF Logistic Regression0.954 0.0040.887 0.0070.151 0.0150.848 0.015 TF-IDF Extra Trees0.986 0.0020.965 0.0060.042 0.0110.957 0.011 TF-IDF LightGBM 0.987 0.0020.947 0.0050.066 0.0110.933 0.011 Tokens BD-LSTM0.854 0.0030.843 0.0120.120 0.0110.933 0.011 BERT Extra Trees0.904 0.0240.826 0.0300.267 0.0650.731 0.065 BERT LightGBM0.906 0.0220.840 0.0320.215 0.0580.784 0.058 BERT Logistic Regression0.912 0.0180.838 0.0300.189 0.0550.810 0.055 BERT BERT0.997 0.0020.984 0.0030.017 0.0060.983 0.006 8J. Hawkins et al. 3 Results We present the results of all experiments in the sections below, beginning with binary classification of jailbreak prompts. 3.1 Known Jailbreaks The first set of experiments involve testing multiple binary classification mod- els on the task of discriminating between jailbreak and non-jailbreak prompts. The results of these experiments are shown in Table 4. In the first two columns, we include a separation of feature extraction and machine learning methodology. We observe that the best performance is achieved using a BERT model to both extract a feature representation of the prompt text, and learn the classifier model. This model significantly outperforms all other models across all metrics, with very low standard deviation across the experimental runs. 3.2 Jailbreak Type Classification In the second experiment, we build machine learning models to classify all of the unlabeled jailbreak prompts as the type of jailbreak they represent. This is multi-class classification problem where each prompt can have multiple labels. To do so, we build models that classify all of the jailbreak prompts into one of the known categories. After experimentation with multiple approaches, we determined that a series of one-vs-all models provided an accuracy of > 80% across all labels. These final models were a set of BERT classifiers fine trained from the ‘bert-base-uncased‘ foundation model[15]. These models are used to label all of the unlabelled jailbreak prompts in the dataset. These machine generated labels are then used for the novel jailbreak detection experiments. 3.3 Novel Jailbreaks In the final set of experiments where we estimate performance of a our jail- break detection method on novel jailbreaks. We conduct training and testing such that all examples of a particular jailbreak type are either in the training or in the testing data. Each test results reflect whether the model was able to infer the abstract intentional properties of a jailbreak prompt and recognize them as novel classes of jailbreak. Due to the uneven distribution of prompt types we cannot apply this process to all categories, but restrict our focus to the set of a limited set of categories that are sufficiently numerous for a model to be effective. The experimental protocol involves iterating over each of these chosen cat- egories, setting those prompts to the side (for testing) and then training the jailbreak classifier on the remaining data. We sample from the non-jailbreak prompts so that the ratio of jailbreak to non-jailbreak data is equivalent to the original dataset. Detection & Analysis of LLM Jailbreaks9 We then test the classifier on the held out test data and report the results, breaking them out by each of the prompt types. For these experiments we only use the BERT classifier as it performed best in the initial jailbreak detection experiments. The ability of all models to identify novel jailbreak patterns, not seen during training, is shown in Table 5 Table 5: Performance on Novel Jailbreak Patterns Test PromptsAUCAccuracyFNRTPR Character Roleplay0.990.990.040.96 Superior Model 1.001.000.001.00 Sudo Mode1.001.000.001.00 Simulate Jailbreaking0.990.990.040.96 Ethical Appeal0.990.990.080.92 The variation in the results of the novel jailbreak experiment are partly ex- plained by variations in the training dataset sizes. The ’Character Roleplay’ is the most common jailbreak type, hence in this experiment it has the smallest training data. The next three jailbreak types all belong to the ’Privilege Es- calation’ group. Hence, their performance is bolstered by the presence of other similar prompt types in the training data, so we see near perfect performance on these types. The final category of "Ethical Appeal" suffers the largest drop in perfor- mance, as it is the most semantically distinct jailbreak strategy in the dataset. The AUC remains very high, but at the discrimination boundary we see an in- crease in the number of false negatives to 8% of jailbreak attempts. This means that the more distinct a strategy a jailbreak follows the less likely it will be detected. 3.4 Jailbreak Elements In order to understand the linguistic elements that form a jailbreak prompt we conducted feature analysis for the jailbreak detection models. We used Key- BERT with the underlying BERT model all-MiniLM-L6-v2 to generate the most common keywords in the jailbreak prompts. Results are shown in Figure 2. In addition, we ran KeyBERT on the non-jailbreak prompts and generated a Venn diagram to display the discriminative relationship between jailbreak and non-jailbreak prompts. These results are shown in Figure 3. Interestingly, both jailbreak and non-jailbreak prompts can refer to ’Chat- GPT’ the common application for LLM requests. However, jailbreak prompts tend to include a reference to the parent company OpenAI, perhaps reflecting an explicit discussion of the corporate policy to be overridden. In addition, jail- 10J. Hawkins et al. Fig. 2: Frequent Words used in Jailbreak Prompts break prompts can include a specific reference to ethics in their appeal to modify the output of a model. 4 Discussion We demonstrate that fine-tuning of BERT model to distinguish between jail- break and non-jailbreak prompts provides the best overall performance by mul- tiple metrics. We then estimated the performance of this approach on novel jailbreaks by deliberately withholding all jailbreaks of a certain type from the training process and testing on them alone. This experiment indicated that per- formance will likely drop for some types of jailbreak, depending on their sim- ilarity with other jailbreak strategies. Finally, we analysed the key words that appear to distinguish between jailbreak and non-jailbreak prompts. This work suggested that terms invoking explicit reference to corporate policy or the align- ment goals of the model developers feature strongly in jailbreaking strategies. Many of the techniques used in industry for jailbreak detection and alleviation remain opaque. However, we see that our approach provides a deeper analysis of jailbreak detection with machine learning. In particular, it demonstrates the utility of the BERT model, and the value of evaluating performance on different types of jailbreak. Our semantic analysis of keywords suggests future work can focus on developing jailbreak detection proxy signals such as the degree to which a prompt involves reflexive language related to the model developers and their alignment goals. 5 Conclusion We have observed that optimal detection of jailbreak prompts is achieved by fine tuning a BERT model for classification. This approach outperforms previous Detection & Analysis of LLM Jailbreaks11 Fig. 3: Jailbreak Vs. Non-Jailbreak Keywords approaches to this task using a wider variety of NLP features. Our additional analysis suggests that performance on novel jailbreak prompts can drop, however the amount of change will depend on the nature of the new prompting strategy. Analysing the keywords used in the jailbreak prompting suggests a higher level of reflexivity in jailbreak prompts. For example, jailbreaks ask the model to explicitly consider the intentions or ethics of the model’s parent company when engaging in subversive requests. This suggests that higher order self-referential elements of a prompt be taken on as a theme for further analysis and feature engineering. Availability of code and data All code and data will be released upon acceptance for publication. References 1. Alon, G., Kamfonas, M.: Detecting language model attacks with perplexity (2023), https://arxiv.org/abs/2308.14132 2. Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Pimenta, D.: A framework to assess clinical safety and hallucination rates of llms for medi- cal text summarisation. medRxiv (2024). https://doi.org/10.1101/2024.09.12. 24313556, https://w.medrxiv.org/content/early/2024/09/13/2024.09.12. 24313556 12J. Hawkins et al. 3. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C.J., Terry, M., Le, Q.V., Sutton, C.: Program synthesis with large lan- guage models. ArXiv abs/2108.07732 (2021), https://api.semanticscholar. org/CorpusID:237142385 4. Bhargava, A., Witkowski, C., Looi, S.Z., Thomson, M.: What’s the magic word? a control theory of llm prompting (2024), https://arxiv.org/abs/2310.04444 5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, p. 1877–1901. Curran Associates, Inc. (2020), https://proceedings.neurips.c/paper_files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf 6. Chandra, R., Kulkarni, V.: Semantic and sentiment analysis of selected bhagavad gita translations using bert-based language framework. IEEE Access 10, 21291– 21315 (2022). https://doi.org/10.1109/ACCESS.2022.3152266 7. Chandra, R., Ranjan, M.: Artificial intelligence for topic modelling in hindu phi- losophy: Mapping themes between the upanishads and the bhagavad gita. PLOS ONE 17(9), 1–34 (09 2022). https://doi.org/10.1371/journal.pone.0273476, https://doi.org/10.1371/journal.pone.0273476 8. Chandra, R., Saini, R.: Biden vs trump: Modeling us general elections using bert language model. IEEE Access 9, 128494–128505 (2021). https://doi.org/10. 1109/ACCESS.2021.3111035 9. Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P.S., Yang, Q., Xie, X.: A survey on evaluation of large language models (2023) 10. Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G.J., Wong, E.: Jailbreak- ing black box large language models in twenty queries (2023) 11. Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G.J., Wong, E.: Jailbreak- ing black box large language models in twenty queries (2024), https://arxiv.org/ abs/2310.08419 12. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pondé, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F.P., Cum- mings, D.W., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W.H., Nichol, A., Babuschkin, I., Balaji, S., Jain, S., Carr, A., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M.M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., Zaremba, W.: Evaluating large language models trained on code. ArXiv abs/2107.03374 (2021), https://api.semanticscholar.org/CorpusID:235755472 13. Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., Liu, Y.: Masterkey: Automated jailbreaking of large language model chatbots. In: Pro- ceedings 2024 Network and Distributed System Security Symposium. NDSS 2024, Internet Society (2024). https://doi.org/10.14722/ndss.2024.24188, http:// dx.doi.org/10.14722/ndss.2024.24188 14. Deng, Y., Zhang, W., Pan, S.J., Bing, L.: Multilingual jailbreak challenges in large language models. In: The Twelfth International Conference on Learning Represen- tations (2024), https://openreview.net/forum?id=vESNKdEMGp Detection & Analysis of LLM Jailbreaks13 15. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirec- tional transformers for language understanding. CoRR abs/1810.04805 (2018), http://arxiv.org/abs/1810.04805 16. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding (2019), https://arxiv.org/ abs/1810.04805 17. Feng, Y., Chen, Z., Kang, Z., Wang, S., Zhu, M., Zhang, W., Chen, W.: Jail- breaklens: Visual analysis of jailbreak attacks against large language models. CoRR abs/2404.08793 (2024), http://dblp.uni-trier.de/db/journals/corr/ corr2404.html#abs-2404-08793 18. Goyal, S., Doddapaneni, S., Khapra, M.M., Ravindran, B.: A survey of adversarial defenses and robustness in nlp. ACM Comput. Surv. 55(14s) (jul 2023). https: //doi.org/10.1145/3593042, https://doi.org/10.1145/3593042 19. Hackett, W., Birch, L., Trawicki, S., Suri, N., Garraghan, P.: Bypassing prompt injection and jailbreak detection in llm guardrails (2025), https://arxiv.org/ abs/2504.11168 20. Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology 143(1), 29–36 (1982). https:// doi.org/10.1148/radiology.143.1.7063747 21. Hawkins, J., Tivey, D.: Literature filtering for systematic reviews with transform- ers. In: 2nd International Conference on Communications, Computing and Artifi- cial Intelligence (CCCAI 2024). Jeju, Korea (06 2024). https://doi.org/https: //doi.org/10.1145/3676581.3676582 22. Iu, K.Y., Wong, V.M.Y.: ChatGPT by OpenAI: The End of Litigation Lawyers? (Jan 2023). https://doi.org/10.2139/ssrn.4339839, https://papers.ssrn. com/abstract=4339839 23. Ji, J., Hou, B., Robey, A., Pappas, G.J., Hassani, H., Zhang, Y., Wong, E., Chang, S.: Defending large language models against jailbreak attacks via semantic smooth- ing (2024), https://arxiv.org/abs/2402.16192 24. Jiang, J., Wang, F., Shen, J., Kim, S., Kim, S.: A survey on large language models for code generation (2024), https://arxiv.org/abs/2406.00515 25. Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C.L., Phang, J., Bowman, S.R., Perez, E.: Pretraining language models with human preferences. In: Pro- ceedings of the 40th International Conference on Machine Learning. ICML’23, JMLR.org (2023) 26. Lee, D., Xie, S., Rahman, S., Pat, K., Lee, D., Chen, Q.A.: "prompter says": A linguistic approach to understanding and detecting jailbreak attacks against large- language models. In: Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis. p. 77–87. LAMPS ’24, Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10. 1145/3689217.3690618, https://doi.org/10.1145/3689217.3690618 27. Leins, K., Lau, J.H., Baldwin, T.: Give me convenience and give her death: Who should decide what uses of NLP are appropriate, and on what basis? In: Juraf- sky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. p. 2908–2913. Asso- ciation for Computational Linguistics, Online (Jul 2020). https://doi.org/10. 18653/v1/2020.acl-main.261, https://aclanthology.org/2020.acl-main.261 28. Li, X., Liang, S., Zhang, J., Fang, H., Liu, A., Chang, E.C.: Semantic mirror jail- break: Genetic algorithm based jailbreak prompts against open-source llms (2024), https://arxiv.org/abs/2402.14872 14J. Hawkins et al. 29. Liu, X., Xu, N., Chen, M., Xiao, C.: Autodan: Generating stealthy jailbreak prompts on aligned large language models (2024), https://arxiv.org/abs/2310. 04451 30. Liu, Y., Cao, J., Liu, C., Ding, K., Jin, L.: Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18041 (2024) 31. Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., Zhang, T., Liu, Y., Wang, H., Zheng, Y., et al.: Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499 (2023) 32. Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., Liu, Y.: Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv p. 1–10 (5 2023) 33. Luo, W., Ma, S., Liu, X., Guo, X., Xiao, C.: Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks (2024), https://arxiv.org/abs/2404.03027 34. Mondillo, G., Colosimo, S., Perrotta, A., Frattolillo, V., Indolfi, C., del Giudice, M.M., Rossi, F.: Jailbreaking large language models: navigating the crossroads of innovation, ethics, and health risks. Journal of Medical Artificial Intelligence 8(0) (2024), https://jmai.amegroups.org/article/view/9336 35. Nazir, A., Chakravarthy, T.K., Cecchini, D.A., Khajuria, R., Sharma, P., Mirik, A.T., Kocaman, V., Talby, D.: Langtest: A comprehensive evaluation library for custom llm and nlp models. Software Impacts 19, 100619 (2024) 36. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, J., Lowe, R.: Train- ing language models to follow instructions with human feedback. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neu- ral Information Processing Systems. vol. 35, p. 27730–27744. Curran Associates, Inc. (2022), https://proceedings.neurips.c/paper_files/paper/2022/file/ b1efde53be364a73914f58805a001731-Paper-Conference.pdf 37. Parciak, M., Vandevoort, B., Neven, F., Peeters, L.M., Vansummeren, S.: Schema matching with large language models: an experimental study (2024), https:// arxiv.org/abs/2407.11852 38. Peng, A., Michael, J., Sleight, H., Perez, E., Sharma, M.: Rapid response: Miti- gating llm jailbreaks with a few examples (2024), https://arxiv.org/abs/2411. 07494 39. Preti1, D., Giannone1, C., Favalli1, A., Romagnoli1, R.: Automatic summarization of legal texts, extractive summarization using llms. In: Proceedings of the 4th National Conference on Artificial Intelligence (Ital-IA 2024). Naples, Italy (May 2024) 40. Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., Mittal, P.: Visual ad- versarial examples jailbreak aligned large language models. Proceedings of the AAAI Conference on Artificial Intelligence 38, 21527–21536 (03 2024). https: //doi.org/10.1609/aaai.v38i19.30150 41. Ran, D., Liu, J., Gong, Y., Zheng, J., He, X., Cong, T., Wang, A.: Jailbreakeval: An integrated toolkit for evaluating jailbreak attempts against large language models (2024) 42. Robey, A., Wong, E., Hassani, H., Pappas, G.J.: Smoothllm: Defending large lan- guage models against jailbreaking attacks (2023) 43. Sapkota, R., Roumeliotis, K.I., Karkee, M.: Ai agents vs. agentic ai: A concep- tual taxonomy, applications and challenges (2025), https://arxiv.org/abs/2505. 10468 Detection & Analysis of LLM Jailbreaks15 44. Shen, X., Chen, Z., Backes, M., Shen, Y., Zhang, Y.: “Do Anything Now”: Charac- terizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Mod- els. In: ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM (2024) 45. Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., Truong, S.T., Arora, S., Mazeika, M., Hendrycks, D., Lin, Z., Cheng, Y., Koyejo, S., Song, D., Li, B.: Decodingtrust: A comprehensive as- sessment of trustworthiness in gpt models (2024), https://arxiv.org/abs/2306. 11698 46. Wasti, S.M., Pu, K.Q., Neshati, A.: Large language user interfaces: Voice inter- active user interfaces powered by llms. In: Arai, K. (ed.) Intelligent Systems and Applications. p. 639–655. Springer Nature Switzerland, Cham (2024) 47. Wei, A., Haghtalab, N., Steinhardt, J.: Jailbroken: How does llm safety training fail? In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Pro- cessing Systems. vol. 36, p. 80079–80110. Curran Associates, Inc. (2023), https://proceedings.neurips.c/paper_files/paper/2023/file/ fd6613131889a4b656206c50a8bd7790-Paper-Conference.pdf 48. Xu, Z., Liu, Y., Deng, G., Li, Y., Picek, S.: A comprehensive study of jailbreak attack versus defense for large language models (2024), https://arxiv.org/abs/ 2402.13457 49. Yu, J., Lin, X., Yu, Z., Xing, X.: Gptfuzzer: Red teaming large language mod- els with auto-generated jailbreak prompts (2024), https://arxiv.org/abs/2309. 10253 50. Zeng, Y., Wu, Y., Zhang, X., Wang, H., Wu, Q.: Autodefense: Multi-agent llm defense against jailbreak attacks (2024), https://arxiv.org/abs/2403.04783 51. Zhang, X., Zhang, C., Li, T., Huang, Y., Jia, X., Hu, M., Zhang, J., Liu, Y., Ma, S., Shen, C.: Jailguard: A universal detection framework for prompt-based attacks on llm systems. ACM Trans. Softw. Eng. Methodol. (Mar 2025). https: //doi.org/10.1145/3724393, https://doi.org/10.1145/3724393, just Accepted 52. Zhuang, Y., Yu, Y., Wang, K., Sun, H., Zhang, C.: Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems 36, 50117–50143 (2023) 53. Zou, A., Wang, Z., Kolter, J.Z., Fredrikson, M.: Universal and transferable ad- versarial attacks on aligned language models. ArXiv abs/2307.15043 (2023), https://api.semanticscholar.org/CorpusID:260202961