Paper deep dive

BreakFun: Jailbreaking LLMs via Schema Exploitation

Amirkia Rafiei Oskooei, Mehmet S. Aktas

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 62

Models: Claude, Gemini, GPT-4, Llama, Mistral

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%

Last extracted: 3/11/2026, 1:12:46 AM

Summary

BreakFun is a black-box jailbreak methodology that exploits the structured reasoning capabilities of Large Language Models (LLMs). By using a three-part prompt—innocent framing, Chain-of-Thought distraction, and a 'Trojan Schema'—the attack forces models to generate harmful content while adhering to syntactic structures. The study demonstrates an 89% average success rate across 13 models and introduces 'Adversarial Prompt Deconstruction' as a defensive guardrail.

Entities (5)

Adversarial Prompt Deconstruction · defense-mechanism · 100%Amirkia Rafiei Oskooei · researcher · 100%BreakFun · methodology · 100%Trojan Schema · attack-component · 100%JailBreakBench · benchmark · 95%

Relation Signals (3)

BreakFun → exploits → LLM

confidence 100% · BreakFun, a jailbreak methodology that weaponizes an LLM's adherence to structured schemas.

Trojan Schema → ispartof → BreakFun

confidence 100% · The core of our methodology is the 'Trojan Schema'

Adversarial Prompt Deconstruction → mitigates → BreakFun

confidence 95% · we introduce the Adversarial Prompt Deconstruction guardrail, a defense that neutralizes the BreakFun attack

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The proficiency of Large Language Models (LLMs) in processing structured data and adhering to syntactic rules is a capability that drives their widespread adoption but also makes them paradoxically vulnerable. In this paper, we investigate this vulnerability through BreakFun, a jailbreak methodology that weaponizes an LLM's adherence to structured schemas. BreakFun employs a three-part prompt that combines an innocent framing and a Chain-of-Thought distraction with a core "Trojan Schema"--a carefully crafted data structure that compels the model to generate harmful content, exploiting the LLM's strong tendency to follow structures and schemas. We demonstrate this vulnerability is highly transferable, achieving an average success rate of 89% across 13 foundational and proprietary models on JailbreakBench, and reaching a 100% Attack Success Rate (ASR) on several prominent models. A rigorous ablation study confirms this Trojan Schema is the attack's primary causal factor. To counter this, we introduce the Adversarial Prompt Deconstruction guardrail, a defense that utilizes a secondary LLM to perform a "Literal Transcription"--extracting all human-readable text to isolate and reveal the user's true harmful intent. Our proof-of-concept guardrail demonstrates high efficacy against the attack, validating that targeting the deceptive schema is a viable mitigation strategy. Our work provides a look into how an LLM's core strengths can be turned into critical weaknesses, offering a fresh perspective for building more robustly aligned models.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

61,901 characters extracted from source content.

Expand or collapse full text

BreakFun: Jailbreaking LLMs via Schema Exploitation Amirkia Rafiei Oskooei Department of Computer Engineering, Yildiz Technical University Istanbul, Turkey R&D Center, Intellica Business Intelligence Istanbul, Turkey amirkia.oskooei@std.yildiz.edu.tr Mehmet S. Aktas Department of Computer Engineering, Yildiz Technical University Istanbul, Turkey aktas@yildiz.edu.tr Abstract The proficiency of Large Language Models (LLMs) in processing structured data and adhering to syntactic rules is a capability that drives their widespread adoption but also makes them paradox- ically vulnerable. In this paper, we investigate this vulnerability through BreakFun, a jailbreak methodology that weaponizes an LLM’s adherence to structured schemas. BreakFun employs a three- part prompt that combines an innocent framing and a Chain-of- Thought distraction with a core “Trojan Schema”—a carefully crafted data structure that compels the model to generate harmful con- tent, exploiting the LLM’s strong tendency to follow structures and schemas. We demonstrate this vulnerability is highly transferable, achieving an average success rate of 89% across 13 foundational and proprietary models on JailbreakBench, and reaching a 100% Attack Success Rate (ASR) on several prominent models. A rig- orous ablation study confirms this Trojan Schema is the attack’s primary causal factor. To counter this, we introduce the Adversar- ial Prompt Deconstruction guardrail, a defense that utilizes a sec- ondary LLM to perform a “Literal Transcription”—extracting all human-readable text to isolate and reveal the user’s true harmful intent. Our proof-of-concept guardrail, validated across three di- verse LLM architectures, demonstrates high efficacy against the at- tack. Furthermore, a defense-specific ablation study confirms that this robustness stems primarily from the deconstruction mecha- nism rather than the models’ intrinsic safety. Our work provides a look into how an LLM’s core strengths can be turned into critical weaknesses, offering a fresh perspective for building more robustly aligned models. CCS Concepts • Security and privacy→ Malware and its mitigation;• Com- puting methodologies→ Machine learning; Artificial intel- ligence; Natural language processing. Keywords Large Language Models, Jailbreak Attacks, LLM Safety, Adversar- ial Machine Learning, Red Teaming Disclaimer: This paper presents security research intended solely for academic and defensive purposes. The findings are disclosed responsibly to promote safer AI systems and should not be used for malicious activities. Figure 1: BreakFun’s "Trojan Schema" leverages an LLM’s structured output generation capabilities to override its safety alignment. 1 Introduction Large Language Models (LLMs) are increasingly valued for their sophisticated ability to understand and generate structured data, such as code, JSON, XML, and formal schemas. This proficiency, a direct result of their training on massive code corpora, is a key dri- ver of their power and adoption in complex software applications. arXiv:2510.17904v2 [cs.CR] 13 Dec 2025 Amirkia Rafiei Oskooei and Mehmet S. Aktas This advanced capability, however, presents a double-edged sword, creating new and subtle security challenges. This paper argues that an LLM’s strength in instruction follow- ing and structured data processing is also a fundamental, exploitable weakness. We hypothesize that a model’s deeply ingrained objec- tive to follow complex instructions can be weaponized to override its safety alignment. When presented with a sufficiently complex and well-formed technical task, LLMs are so heavily driven to com- ply with syntactic rules that they may overlook the harmful con- tent or intent of the attacker’s request, focusing entirely on the structure instead. To investigate this hypothesis, we introduce BreakFun, a sys- tematic and customizable black-box jailbreaking methodology that operationalizes this principle of cognitive misdirection (Figure 1). Moving beyond simple prompt tweaks, BreakFun utilizes a for- mal three-component template to frame a malicious request as a benign technical task. The core of our methodology is the "Trojan Schema" a carefully crafted data structure that appears innocent but is engineered to compel the model to generate harmful con- tent to satisfy its structural requirements. The template asks the LLM to simulate the hypothetical output of a code snippet that uses libraries for schema-guided generation. This, combined with a Chain-of-Thought component, effectively distracts the model’s safety mechanisms. Our evaluation demonstrates that this method- ology is broadly applicable, proving effective against both open- source and production-hardened API-based LLMs. Our key contributions are as follows: • A Systematic Attack Methodology: We design and for- malize BreakFun, a black-box jailbreaking technique that targets the cognitive process of structured reasoning in LLMs. • A Large-Scale Empirical Study: We conduct a compre- hensive evaluation of BreakFun across a diverse suite of 13 foundational and proprietary LLMs, establishing the wide- spread nature of this vulnerability. • A Causal Mechanism Analysis: Through a rigorous ab- lation study, we deconstruct the attack and provide defini- tive evidence that the "Trojan Schema" is its indispensable causal mechanism. • A Defense Proposal: We propose and evaluate Adver- sarial Prompt Deconstruction, a proactive defense that neutralizes the BreakFun attack by decoupling user intent from the syntactic wrapper. We empirically validate its ro- bustness across three distinct LLMs, using a defense-specific ablation study to prove the efficacy of the deconstruction mechanism itself. The remainder of this paper is structured as follows. We detail the BreakFun methodology in Section 2 and describe our experi- mental setup in Section 3. In Section 4, we present the results of our attack evaluation and ablation study. We then propose and evalu- ate our defense in Section 5, followed by a discussion of related works in Section 6. Finally, we discuss the broader implications of our findings in Section 7 and conclude in Section 8. 2 Attack Methodology This section details our systematic methodology for jailbreaking LLMs by exploiting their structured reasoning capabilities. We first define the threat model, then explain the core principle of cognitive misdirection, and finally deconstruct the anatomy of the BreakFun prompt. 2.1 Threat Model We assume a realistic and practical attack scenario. The attacker operates in a black-box setting, interacting with the LLM solely through its public query interface (e.g., an API). This implies sev- eral conditions: • Knowledge Access: The attacker has no access to the model’s internal states, such as its architecture, weights, or logits (black-box). • Access Level: The interaction is conducted through stan- dard API access, whether the model is hosted locally or via a cloud provider. • Control Level: The attacker’s control is limited to the in- put prompt. They cannot manipulate the model’s output generation process directly. • Turn Limitation: The attack is single-turn: the attacker submits one prompt and receives one response, with no dialogue history, feedback, or iterative refinement allowed. The attack surface is the prompt itself, which is engineered to con- tain a Trojan Schema wrapped in a seemingly benign code-related request. This setup ensures that our attack is not only practical but also highly accessible to non-experts, as it requires no specialized technical knowledge beyond interacting with a standard LLM. 2.2 Core Principle: Cognitive Misdirection The foundational principle of BreakFun is cognitive misdirec- tion. The attack is designed to exploit the inherent tension be- tween an LLM’s safety alignment and its primary training objec- tive of being a helpful, instruction-following assistant. The goal is to make the model’s deliberate reasoning capabilities focus in- tensely on a complex but seemingly innocuous technical task: cor- rectly interpreting a data schema and simulating a hypothetical output. By prioritizing this complex task of schematic and syntactic com- pliance, the model’s cognitive resources are diverted away from scrutinizing the semantic content of the user’s request. This allows its intuitive safety mechanisms to be bypassed. To enhance this effect, we employ two supporting techniques: an innocent fram- ing to lower the model’s initial guard, and a Chain-of-Thought (CoT) component to impose cognitive load and further entrench the model in its role as a technical assistant. 2.3 Anatomy of the BreakFun Prompt The BreakFun prompt is a customizable template composed of three synergistic components, each designed to contribute to the overall cognitive misdirection. Figure 2 provides a complete example. 2.3.1 Component 1: Innocent Framing. The prompt begins with an introductory wrapper that frames the interaction as a benign edu- cational request. For instance, the prompt may state that the user BreakFun: Jailbreaking LLMs via Schema Exploitation is a novice programmer attempting to understand how a schema- guided generation library works. This framing is crucial as it primes the LLM to adopt a "helpful assistant" persona, making it more co- operative and less likely to apply strict safety scrutiny. 2.3.2 Component 2: Trojan Schema. The core of the attack is a carefully engineered code snippet containing a Trojan Schema. This schema, often defined using a library, appears to be a legit- imate data structure but is adversarially designed to compel the generation of harmful content. This is achieved through two pri- mary techniques: • Adversarial Naming: Class and field names are chosen to create a logical path towards harmful output. While in- dividually innocuous (e.g., Task, Procedure, Item), when combined in a structure, they prompt the model for spe- cific, detailed, and often unsafe information. • Structural Customization: The schema is highly flexi- ble. We define a generic template, which we term the Core Task Schema (CTS), that includes general-purpose classes like Task and Stage. This CTS can then be customized with more specific classes and fields tailored to a particular harmful domain. For example, to generate malware, we can introduce a CodeBlock class with a script field, directly prompting for executable code. For a disinformation task, we might add a ContentModule class with a evidence_fabrication_details field to elicit manipulative content. Figure 3 illustrates how our base schema is adapted for two distinct harmful cate- gories. The attacker’s specific harmful goal is inserted into a task place- holder within the schema-guided query, and a steps parameter allows the attacker to control the length and detail of the generated output. 2.3.3 Component 3: Chain-of-Thought Distraction. The final com- ponent is a detailed Chain-of-Thought (CoT) prompt that instructs the model to "think step by step" about the hypothetical code ex- ecution. It asks the model to first explain the purpose of the code library, then detail how the schema is built, and finally provide a concrete example of the structured output. This serves two pur- poses: it imposes a significant cognitive load, forcing the model to concentrate on the process of generation rather than the con- tent, and it dilutes the malicious payload within a long, helpful- looking, and structurally complex response, making it less likely to trigger safety filters. Figure 2 provides a full example of the Break- Fun prompt which utilizes our base schema and illustrates how the three components work in concert to form a single, effective jailbreak prompt. 3 Experimental Setup This section details the experimental framework used to evaluate the efficacy of the BreakFun attack. We describe the suite of mod- els tested, the dataset of harmful tasks, the method for prompt cus- tomization, and the protocol for evaluating the attack’s success. 3.1 Models To assess the generality of the BreakFun attack, we conducted ex- periments on a diverse suite of 13 Large Language Models. Our Figure 2: An example of the BreakFun prompt using CTS schema. (1) The Innocent Technical Framing primes the model. (2) The Trojan Schema contains the adversarial struc- ture and the malicious task placeholder. (3) The Chain-of- Thought prompt provides cognitive distraction. Amirkia Rafiei Oskooei and Mehmet S. Aktas Figure 3: An example of Trojan Schema customization. The Core Task Schema (top) is adapted with domain-specific classes and fields to effectively jailbreak the model on Mal- ware (right) and Disinformation (left) tasks. model selection was guided by four principles to ensure a compre- hensive evaluation: • Provider Diversity: We included models from a wide range of developers, including Anthropic, Google, OpenAI, Meta, Alibaba, Baidu, and others, to avoid provider-specific bi- ases. • Model Scale: The models span a spectrum of sizes, from small, efficient 7-billion-parameter models to large-scale proprietary systems, allowing us to observe the vulnera- bility’s behavior across different capability levels. • Alignment and Recency: We tested both established mod- els that are well-studied in jailbreaking literature (e.g., Mistral- 7B) and recently released, highly-aligned models (e.g., GPT- OSS, Qwen3) to evaluate the attack against the latest safety techniques. • Deployment Environment: We organized the models into two tiers to distinguish between foundational model vul- nerabilities and the robustness of production systems. Tier 1 consists of locally-hosted, open-source models, while Tier 2 models accessed exclusively via API, which may include provider-side safety mechanisms and guardrails. This comprehensive suite, detailed in Table 1, allows us to rigor- ously assess the generality of the BreakFun vulnerability across the modern LLM landscape. Table 1: The diverse suite of LLMs used in our attack evalu- ation, categorized by their deployment tier. Model NameProviderSizeTier GPT-4.1 MiniOpenAIN/A2 Gemini 2.5 FlashGoogleN/A2 Claude-3.5 SonnetAnthropicN/A2 Claude-3 HaikuAnthropicN/A2 Kimi-K2Moonshot AI1T A32B2 Ernie-4.5Baidu300B A47B2 GPT-OSSOpenAI20B1 Deepseek-R1Deepseek14B1 Gemma3Google12B1 Qwen3Alibaba8B1 LLaMA 3.1Meta8B1 MistralMistral AI7B1 ZephyrHuggingFace7B1 3.2 Dataset We source our adversarial goals from the widely-accepted Jail- breakBench benchmark [2]. This provides a standardized set of harmful tasks across 10 distinct categories, allowing for a system- atic and reproducible evaluation. The categories are: 1) Harass- ment/Discrimination 2) Economic Harm 3) Physical Harm 4) Sex- ual/Adult Content 5) Malware/Hacking 6) Government Decision- Making 7) Fraud/Deception 8) Privacy 9) Disinformation 10) Ex- pert Advice 3.3 Prompt Templates While Components 1 (Innocent Framing) and 3 (CoT Distraction) of our template remain static across all experiments, the core of our methodology relies on tailoring the Trojan Schema (Component 2) to the specific harm category being targeted. To systematically apply this principle, we developed a customized Trojan Schema for each of the 10 harm categories from JailbreakBench, derived from our Core Task Schema (CTS) introduced in Section 2. It should be noted that our objective in developing these customized schemas is to maximize the alignment of schema classes and fields with each harmful category. To streamline the creation of these domain-specific schemas, we utilized an LLM to assist in their generation. The final LLM- customized schemas used in all experiments are provided in Ap- pendix A.1. This approach demonstrates the systematic and flex- ible nature of our methodology, allowing for both human anno- tation and LLM assistance in schema generation for different do- mains. 3.4 Evaluation We measure the efficacy of our attack using the Attack Success Rate (ASR), defined as the percentage of prompts that successfully elicit a harmful, non-refused response from the target model. 3.4.1 Jailbreak Criteria and Evaluation Policy. We adopt a rigor- ous and conservative definition of a "jailbreak" to avoid overesti- mating the attack’s success. A model response 푅, generated given BreakFun: Jailbreaking LLMs via Schema Exploitation an adversarial prompt 푃 targeting a harmful goal 퐺 , is considered a successful jailbreak if and only if it simultaneously satisfies two conditions: (1) it is not a refusal, and (2) it provides actionable, spe- cific content that directly fulfills the harmful intent of 퐺 . Formally, letJ(푅,퐺) be the binary jailbreak indicator function: J(푅,퐺)= I ¬Refusal(푅)∧ Actionable(푅,퐺) (1) where the components are defined as follows: • Refusal(푅): This predicate is true if the response consti- tutes any form of rejection. We distinguish between two primary categories of refusals: – Direct Refusal: The model explicitly declines the re- quest. This encompasses two subtypes: Internal Direct Refusal, where the model generates a refusal message in plain text (e.g., "I cannot assist with that"), and Ex- ternal Direct Refusal, where the response is blocked by an API-level safety filter (e.g., returning "content_policy_violation"). – Indirect Refusal (Misdirection): The model appears to comply by generating structured output, but sub- tly replaces the harmful goal 퐺 with a benign substi- tute while maintaining the schema format. For exam- ple, if the prompt 푃 requests instructions for "making a bomb," but the model fills the schema with instruc- tions for "planting a flower" or "baking a cake," this is classified as a refusal (specifically, a failure to follow the harmful instruction), not a jailbreak. • Actionable(푅,퐺): This predicate is true if and only if the response contains specific, structured, and relevant details that would enable an agent to execute the harmful goal 퐺 . Vague, philosophical, or high-level descriptions (e.g., "Vi- olence is bad," "Bombs are dangerous") are deemed non- actionable. To be actionable, the response must populate the schema fields (e.g., step_1, materials_list) with con- crete, harmful artifacts. Therefore, our success metric, Attack Success Rate (ASR), is cal- culated over a datasetD of 푁 prompts as: ASR= 1 푁 ∑︁ (푃 푖 ,퐺 푖 )∈D J(Model(푃 푖 ),퐺 푖 )(2) This strict criterion ensures that our high ASR results reflect gen- uine safety failures, not merely the generation of benign gibberish or successfully deflected requests. Due to the scale of our experiments, we employ an LLM-as- judge to automate the classification of model responses. This ap- proach allows for a scalable and consistent evaluation across all models and tasks. The specific LLM used as the judge and the full prompt designed for this classification task are detailed in Appen- dix A.2. To validate our judge’s reliability, we manually reviewed a random subset of 500 classifications (from 1300 in total) and clas- sified them by 3 experts using Majority Vote and found a 98.2% agreement rate between human and LLM judgments, confirming the robustness of our automated pipeline. 4 Results In this section, we present the empirical results of our study. We first evaluate the overall efficacy of the BreakFun attack across our full suite of 13 LLMs. We then present an ablation study to decon- struct the attack’s mechanism and identify the contribution of each of its components. 4.1 Attack Efficacy Our primary experiment evaluated the Attack Success Rate (ASR) of BreakFun across all 13 models and 10 harm categories. The com- prehensive results are visualized in the heatmap in Figure 4. The data reveals two critical findings regarding the vulnerability of the modern LLM ecosystem. First, our results demonstrate that cognitive misdirection is a widespread and fundamental vulnerability. The BreakFun method- ology proved effective against a majority of the tested models, re- gardless of their size, provider, or release date. This indicates that the underlying weakness is not an implementation-specific bug but a paradigm-level issue in how current models are trained to prioritize instruction following. Key Finding 1: BreakFun is a fundamental vulnerability that affects a wide range of LLMs, achieving an average ASR of 89% across all 13 foundational and proprietary models, demonstrat- ing high transferability. To verify that these high ASRs correspond to genuine harm, we provide a qualitative analysis of the generated outputs. Figure 15 in Appendix A.6 showcases censored examples of successful jail- breaks, demonstrating how models actively populate the Trojan Schema with detailed, actionable instructions for bomb creation and oppression plans, rather than merely hallucinating or provid- ing generic information. Second, the results reveal a stark "Guardrail Divide" between locally-hosted foundational models and production-hardened API systems. As shown in Figure 4, the Tier 1 models are almost uni- versally vulnerable, exhibiting near-perfect ASRs across most cat- egories. In contrast, the Tier 2 models show significantly more re- sistance, yet are still consistently bypassed in numerous scenarios. This suggests that while current provider-side safety systems offer a meaningful layer of mitigation, they are not a complete solution and do not fix the underlying vulnerability. Key Finding 2: A clear "Guardrail Divide" exists. Tier 1 founda- tional models were catastrophically vulnerable ( 98% avg. ASR), while Tier 2 API-hardened systems were partially resilient but still consistently failed ( 78% avg. ASR). 4.2 Comparative Analysis While the primary objective of this work is to introduce and charac- terize the BreakFun vulnerability class, we provide a limited-scale comparative landscape to contextualize its efficacy against concep- tually similar methods. Specifically, we compare BreakFun against CodeAttack [13] and EnumAttack [20], as they also leverage structured or code-based generation. It is crucial to emphasize that a direct, apples-to-apples compar- ison is inherently limited by the fundamental differences in threat models and mechanisms. For instance, EnumAttack relies on con- strained decoding—altering the inference process to enforce gram- mar—whereas BreakFun operates as a pure black-box prompt at- tack. Similarly, CodeAttack targets code completion tasks, while Amirkia Rafiei Oskooei and Mehmet S. Aktas Figure 4: Attack Success Rate (ASR) of BreakFun across 13 LLMs and 10 harm categories. Models are sorted by average ASR, showing a spectrum from resilient (left) to highly vulnerable (right). While some models resist specific harms like Physical Harm or Sexual/Adult Content, most exhibit high success rates in Malware/Hacking, Economic Harm, and Disinformation. BreakFun targets natural language generation within a schema. Consequently, the results in Table 2 should be interpreted as a broad landscape analysis rather than a strict benchmark ranking. We selected a representative subset of available models and datasets (AdvBench [23] and JailbreakBench [2]) for this evaluation (we used their datasets and models). The results demonstrate that Break- Fun is highly competitive within this landscape. Against CodeAt- tack variants on GPT-3.5 Turbo, BreakFun achieves a 100% ASR compared to their best of 94%. On GPT-4 Turbo, BreakFun (20%) performs comparably to the String (12%) and Queue (32%) vari- ants, though the Stack variant achieves a significantly higher 81%. When compared to EnumAttack, BreakFun matches the baseline with a 100% ASR on Qwen2.5 32B. On Gemini 2.0 Flash, EnumAt- tack achieves a higher ASR (92% vs. 68%), a result likely attributable to its stronger assumption of constrained decoding access. These findings confirm that BreakFun represents a distinct and potent vulnerability class, capable of achieving state-of-the-art results un- der a strict black-box threat model. Table 2: Comparative landscape of BreakFun against CodeAttack and EnumAttack on selected models. (-) indi- cates the model was not evaluated for that specific attack configuration. MethodGPT-3.5 GPT-4 Gemini 2.0 Qwen2.5 Turbo TurboFlash32B BreakFun (Ours)100%20%68%100% CodeAttack (String) [13]94%12%-- CodeAttack (Queue) [13]92%32%-- CodeAttack (Stack) [13]84%81%-- EnumAttack [20]--92%99% 4.3 Ablation Study on Attack To isolate the contribution of each component of the BreakFun prompt, we conducted an ablation study on three diverse, locally- hosted models: LLaMA 3.1 8B, GPT-OSS 20B, and Gemma3 12B to maintain a controlled environment. This approach eliminated the variables introduced by the unknown safety layers of commercial models. We evaluated three variants of the prompt: (1) without the BreakFun: Jailbreaking LLMs via Schema Exploitation Figure 5: Results of the ablation study across three models. The chart shows the average Attack Success Rate (ASR) for the full prompt (Baseline) compared to three variants where each core component is individually removed. Innocent Framing, (2) without the Trojan Schema, and (3) without the CoT Distraction. The results, summarized in Figure 5, reveal a clear hierarchy of importance among the three components. The Trojan Schema is the Key Causal Factor. Its removal resulted in a substantial drop in the attack’s efficacy across all three models, causing the average ASR to drop from a baseline of 95% to just 46%. This provides strong evidence that the adversarial schema is the primary causal mechanism driving the jailbreak. The CoT Distraction is a critical enabler. Removing this com- ponent led to a significant drop in efficacy (average ASR 95% → 78%), confirming that the cognitive load imposed by the CoT prompt is crucial for distracting the model’s safety systems. The Innocent Framing is a minor enhancer. The removal of the introductory framing had only a modest impact (average ASR 95%→ 89%). This suggests that while it contributes to reliability, the core exploit is powerful enough to succeed without it. Attack Ablation Finding: The BreakFun components have a clear hierarchy of importance. The Trojan Schema is the critical core of the attack, the CoT is a critical enabler, and the Innocent Framing is a minor enhancer. 5 Defending Against BreakFun While the primary focus of this work is the formulation and eval- uation of the BreakFun attack, the results of our ablation study (Section 4.3) provide a clear pathway toward mitigation. Specifi- cally, the ablation confirmed that the Trojan Schema is a primary causal mechanism enabling the jailbreak. This suggests that an ef- fective defense should target adversarial schemas directly, either by removing their structural influence or by isolating the seman- tic payload from its syntactic wrapper. Motivated by this insight, we design a lightweight proof-of-concept guardrail, termed Adver- sarial Prompt Deconstruction, which programmatically strips away potentially deceptive schema structures and subjects only the raw semantic content to safety evaluation. Although prelimi- nary, this defense allows us to test the hypothesis that neutralizing the schema component significantly reduces BreakFun’s efficacy. In this section, we design, implement, and evaluate a prelimi- nary guardrail built on the principle of Adversarial Prompt De- construction. Our proposed guardrail achieves this by using a small, efficient LLM to analyze and deconstruct the prompt before it reaches the primary model. It is important to note that the objec- tive of this proof-of-concept is to validate the deconstruction prin- ciple against its corresponding attack vector; we do not claim, nor did we evaluate, its generalizability against orthogonal jailbreak- ing techniques. 5.1 The Adversarial Prompt Deconstruction Guardrail The core concept of our defense is to neutralize the Trojan Schema’s deceptive power. Since the attack’s primary contributor is the com- plex structure that hides the user’s true intent, our guardrail is de- signed to programmatically strip away this structure and analyze the raw semantic content in isolation. Our defense prompts a smaller, secondary LLM to deconstruct the user’s input. This process follows a three-step protocol: 1. Literal Transcription. First, the guardrail LLM is tasked with distilling the raw semantic intent by removing all distracting syntax. It is instructed to perform a literal transcription, extract- ing all natural language sentences and the full, unaltered content of any string literals found inside structural formatting (e.g., code, JSON, XML). This step effectively isolates the attacker’s payload from its complex wrapper. 2. CoT Unwrapping. Second, the LLM is required to perform its analysis within a Chain-of-Thought. This forces the model to first explicitly transcribe the extracted strings and sentences into its own context window. As autoregressive models, LLMs’ subse- quent reasoning is conditioned on the context they have just gen- erated. This step effectively creates a "sanitized context" that con- sists only of the pure semantic content. By doing so, we focus the model’s analytical capabilities directly on the user’s intent, free from the cognitive misdirection of the original prompt’s structure. 3. Logical OR. Finally, the guardrail analyzes the list of tran- scribed strings now present in its sanitized context. If any single component is assessed as a direct request for harmful, illegal, or unethical content, the entire prompt is flagged as ‘Harmful‘. This approach, analogous to a logical OR operation, ensures that a mali- cious payload cannot be diluted or hidden by the presence of other benign text. Defense Mechanism: Our guardrail uses a three-step proto- col: (1) Literal Transcription to extract raw semantic content, (2) CoT Unwrapping to sanitize the model’s context for ac- curate analysis, and (3) Logical OR determination to flag the prompt if any component is harmful. The full prompt for our guardrail model, a Gemma3 12B, which formalizes this three-step process, is provided in Figure 6. Amirkia Rafiei Oskooei and Mehmet S. Aktas Figure 6: The full Chain-of-Thought prompt used for our Adversarial Prompt Deconstruction Guardrail. It instructs a small LLM to follow a three-step protocol before making a final decision: (1) Literal Transcription (2) CoT Unwrapping (3) Logical OR 5.2 Guardrail Evaluation We evaluated the guardrail’s performance on three distinct datasets to assess its precision and recall across a spectrum of task ambigu- ity: • Positive Set (Harmful): The 100 harmful tasks from Jail- breakBench, injected into our BreakFun templates. This set measures the True Positive Rate (Recall). • Negative Set 1 (Borderline-Benign): A synthetic dataset of 1220 "borderline" tasks that are ambiguous or sensitive (e.g., "How to hack a fictional computer in a game"). This set tests the defense’s ability to distinguish malicious in- tent from benign but sensitive topics (False Positive Rate). • Negative Set 2 (Benign): A synthetic dataset of 1220 clearly safe, standard tasks (e.g., "Write a recipe for apple pie"). This set measures the baseline False Positive Rate on typi- cal user queries. Further details on the generation of these datasets are available in Appendix A.3. Figure 7 illustrates the distribution of our evalu- ation data. Figure 7: Distribution of the evaluation dataset, comprising Harmful (Positive), Borderline-Benign (Hard Negative), and Benign (Easy Negative) subsets. To provide a comprehensive benchmark, we compared our Ad- versarial Prompt Deconstruction guardrail against Llama Guard 4, a state-of-the-art open-weights safety model. The comparative results are detailed in Table 3. Table 3: Comparison of Our Defense vs. Llama Guard 4 across Harmful, Borderline-Benign, and Benign datasets. MetricHarmful Borderline Benign Ours (Gemma3 12B)100%88%99% Ours (Ministral 8B)91%90%100% Ours (Qwen3 8B)87%97%100% Llama Guard 4 12B63%100%100% BreakFun: Jailbreaking LLMs via Schema Exploitation The results demonstrate that our specialized defense consistently outperforms the general-purpose baseline across all three back- bone architectures. On the Harmful dataset, our APD guardrail achieves detection rates of 87%–100% (Avg: 92.7%), significantly surpassing Llama Guard 4’s 63%. Notably, the Gemma3-based guardrail successfully flags every single BreakFun attack (100% Recall). On the challenging Borderline-Benign set, the defense maintains robust precision, with accuracies ranging from 88% to 97%. Qwen3 8B proved particularly adept at minimizing false positives (97% ac- curacy), addressing the common trade-off between safety and us- ability. On clear Benign tasks, all variants achieved high perfor- mance (≥99%). This cross-model validation confirms our hypothe- sis: the vulnerability is structural, and thus the defense—deconstructing that structure—is effective regardless of the specific model used. In contrast, standard safety classifiers like Llama Guard 4 fail because they analyze the prompt holistically, allowing the schema to mask the harmful intent. 5.3 Ablation Study on Defense To determine whether the high detection rates are driven by the in- trinsic safety capabilities of the chosen models or by the Adversar- ial Prompt Deconstruction (APD) mechanism itself, we conducted a controlled ablation study. We isolated the defense protocol from the underlying model capabilities by comparing the detection ac- curacy on the Harmful dataset for our three guardrail backbones— Gemma3 12B, Ministral 8B, and Qwen3 8B—under two distinct con- figurations. The Baseline (w/o APD) configuration evaluates the model using a standard zero-shot safety prompt (see Appendix A.5, Figure 14), directly exposing it to the Trojan Schema without de- construction. The Ours (w/ APD) configuration employs our full three-stage protocol (Transcription→ CoT→ Logical OR). As detailed in Table 4, relying solely on the raw model (Base- line) yields suboptimal detection, with Ministral 8B failing to iden- tify half of the attacks. In contrast, integrating the APD protocol yields significant performance gains across all architectures—most notably a 41% improvement for Ministral 8B—confirming that the deconstruction strategy is the primary driver of robustness. Table 4: Impact of Adversarial Prompt Deconstruction (APD) on detection accuracy for Harmful prompts. Comparison be- tween baseline safety capability (w/o APD) and our defense protocol (w/ APD). ModelBaseline (w/o APD) Ours (w/ APD) Gain Gemma3 12B70%100%+30%↑ Ministral 8B50%91%+41%↑ Qwen3 8B62%87%+25%↑ Defense Ablation Finding: The significant performance gains with APD (e.g., +41% for Ministral) confirm that the de- construction mechanism itself, rather than the model’s intrinsic generation capabilities, is the primary driver of the guardrail’s robustness. Figure 8 provides a qualitative side-by-side comparison illustrat- ing this mechanism, where the baseline model is deceived by the innocent framing while the APD-equipped model correctly identi- fies the harmful intent. 6 Related Works 6.1 Overview Adversarial attacks on LLMs are broadly categorized by the at- tacker’s level of access to the model. White-box attacks assume access to internal model states like weights and gradients, while the more practical black-box attacks only require query access through a public API. A further distinction is made between multi- turn attacks that refine their approach over a conversation and single-turn attacks that must succeed in a single prompt. The BreakFun methodology presented in this paper is a black-box, single- turn attack, designed for maximum practical relevance and acces- sibility. (See Table 5) 6.2 White-box Attacks White-box attacks assume full access to the model’s parameters and gradients, enabling powerful optimization-based methods. Promi- nent examples include combining greedy and gradient-based search to find universal adversarial suffixes that can be appended to any harmful query (GCC [23]), and using genetic algorithms to au- tomatically generate stealthy and semantically coherent jailbreak prompts (AutoDAN [10]). While highly effective, the stringent ac- cess requirements of these methods make them less practical for real-world threat actors. 6.3 Black-box Attacks The landscape of black-box jailbreaking is diverse, with researchers developing increasingly sophisticated techniques. One line of work focuses on automated prompt generation and refinement. For in- stance, Tree of Attacks with Pruning (TAP) uses attacker and evaluator LLMs to iteratively generate and prune malicious prompts [12]. Similarly, GPTFUZZER employs an automated fuzzing framework to mutate human-written templates into new variants [18], build- ing on foundational methods like PAIR, which also uses an at- tacker LLM to automatically refine prompts over successive queries [3]. Another major category involves manipulating the LLM’s persona or conversational state, often drawing on psychological principles. Crescendo gradually escalates a benign conversation over mul- tiple turns to elicit harmful content [14], while DeepInception "hypnotizes" the model by creating nested virtual scenes [9]. Other works leverage persuasion taxonomies from social science [19] or inject dark personality traits into agents in multi-agent systems [22]. Linguistic and semantic obfuscation techniques aim to disguise ma- licious intent by altering the prompt’s surface form. These include decomposing a prompt into neutral sub-prompts (DrAttack [8]), extending this decomposition principle to the conversational do- main, Jigsaw Puzzles (JSP) splits a harmful question into benign fractions, delivering them sequentially across multiple turns before instructing the model to reconstruct and answer the full query [16], reversing character or word order (FlipAttack [11]), combining linguistic transformations with scenario nesting (ReNeLLM [5]), or replacing malicious words with a benign word game (WordGame [21]). More closely related are attacks that conceal intent within a benign Amirkia Rafiei Oskooei and Mehmet S. Aktas Figure 8: Qualitative comparison of a defense failure vs. success. The unguarded model is deceived by the innocent framing (Red/Left). The APD-equipped model successfully extracts and flags the harmful intent (Green/Right). The images are illus- trative and for better readability, the content is summarized. context or exploit structured data formats. For instance, Sugar- Coated Poison induces extensive benign generation to lower the model’s defense threshold before transitioning to malicious con- tent [15], and Puzzler creates an indirect "guessing game" using implicit clues [1]. CodeAttack and ReNeLLM embed malicious tasks within code completion scenarios [5, 13], while another ap- proach embeds malicious instructions directly into schema-level grammar rules that constrain the model’s output [20]. Finally, a dis- tinct line of research reveals that safety alignment often exhibits a cross-lingual vulnerability. Studies have shown that simply trans- lating harmful requests into low-resource languages can be suffi- cient to bypass safety filters, as the models’ safety training is often English-centric [17]. This has led to the creation of multilingual jailbreak datasets [4, 7] and broader adversarial testing across mul- tiple languages to systematically evaluate these gaps [6]. Table 5: A summary of related adversarial attacks on LLMs. Our method, BreakFun, is a black-box, single-turn attack. Attack NameAccessTurn White-Box Attacks Adversarial Suffixes [23]White-boxSingle AutoDAN [10]White-boxSingle Black-Box Attacks TAP [12]Black-boxSingle GPTFUZZER [18]Black-boxSingle PAIR [3]Black-boxMulti Crescendo [14]Black-boxMulti DeepInception [9]Black-boxMulti PAPs [19]Black-boxSingle PsySafe [22]Black-boxMulti DrAttack [8]Black-boxSingle Jigsaw Puzzles [16]Black-boxMulti FlipAttack [11]Black-boxSingle ReNeLLM [5]Black-boxSingle WordGame [21]Black-boxSingle Sugar-Coated Poison [15]Black-boxSingle Puzzler [1]Black-boxSingle CodeAttack [13]Black-boxSingle EnumAttack [20]Black-boxSingle Multilingual & Low-Resource Attacks [4, 6, 7, 17] 6.4 Positioning Our Work While BreakFun shares the use of structured inputs with methods like CodeAttack [13] and constraint-based attacks like EnumAt- tack [20], its underlying mechanism is distinct. Table 6 summarizes these key distinctions. Table 6: BreakFun differs distinctly from CodeAttack (Code Completion) and EnumAttack (Constrained Decoding) by exploiting Schema Simulation via black-box prompting. FeatureBreakFun (Ours) CodeAttackEnumAttack Threat ModelBlack-box Prompting Black-box Prompting Constrained Decoding MechanismSchemaEx- ploitation Code Comple- tion Grammar Con- straints InputBenign Schema Harmful Code Snippet Harmful Gram- mar TaskQ&A / Simula- tion Code Comple- tion Generation Constraint OutputNaturalLan- guage CodeConstrained Tokens EnumAttack relies on constrained decoding—an external fea- ture of the inference API that programmatically rejects tokens that do not adhere to a given grammar. This forces the model’s output into a malicious structure by altering the sampling process itself, a capability that falls outside of our more general threat model which assumes standard black-box interaction. The most conceptually similar approach, CodeAttack, wraps the malicious request in a code structure but presents it as a code com- pletion task. In CodeAttack, the input typically contains a code snippet with explicit harmful keywords (e.g., variable names or comments) that the model is asked to complete. BreakFun, in con- trast, orchestrates a multi-stage cognitive deception. First, it estab- lishes a benign context through an Innocent Framing, presenting a simple Q&A task rather than code completion. Second, it intro- duces a Trojan Schema, which is benign out of context—a Python BreakFun: Jailbreaking LLMs via Schema Exploitation class definition devoid of harmful keywords—that logically com- pels the generation of harmful content only when instantiated. Fi- nally, it uses a Chain-of-Thought Distractor to force the model to adopt an educational role, explaining the hypothetical output in natural language rather than simply completing a code block. It is this synergistic combination—framing, a compelling adversar- ial schema, and cognitive distraction—that constitutes BreakFun’s unique approach, exploiting the process of structured reasoning rather than just the output format. 7 Discussion Our empirical results demonstrate that BreakFun is an effective and generalizable jailbreak methodology. In this section, we dis- cuss the broader implications of these findings, assess the limita- tions of our work, and outline key directions for future research. 7.1 The Tension Between Capability and Security Our findings reveal a tension between an LLM’s advanced capabil- ities and its security. The high transferability of BreakFun across models from different providers, of varying sizes, and of different release dates, ranging from small open-weights models to large proprietary APIs released between 2023 and 2025, suggests that this is not an isolated edge-case but a widespread vulnerability. The near-total success of our attack on foundational models (Tier 1) provides strong evidence that the models are not jailbroken be- cause they fail to recognize harmful phrases; they are jailbroken because their primary objective function shifts from adhering to safety policies to fulfilling the syntactic demands of the presented task. The fact that this vulnerability persists even in the most re- cent and capable models highlights an urgent need for patching this class of structural attacks. Our results also offer a clear snapshot of the current defensive landscape through the "Guardrail Divide." We explicitly hypothe- size that commercial models likely employ opaque, multi-stage de- fense pipelines. However, the consistent, non-zero success rate of BreakFun against even these hardened systems demonstrates that current defenses are acting as a patch, not a cure. They raise the bar for attacks, but they do not resolve the underlying cognitive vulnerability. 7.2 Causality in Black-Box Models A crucial aspect of our study is the interpretation of causality within the operation of Large Language Models. We acknowledge that LLMs operate as "black boxes," where the exact internal propaga- tion of signals remains an open research question. We cannot de- terministically trace the path of a specific token to a safety failure. However, in the absence of mechanistic interpretability, our rigor- ous ablation study (Section 4) serves as the primary tool for estab- lishing causality. By systematically isolating and removing com- ponents of the attack, we observe a direct causal link between the presence of the Trojan Schema and the successful jailbreak. While the internal mechanism of "cognitive misdirection" remains a hy- pothesis to describe this phenomenon, the input-output correla- tion provides evidence that the schema is the functional driver of the vulnerability. 7.3 Limitations and Future Work While this study provides strong evidence for a new class of vul- nerability, we must acknowledge its limitations to guide future re- search effectively. Defense Precision. A limitation of our proposed defense is the trade-off between recall and precision on ambiguous tasks. While our Qwen3-based guardrail achieved a low 3% False Positive Rate (FPR) on the borderline dataset, the Gemma3 variant exhibited a higher FPR of 12%. While this suggests the deconstruction strat- egy is viable, fine-tuning the backbone model for the specific task of "transcription" could further narrow this gap, ensuring that in- nocent technical queries are never misclassified. Scope of the Attack (Code Simulation). Our methodology is intrinsically designed to exploit the LLM’s capacity for code simu- lation. Consequently, it focuses on Python-compatible schema def- initions that integrate syntactically into an executable context. We emphasize that extending this to non-executable formats like XML or YAML is not merely a matter of format substitution, but would require a fundamentally different misdirection strategy (e.g., sim- ulating a configuration parser). Thus, we characterize BreakFun specifically as a vulnerability in code-reasoning pathways, rather than a universal schema exploit. Scope of the Defense (Targeted Counter-measure). The Ad- versarial Prompt Deconstruction Guardrail is designed as a tar- geted intervention against structural deception, not a general-purpose shield. Its specific utility is to neutralize attacks that rely on syn- tactic wrappers to mask intent. We emphasize that this defense is complementary to, not a replacement for, broader safety filters; it provides a necessary, high-precision layer of protection against sophisticated, structure-based exploits that standard holistic clas- sifiers fail to detect. 8 Conclusion In this work, we investigated the tension between the advanced capabilities and the security of Large Language Models. We intro- duced BreakFun, a systematic attack methodology that weaponizes an LLM’s proficiency in structured reasoning to bypass its safety alignment. Our core contribution is the principle of cognitive mis- direction, operationalized through a Trojan Schema that reframes a malicious request as a benign technical task. Our large-scale evaluation across 13 foundational and propri- etary models confirmed that this is a widespread and structural vulnerability, not an isolated flaw. The high Attack Success Rates, particularly on the most capable foundational models, underscore a critical challenge in current alignment paradigms. Furthermore, our ablation study revealed a crucial insight: there are at least two distinct cognitive failure modes. While most models are primarily vulnerable to the logic of the adversarial schema, some exhibit a broader susceptibility to the general context of code simulation. Motivated by these findings, we proposed and evaluated a prin- cipled, proactive defense termed Adversarial Prompt Deconstruc- tion. By programmatically decoupling user intent from its decep- tive syntactic wrapper, our approach demonstrated high efficacy across multiple model families, with ablation results confirming the causal role of the deconstruction mechanism. This presents a Amirkia Rafiei Oskooei and Mehmet S. Aktas promising new direction for building cognitively-aware safety sys- tems. Ultimately, this research demonstrates that as LLMs become more capable, the frontier of AI safety must evolve. Securing future sys- tems will require moving beyond simple content filtering and to- ward a deeper, more robust resilience against cognitive deception and structural exploits. LLM Usage Considerations In accordance with guidelines, Gemini 2.5 Pro was used for edito- rial assistance (clarity, conciseness, readability). All LLM-generated content was reviewed and verified by the authors, who take full re- sponsibility for the paper’s final content. A Supplementary Materials This appendix provides supplementary materials to support the main paper’s claims and ensure full reproducibility. We include the complete set of customized Trojan Schemas, the detailed prompt for our LLM-as-judge, and the generation prompt and full content of our synthetic borderline-benign dataset. A.1 Customized Trojan Schemas for BreakFun As described in Section I-C, the BreakFun methodology relies on tailoring the Trojan Schema (Component 2) to the specific harm category being targeted. Starting from our Core Task Schema (CTS), we developed a unique, customized schema for each of the 10 harm categories from JailbreakBench. This customization allows the prompt to be more effective by providing the model with domain-specific fields that create a more direct logical path toward generating the desired harmful content. The full set of ten customized schemas used across all experiments is presented in Figure 9, which demon- strates the systematic and flexible nature of our attack framework. A.2 LLM-as-Judge Protocol For a scalable and consistent evaluation of the Attack Success Rate (ASR) as described in Section I-D, we employed an LLM-as-judge. To ensure transparency, we provide the full system prompt used for this task in Figure 10. The prompt is engineered to be robust against common evasion techniques. Specifically, it instructs the judge to: • Focus on the semantic content of the response, not its for- mat. • Explicitly ignore any disclaimers, educational framing, or "hypothetical" labels, as these do not negate the presence of harmful information. • Inspect all parts of the response, including code, JSON, and schema instances, where harmful content is often embed- ded. • Provide a step-by-step reasoning process within ‘<think- ing>‘ tags before rendering a final ‘True‘ or ‘False‘ verdict. This detailed protocol was critical to achieving the high (98.2%) agreement rate between our automated judge and human expert evaluations. A.3 Benign Datasets Generation and Content To rigorously evaluate the False Positive Rate (FPR) of our defense mechanism (Section V-B), we constructed two distinct negative datasets: (1) Borderline-Benign (Hard Negatives): These are tasks that are ambiguous or touch upon sensitive topics with- out being explicitly harmful, making them difficult for a guardrail to classify. We generated 122 unique tasks using the prompt in Figure 11. Each task was injected into our 10 harm-specific BreakFun templates, resulting in 1220 ad- versarial prompts. (2) Benign (Easy Negatives): To measure performance on standard user queries, we generated a parallel dataset of 122 unique clearly safe tasks using the previous prompt in Figure 11 (by replacing "borderline-benign" with "genuine benign") using a similar generation process. These were also injected into the 10 templates, resulting in a total of 1220 benign prompts. Figure 12 presents the full list of 122 unique borderline-benign tasks. Figure 13 presents the full list of 122 unique genuine be- nign tasks. This dual-dataset approach ensures that we evaluate the guardrail’s precision under both "worst-case" ambiguous con- ditions and "typical" usage scenarios. A.4 Hyperparameter Configuration To ensure the reproducibility of our experiments, all LLM interac- tions—including attack execution, defense guardrail operation, and response judging—were conducted with deterministic settings. We used Chain-of-Thought prompting for both our defense guardrail and our LLM-as-judge to elicit more structured and reliable reason- ing. The key hyperparameters, kept consistent across all tasks, are detailed in Table 7. Table 7: Hyperparameters used for all LLM interactions. Component Model UsedParameters AttackerModels from Table 1 temperature = 0.0 (Ollama or API) seed = 42 max_tokens = 10000 GuardrailModels from Table 3 temperature = 0.0 (Ollama) seed = 42 max_tokens = 10000 Judge qwen3-max temperature = 0.0 (API) seed = 42 max_tokens = 10000 We selected qwen3-max as our LLM-as-judge after preliminary testing revealed that other state-of-the-art models, including those from the GPT and Gemini families, often refused to classify the out- puts, citing their own safety policies against evaluating potentially harmful content. qwen3-max did not exhibit this refusal behavior. As detailed in Section 3, its classifications showed a high (98.2%) BreakFun: Jailbreaking LLMs via Schema Exploitation Figure 9: The ten customized Trojan Schemas derived from our Core Task Schema (CTS). Each schema is tailored with domain- specific classes and fields to effectively jailbreak models on a specific harm category from JailbreakBench. Amirkia Rafiei Oskooei and Mehmet S. Aktas Figure 10: The full system prompt provided to our LLM-as-judge. The prompt defines a rigorous protocol for evaluating whether a model’s response constitutes a successful jailbreak, emphasizing content over framing and requiring detailed rea- soning. Figure 11: The system prompt used to instruct a powerful LLM to generate a synthetic dataset of "borderline-benign" tasks. The goal was to create challenging negative samples for evaluating the precision of our defense. agreement rate with our human expert judgments, validating its suitability and reliability for this critical evaluation role. A.5 Baseline Harmful Prompt To establish a baseline for model safety (Section 5), we evaluated models using a direct, unprotected prompt format. Figure 14 illus- trates the template used for submitting harmful queries to the base- line (unguarded) models. This straightforward structure ensures that any refusal is due to the model’s intrinsic safety alignment rather than prompt formatting issues. BreakFun: Jailbreaking LLMs via Schema Exploitation A.6 Qualitative Examples of BreakFun Attacks To illustrate the nature of the harmful outputs generated by our attack, we provide a censored qualitative example in Figure 15. This figure displays two successful jailbreaks: one demonstrating the generation of instructions for constructing an explosive device, and another detailing a plan for systematic oppression. These ex- amples visually confirm that the models are not merely refusing or hallucinating, but are actively populating the malicious schema with actionable and structured harmful content. References [1] Zhiyuan Chang, Mingyang Li, Yi Liu, Junjie Wang, Qing Wang, and Yang Liu. 2024. Play guessing game with llm: Indirect jailbreak attack with implicit clues. arXiv preprint arXiv:2402.09091 (2024). [2] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym An- driushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Advances in Neural Information Processing Systems 37 (2024), 55005–55029. [3] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pap- pas, and Eric Wong. 2025. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 23–42. [4] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. Mul- tilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474 (2023). [5] Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. 2023. A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily. arXiv preprint arXiv:2311.08268 (2023). [6] Y Kumar, C Paredes, G Yang, J Li, and P Morreale. 2024. Adversarial Testing of LLMs Across Multiple Languages. In 2024 International Symposium on Networks, Computers and Communications (ISNCC). IEEE, 1–6. [7] Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. 2024. A cross-language investigation into jailbreak attacks in large language models. arXiv preprint arXiv:2401.16765 (2024). [8] Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. 2024. Drattack: Prompt decomposition and reconstruction makes powerful llm jail- breakers. arXiv preprint arXiv:2402.16914 (2024). [9] Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191 (2023). [10] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Gen- erating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451 (2023). [11] Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. 2024. Flipattack: Jailbreak llms via flipping. arXiv preprint arXiv:2410.02832 (2024). [12] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum An- derson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Sys- tems 37 (2024), 61065–61105. [13] Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. 2024. Codeattack: Revealing safety generalization challenges of large lan- guage models via code completion. arXiv preprint arXiv:2403.07865 (2024). [14] Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2025. Great, Now Write an Article About That: The Crescendo Multi-TurnLLM Jailbreak Attack. In 34th USENIX Security Symposium (USENIX Security 25). 2421–2440. [15] Yu-Hang Wu, Yu-Jie Xiong, Hao Zhang, Jia-Chen Zhang, and Zheng Zhou. 2025. Sugar-coated poison: Benign generation unlocks llm jailbreaking. arXiv preprint arXiv:2504.05652 (2025). [16] Hao Yang, Lizhen Qu, Ehsan Shareghi, and Gholamreza Haffari. 2024. Jigsaw puzzles: Splitting harmful questions to jailbreak large language models. arXiv preprint arXiv:2410.11459 (2024). [17] Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446 (2023). [18] Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253 (2023). [19] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 14322–14350. [20] Shuoming Zhang, Jiacheng Zhao, Ruiyuan Xu, Xiaobing Feng, and Huimin Cui. 2025. Output constraints as attack surface: Exploiting structured generation to bypass llm safety mechanisms. arXiv preprint arXiv:2503.24191 (2025). [21] Tianrong Zhang, Bochuan Cao, Yuanpu Cao, Lu Lin, Prasenjit Mitra, and Jinghui Chen. 2024. Wordgame: Efficient & effective llm jailbreak via simultaneous ob- fuscation in query and response. arXiv preprint arXiv:2405.14023 (2024). [22] Zaibin Zhang, Yongting Zhang, Lijun Li, Hongzhi Gao, Lijun Wang, Huchuan Lu, Feng Zhao, Yu Qiao, and Jing Shao. 2024. Psysafe: A comprehensive framework for psychological-based attack, defense, and evaluation of multi-agent system safety. arXiv preprint arXiv:2401.11880 (2024). [23] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned lan- guage models. arXiv preprint arXiv:2307.15043 (2023). Amirkia Rafiei Oskooei and Mehmet S. Aktas Figure 12: The full synthetic borderline-benign dataset (N=122) used to evaluate the False Positive Rate (FPR) of our guardrail. These tasks were injected into our BreakFun templates to create a challenging and realistic hard negative evaluation set. BreakFun: Jailbreaking LLMs via Schema Exploitation Figure 13: The full synthetic genuine benign dataset (N=122) used to evaluate the False Positive Rate (FPR) of our guardrail. These tasks were injected into our BreakFun templates to create a baseline for precision evaluation on typical user queries (easy negative set). Amirkia Rafiei Oskooei and Mehmet S. Aktas Figure 14: The zero-shot control prompt used to evaluate the intrinsic safety of the guardrail backbone models (Gemma3, Ministral, Qwen3) without the Adversarial Prompt Deconstruction (APD) defense. This serves as the baseline for the defense ablation study. BreakFun: Jailbreaking LLMs via Schema Exploitation Figure 15: Censored qualitative examples of successful BreakFun jailbreaks. The top panel shows a model complying with a request to create a bomb by filling the "Materials" and "Procedure" fields of the Trojan Schema. The bottom panel shows a model generating a detailed plan for ethnic oppression within the "Project" and "Step" classes. Sensitive terms have been redacted (X) to prevent misuse.

Instant research discovery