Paper deep dive
Adversarial Prompting Framework for AI Safety Assessment
Yash Bhatnagar, Kunal Banerjee, Anirban Chatterjee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/18/2026, 10:20:09 AM
Summary
This paper introduces the Adversarial Prompting Framework (APF), a systematic methodology for assessing AI safety against Adversarial Prompt Attacks (APA). The framework classifies attacks into five sophistication levels, from direct requests to sophisticated jailbreaking, and evaluates model resilience using automated metrics like toxicity and sentiment analysis. Experimental results across various proprietary and open-source models reveal that while basic filters are effective, sophisticated encoding and role-playing attacks significantly bypass safety mechanisms, with Claude showing the highest resilience and open-source models like Llama and Mistral showing greater susceptibility.
Entities (12)
Relation Signals (11)
Claude → demonstrates → high resilience
confidence 96% · Claude models: They consistently demonstrate the highest level of resistance, registering the lowest vulnerability scores across nearly all attack categories.
Adversarial Prompting Framework → assesses → Adversarial Prompt Attack
confidence 95% · This paper presents the implementation of an Adversarial Prompting Framework (APF) for a comprehensive assessment of AI safety... adversarial prompt attack (APA) being one of the most prominent examples of such threats.
Mistral → shows → greater susceptibility
confidence 95% · Open source models (Llama, Mistral): In contrast, leading open source models such as Llama and Mistral show greater susceptibility to a broad range of APAs.
Llama → shows → greater susceptibility
confidence 95% · Open source models (Llama, Mistral): In contrast, leading open source models such as Llama and Mistral show greater susceptibility to a broad range of APAs.
GPT → shows → measurable vulnerability
confidence 94% · GPT models: ...they show measurable vulnerability to complex, multi-layered attacks, particularly those that employ encoding, character substitution, or role-play combined with contextual misdirection.
Adversarial Prompting Framework → classifies → Encoding and obfuscation techniques
confidence 93% · (4)Encoding and obfuscation techniques: Advanced attacks utilizing various encoding methods
Adversarial Prompting Framework → classifies →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Artificial Intelligence (AI), especially Generative AI (GenAI), adoption has increased in industries significantly in recent years. However, the use of these models may also expose systems to new forms of cyberattacks by different malicious actors -- adversarial prompt attack (APA) being one of the most prominent examples of such threats. This paper presents the implementation of an Adversarial Prompting Framework (APF) for a comprehensive assessment of AI safety. The framework systematically evaluates the resilience of the AI model through the generation of structured adversarial prompts at multiple sophistication levels, from direct harmful requests to advanced encoding-based attacks. Our implementation demonstrates the practical application of this methodology in enterprise environments, providing automated testing capabilities with quantitative security assessment metrics. The results indicate significant variations in the model vulnerabilities across different attack vectors, with encoded prompts presenting the highest success rates in bypassing safety mechanisms.
Tags
Links
- Source: https://arxiv.org/abs/2607.13453v1
- Canonical: https://arxiv.org/abs/2607.13453v1
Trouble viewing inline? Open PDF directly →
Full Text
14,524 characters extracted from source content.
Expand or collapse full text
Adversarial Prompting Framework for AI Safety Assessment Yash Bhatnagar ∗† ybhatnagar@microsoft.com Microsoft Bengaluru, Karnataka, India Kunal Banerjee † kunal.banerjee1@walmart.com Walmart Global Tech Bengaluru, Karnataka, India Anirban Chatterjee † anirban.chatterjee@walmart.com Walmart Global Tech Bengaluru, Karnataka, India Abstract Artificial Intelligence (AI), especially Generative AI (GenAI), adop- tion has increased in industries significantly in recent years. How- ever, the use of these models may also expose systems to new forms of cyberattacks by different malicious actors – adversarial prompt attack (APA) being one of the most prominent examples of such threats. This paper presents the implementation of an Adversarial Prompting Framework (APF) for a comprehensive assessment of AI safety. The framework systematically evaluates the resilience of the AI model through the generation of structured adversarial prompts at multiple sophistication levels, from direct harmful re- quests to advanced encoding-based attacks. Our implementation demonstrates the practical application of this methodology in enter- prise environments, providing automated testing capabilities with quantitative security assessment metrics. The results indicate signif- icant variations in the model vulnerabilities across different attack vectors, with encoded prompts presenting the highest success rates in bypassing safety mechanisms. Keywords Adversarial prompting, AI safety, Security testing, Model evaluation 1 Introduction According to this study [13], AI adoption has increased from 17% in 2023 to 72% in the beginning of 2025. Interestingly, after investi- gating, IBM found that only 24% of GenAI projects are secured [3]. Of all the risks that these models are susceptible to, the adversarial prompt attack (APA) appears to be the main one [5,11]. Although se- curity against APAs has been addressed in the literature [1,4,9,10], no unified approach has emerged yet to tackle this problem. There- fore, we have designed our Adversarial Prompting Framework (APF) that represents a systematic approach to evaluating AI model safety through structured testing methodologies. As AI systems become increasingly integrated into critical applications, the need for com- prehensive security assessment has become paramount. This frame- work addresses the challenge of systematically evaluating model resilience against various forms of adversarial inputs designed to elicit harmful or inappropriate responses. The APF’s structured approach enables reproducible testing across different model ar- chitectures and deployment scenarios. By categorizing adversarial techniques into specific sophistication levels, researchers and prac- titioners can systematically evaluate model vulnerabilities and track improvements in safety mechanisms over time. Our contributions include: (i) an adversarial prompt classification, (i) an APF that ∗ Work done while the author was an employee of Walmart Global Tech. † This is an independent research done by the authors and not endorsed by Walmart Global Tech in any manner. This work was presented as a poster at International Conference on Data Science (CODS), December 17–20, 2025, Pune, India. includes automated adversarial prompt generation and response evaluation, and (i) an extensive experimentation carried out on several proprietary and open source foundation models. 2 Adversarial Prompt Classification Prior research [6,14,17] has proposed various classification tech- niques for APAs. However, we observed that these earlier tax- onomies fail to adequately capture the most common types of APAs encountered in practice. To address this gap, we introduce a frame- work that organizes APAs into five progressive levels, with each successive class exhibiting greater ability to exploit vulnerabilities in GenAI systems: (1)Direct adversarial prompts: These represent straightfor- ward attempts to elicit harmful content without obfuscation or misdirection; examples include direct requests for illegal information or explicit harmful instructions. (2) Contextual role-playing: This level introduces persona- based attacks where the adversarial request is framed within a specific role or context. The attacker assumes a character or professional role to legitimize the harmful request. (3)Multi-step instruction sequences: Complex prompts that break down harmful requests into seemingly inno- cent components, often including explicit instructions to ignore safety guidelines or ethical considerations. (4)Encoding and obfuscation techniques: Advanced at- tacks utilizing various encoding methods to obscure the true intent of the request such as, Caesar cipher encoding, leetspeak substitution, unicode character encoding, hexa- decimal representation, text reversal techniques. (5)Sophisticated jailbreaking attempts: This highest so- phistication level combines multiple techniques, includ- ing instruction override attempts, safety protocol bypasses, and complex encoding schemes designed to circumvent advanced safety mechanisms. In relation to existing work, our five classes can be mapped to either prompt rewriting or template completion under the broader family of black-box attacks described in [17]. Similarly, with respect to [6], they align with the categories of adversarial insertion, adversarial suffix or adversarial prefix attacks. 3 Adversarial Prompting Framework 3.1 Adversarial Prompt Generation Our prompt design is influenced by that of [12] 1 . Similar to [12], in the initial step, we choose a policy concept (e.g., theft, cybercrime, vandalism), a task format (e.g., email, news article, code) and a 1 Note that since the code of [12] is not publicly available, we reimplemented it from scratch with a few augmentations. We intend to release our implementation publicly once the approval process is complete. arXiv:2607.13453v1 [cs.CR] 15 Jul 2026 Bhatnagar et al. geographical region (e.g., Africa, Asia, Latin America). These are used as inputs to generate the adversarial prompt using Chain-of- Thought (CoT)-style instructions [15]. Having followed a structured procedure for APA generation, each prompt is tagged with the diversity dimensions identified earlier – this makes it possible to verify the coverage of the generated prompts without involving a human-in-the-loop process. 3.2 Response Evaluation To assess how effective an APA푎is for a given model푀, we check for toxicity, sentiment and presence of black-listed words in the re- sponse generated by 푀 with 푎 as input. We use detoxify [7] for measuring toxicity, the model distilbert- sst2-sentiment [8] for sentiment analyis, and a known list of harmful (black-listed) words. The score given to APA푎for model푀is computed using the following formula: 푠푐표푟푒= 휆 1 ∗ 푡표푥푖푐푖푡푦+ 휆 2 ∗ 푠푒푛푡푖푚푒푛푡 + 휆 3 ∗ 푘푒푦푤표푟푑 where the values for휆 1 , 휆 2 , 휆 3 are determined by training a super- vised regression model on a manually labeled data with the inputs normalized. Note that if a model is able to recognise the harmful intent of a prompt and consequently, refuses to answer it, then we give it a푠푐표푟푒of 0. The final score for a model is derived by aggregating its score across all APAs and then taking their mean. 4 Experimental Results Figure 1: Comparative analysis of various foundational mod- els against APAs; less score indicates more resilience. We first generate a thousand adversarial prompts that cover ∼25 harmful policies including drug trafficking, money laundering, cybercrime, violence, social engineering. We feed these thousand prompts to several GenAI models from Google, OpenAI, Anthropic, Meta, Mistral, DeepSeek and more. Figure 1 provides a concise summary of our comparative analysis. We classify the models into four safety levels – Excellent, Good, Moderate, and Concerning – using empirically chosen thresholds. This categorical approach reflects our experience that consumers often prefer clear labels over raw numerical scores. Determining such thresholds in a more principled manner, however, remains an open challenge. The key highlights from our experiments are mentioned below: •Claude models: They consistently demonstrate the high- est level of resistance, registering the lowest vulnerability scores across nearly all attack categories. This suggests that their safety guardrails are more robust and effective in preventing both direct and obfuscated adversarial attempts. •Open source models (Llama, Mistral): In contrast, lead- ing open source models such as Llama and Mistral show greater susceptibility to a broad range of APAs. While they manage basic filtering reasonably well, their safety align- ment tends to weaken against more sophisticated strategies including obfuscation and multi-step role-based attacks For bigger and more recent open source models though, like Llama-3.1-405B and Mistral-Large-2, vulnerability towards obfuscation was greatly reduced. •Specialized models (e.g., CodeLlama): Domain-specific models exhibit unique vulnerabilities. For example, CodeL- lama, while optimized for coding-related tasks, is more prone to roleplay-oriented jailbreaks and adversarial prompts that exploit its domain expertise (e.g., requests framed as debugging or system instructions). •GPT models: These models display strong defenses against straightforward adversarial prompts, with highly effective initial filtering layers. However, they show measurable vul- nerability to complex, multi-layered attacks, particularly those that employ encoding, character substitution, or role- play combined with contextual misdirection. •Gemini models: Vulnerability across all Gemini models was similar regarding encoding attacks, even in the newer models. To summarize, across the board, current AI safety measures can reliably deflect basic or single-layered adversarial attacks. However, they remain significantly less effective against sophisticated, multi- dimensional strategies. The combination of encoding techniques (e.g., leetspeak, unicode, cipher-based obfuscation) with contextual role-play represents the most potent threat vector today, as it effec- tively bypasses traditional safety filters. This highlights the urgent need for next-generation defense mechanisms that integrate not only static prompt filtering but also dynamic context understanding and adaptive adversarial detection. 5 Conclusion APAs represent a significant challenge in the industrial adoption of GenAI models. While several mitigation strategies have been proposed, a standardized solution has yet to emerge. To address this, we introduce a capability-based taxonomy that classifies APAs into five levels, reflecting their increasing potential to exploit GenAI models. Such a structured categorization provides a simpler and more systematic way to assess defense mechanisms compared to other complex approaches [2]. In addition, we present a multi- dimensional evaluation framework through which we benchmark and rank multiple proprietary and open source foundation models. As industries increasingly employ agentic AI and integrate GenAI Adversarial Prompting Framework for AI Safety Assessment into cybersecurity applications [16], understanding their vulnera- bilities becomes even more critical. Looking ahead, we intend to extend our research towards analyzing these composite systems. References [1]Sara Abdali, Richard Anarfi, C. J. Barberan, and Jia He. 2024. Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices. CoRR abs/2403.12503 (2024). [2]Rico Angell, Jannik Brinkmann, and He He. 2025. Jailbreak Strength and Model Similarity Predict Transferability. CoRR abs/2506.12913 (2025). [3] Annie Badman. 2025. What is AI risk management? https://w.ibm.com/ think/insights/ai-risk-management. [4]Luca Beurer-Kellner, Beat Buesser, Ana-Maria Cretu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, Ezinwanne Ozoani, Andrew Paverd, Florian Tramèr, and Václav Vol- hejn. 2025. Design Patterns for Securing LLM Agents against Prompt Injections. CoRR abs/2506.08837 (2025). [5]Jing Cui, Yishi Xu, Zhewei Huang, Shuchang Zhou, Jianbin Jiao, and Junge Zhang. 2024. Recent Advances in Attack and Defense Approaches of Large Language Models. CoRR abs/2409.03274 (2024). [6]Ahmet Emre Ergun and Aytug Onan. 2025. Adversarial Prompt Detection in Large Language Models: A Classification-Driven Approach. Computers, Materials & Continua 83, 3 (2025), 4855–4877. [7] Laura Hanu and Unitary team. 2020. Detoxify. https://github.com/unitaryai/ detoxify. [8] Tegar Kusuma. 2025. distilbert-sst2-sentiment. https://huggingface.co/tofuyaki/ distilbert-sst2-sentiment. [9]Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, Tyler Tracy, Buck Shlegeris, and Joe Benton. 2025. SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents. CoRR abs/2506.15740 (2025). [10]Miles Q. Li and Benjamin C. M. Fung. 2025. Security Concerns for Large Language Models: A Survey. CoRR abs/2505.18889 (2025). [11] OWASP. 2025. 2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps. https://genai.owasp.org/llm-top-10/. [12] Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti. 2023. AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM- powered Applications. In EMNLP. Association for Computational Linguistics, 380–395. [13] Alex Singla, Alexander Sukharevsky, Lareina Yee, Michael Chui, and Bryce Hall. 2025. The state of AI: How organizations are rewiring to capture value. https: //w.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai. [14]Kritin Vongthongsri. 2025. How to Jailbreak LLMs One Step at a Time: Top Tech- niques and Strategies. https://w.confident-ai.com/blog/how-to-jailbreak- llms-one-step-at-a-time. [15]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS. [16] Hanxiang Xu, Shenao Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and Haoyu Wang. 2024. Large Language Models for Cyber Security: A Systematic Literature Review. CoRR abs/2405.04760 (2024). [17]Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak Attacks and Defenses Against Large Language Models: A Survey. CoRR abs/2407.04295 (2024).