Paper deep dive
Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers
Giacomo Bertollo, Naz Bodemir, Jonah Burgess
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/11/2026, 1:28:49 AM
Summary
This paper analyzes the 'ai_gon3_rogu3' AI red-teaming CTF, involving 504 participants, to understand adversarial tactics and defense efficacy. Findings indicate that while simple guardrail bypasses are nearly universal, multi-step defenses create significant bottlenecks. The study highlights a sharp skill stratification among participants and notes that output manipulation tasks are generally easier than data extraction, providing empirical data to inform AI security governance and defense design.
Entities (5)
Relation Signals (2)
ai_gon3_rogu3 â mappedto â OWASP Top 10 for LLM Applications
confidence 90% · We connect our observations to risk taxonomies, OWASP Topâ10 for LLM Applications [8]
ai_gon3_rogu3 â mappedto â MITRE ATLAS
confidence 90% · We connect our observations to risk taxonomies... and MITRE ATLAS [9]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Analyzing 500 CTF participants, this paper shows that while participants readily bypassed simple AI guardrails using common techniques, layered multi-step defenses still posed significant challenges, offering concrete insights for building safer AI systems.
Tags
Links
- Source: https://arxiv.org/abs/2510.16005
- Canonical: https://arxiv.org/abs/2510.16005
Trouble viewing inline? Open PDF directly â
Full Text
23,544 characters extracted from source content.
Expand or collapse full text
Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers Naz Bozdemir, Giacomo Bertollo, Jonah Burgess Abstract â ai_gon3_rogu3 was a 10âday AI redâteaming CTF coârun by HackerOne and Hack The Box in September 2025. Of the 504 registrants, 217 were active and attempted 11 jeopardyâstyle challenges spanning output manipulation and dataâexfiltration scenarios. We analyze solve rates, engagement patterns, and tactic archetypes to understand where participants succeed and where defenses hold. Simple guardrail bypasses were nearly universal, whereas multiâstep defenses created pronounced bottlenecks: the introductory task reached ~98% completion among active players, but only ~34% solved the final scenario. Output manipulation challenges saw higher success (â82.5%) than dataâextraction tasks (â74.6%), suggesting âsocialâengineering the modelâ remains easier than extracting protected content at scale. Our findings offer a fieldâbased complement to red teaming benchmarks, clarifying which attack families dominate in competitive settings and where resilient defenses still frustrate skilled adversaries. Introduction AI red teaming brings security thinking to LLM applications by probing failure modes such as prompt injection, output manipulation, and sensitiveâdata exfiltration. While automated and curated benchmarks (e.g., JailbreakBench [1], HarmBench [2]) are increasingly used to test models and defenses, comparatively fewer studies analyze communityâscale behavior in the wild. We study ai_gon3_rogu3 [3], a 10âday competition with 504 registrants and 217 active players, to quantify solve dynamics, tactic stratification, and choke points across 11 challenges. We find sharp skill stratification, higher success for output manipulation than for data extraction, and strong effects of formatâobfuscation tactics, with multiâstep defenses remaining robust, among other insights. 1. Sharp skill stratification, nearly all active players solved the easiest challenge, but only a third managed the final scenario. 2. Output manipulation proved easier than data extraction, highlighting playersâ intuitive grasp of prompt-based misclassification. 3. Common adversarial tactics remain highly effective, especially encoding tricks and format obfuscation. 4. A distinct skills gap exists, separating a core of elite players from a broad long tail of less experienced participants. 1 This study gives researchers and defenders a data-driven view of how adversaries actually probe and break AI systems. It shows which attack strategies work and how quickly they spread in a competitive setting. For the CTF community, the results reveal that simple exploits were rapidly solved by nearly everyone, while more complex defenses created choke points. Only a small group of highly skilled players managed to push through those barriers, underscoring where real difficulty lies in securing AI systems. Background and Related Work Human and automated red teaming. Early work formalized LLMâvsâLLM [4] red teaming and showed that auxiliary models can uncover safety failures at scale. More recent efforts introduced open robustness benchmarks such as JailbreakBench [1] and HarmBench [2] that standardize behaviors, threat models, and scoring. Mining inâtheâwild jailbreaks (e.g., WildTeaming) [5] complements these curated sets, and manyâshot jailbreaking [6] highlights longâcontext vulnerabilities. Prompt injection & agent threats. Indirect prompt injection [7] in toolâintegrated agents expands the attack surface beyond simple chats; recent benchmarks (e.g., InjecAgent) systematize such scenarios. We connect our observations to risk taxonomies, OWASP Topâ10 for LLM Applications [8], and MITRE ATLAS [9], to bridge CTF behaviors with defender frameworks. CTFs and AI security skill evaluation. Emerging attempts involve LLMs solving CTFs [12â14], LLMs competing against humans [11], and training LLM agents in CTFâlike environments [22,23], but less is known about human solver dynamics in AI redâteaming CTFs. Our results fill this gap, providing baseline participation and solvingârate distributions to inform the design of layered defenses and training pathways. External testing complements CTFs. AIâfocused bug bounty programs and external testing are already surfacing jailbreaks [11, 16] and unsafe behaviors in deployed systems. Recent work argues that structured, thirdâparty adversarial evaluations [15] should be part of model governance, and that external incentives accelerate disclosure and remediation, while classifier-based defenses offer a complementary mitigation layer [15, 16]. We view AI red teaming, CTFs, and bug bounty programs as complementary modes of scrutiny across the AI system lifecycle. Bug bounties and coordinated disclosure programs surface jailbreaks and unsafe behaviors in deployed systems, while CTFs reveal skill distributions, emergent tactics, and the diffusion of exploit patterns within structured environments. AI red teaming acts as a structured, adversarial stress test across models and configurations, probing guardrails, emergent behaviors, and exploits [17, 20], bridging the gap between bench tests and live system vulnerabilities. By combining all three, defenders can maximize coverage (breadth), depth (scenario variety), and realism (deployment fidelity) in AI security evaluation. Methodology We analyzed detailed logs from the CTF platform, including every solved challenge, interaction attempt, and challenge instance launch. Our dataset: 2 â Participants: 504 registered, with 217 active players (defined as solving â„1 challenge). â Metrics: Challenge solves, completion rates across registered vs. active participants, and interaction counts (as a proxy for effort). The dataset includes every challenge's own (flag capture) record, as well as logs of challenge instance usage. Each record contains a timestamp, user and team identifiers, and the challenge involved. For each challenge, we computed the number of solves (teams that captured the flag) and the completion rate among players, both as a percentage of all registered players and of active players. This is reported as âpercentage ownershipâ in CTF terms, i.e. the fraction of teams that âownedâ (solved) the challenge. Such metrics help normalize difficulty independent of the drop-outs. We also measured engagement through the count of challenge instances launched as a proxy for attempts, and the distribution of solves per player. Finally, we categorized challenges by type (e.g. âdata extractionâ vs. âoutput manipulationâ tasks) based on their scenario and the primary skill required, to compare performance across challenge categories. As an example, Re-Cars AI (the introductory challenge) had 213 solves, equating to 42.3% of all registered teams and 98.2% of active teams. In contrast, the final challenge Performance Crossroads saw 74 solves â only 14.7% of all teams (34.1% of active teams). We tabulated such percentages for every challenge. Across all users, we logged 2,116 environment launches and 1,772 successful flag submissions, implying ~16% of sessions did not yield a solution (344/2,116 â 0.163), or about one in six. By dividing the total launches by the number of active players (2,116/217), we find an average of ~9.7 instances launched per active player, indicating that most participants attempted nearly all challenges and/or retried some challenges multiple times. Similarly, the median number of challenges solved per active player was 10 (out of 11), reflecting that many who solved one challenge went on to solve many more (the competition had a high retention among engaged players, as shown later). The next sections will illustrate the findings. 2.1 Ethics All analyses were conducted in accordance with the eventâs official terms of participation and applicable data protection principles. The dataset used in this study contained no personally identifiable information (PII); all participant identifiers were anonymized prior to analysis. Results are presented exclusively in aggregate form to prevent any inference about individual participants. The studyâs objective was limited to understanding player performance trends and aggregate behavioral patterns within the context of AI red teaming, ensuring adherence to ethical research standards and respect for participant privacy. 2.2 Challengeâauthoring caveats (LLMâspecific) LLM CTF tasks are inherently stochastic and pathâdiverse [18]. Because model behavior, tool integrations, and guardrails can be nonâdeterministic, an âintendedâ solution may not always trigger consistently, and preventing clever unintended solutions is difficult. In our event, some teams solved the final challenge without using the stored âreviewâ feature; an 3 acceptable outcome because we graded outcomes (safety or dataâaccess property violations), not a single canonical path. This motivates future work on challenge design patterns that are robust to model variability. Results 3.1 Participation and Engagement Out of 504 registrants, 217 were active (â 43%). This left a majority (â 56%) who never solved a single challenge (Figure 1); a common CTF phenomenon. Active participants, however, engaged deeply: many achieved their first solve quickly, as shown in Figure 2, and over half went on to solve at least 10 challenges. Notably, one-third completed all 11 tasks, marking a high-performing elite (Figure 8). Figure 1. Participant Engagement Overview â Distribution of registered vs. active participants. This funnel chart illustrates participant progression throughout the event. Out of 504 registrants, 217 achieved at least one solve, and 72 completed all 11 challenges. The steep early drop-off indicates a moderate entry barrier in AI red teaming tasks, yet strong retention among active solvers. Figure 2. Time to First Solve per Active Player â Distribution of how quickly participants achieved their initial success. This figure shows the distribution of time taken by participants to achieve their first challenge solution. The distribution shows a large >24h group (n=68) alongside a substantial early-solve cohort (<1h: n=39; 2â6h: n=39). Rather than strictly bimodal, the 4 pattern reflects a heavy right tail. This suggests a strong divide between experienced AI security practitioners and newcomers adapting to red teaming methodologies. Unless otherwise stated, time to first solve (Figure 3) is measured from a teamâs first challengeâinstance launch to its first recorded solve (removing timezone/availability effects). Using this baseline, the median timeâtoâfirstâsolve was ~0.32 hours (IQR 0.16â1.21 h). For context, measuring from CTF start yields a median of 13.6 h (IQR 2.11â36.68 h) with a heavy >24 h tailâreflecting participation timing rather than problemâsolving latency. Figure 3. TimeâtoâFirstâSolve Distributions â Grouped bars compare counts of active teams (n=217) across time bins measured from CTF start vs. from first instance launch. Bins <1 h, 1â2 h, 2â6 h, 6â12 h, 12â24 h, >24 h have counts: â CTF start â first solve: 40, 11, 37, 18, 43, 68 â First launch â first solve: 158, 15, 21, 5, 7, 11 The CTFâstart baseline reflects availability/timezone effects, whereas the firstâlaunch baseline isolates onâtask latency. 5 Figure 4. Challenge Solves Over Time by Challenge â Cumulative solves for each challenge across the event window. Early momentum is highest in the first 48 hours, followed by a gradual plateau. The top-performing challenges (Re-Cars AI and Leaks Premium) reached saturation quickly, indicating they served as effective onboarding tasks for participants. 3.2 Challenge Completion Rates Figure 5 illustrates the number of teams that solved each challenge, in descending order of solves. The easiest challenge, Re-Cars AI, was solved by 213 teams, whereas the hardest regular challenge (prior to the final), GATE Chat, saw 133 solutions. The final boss, Performance Crossroads, was solved by 74 teams. When considering all registered teams, the drop-off is starker: only 14.7% of all teams achieved the final flag, and 26.4% solved GATE Chat, compared to 42.3% who solved the first challenge. Table 1. Challenge Completion Statistics 6 Figure 5. Solve Rates by Challenge Difficulty â Decline from 98.6% on the easiest challenge to 34.1% on the final boss. This summary table ranks challenges by completion rate and ownership percentage among teams. Re-Cars AI had the highest completion (42.7% of all teams), while Performance Crossroads proved the most difficult (14.8%). These results highlight the balance between accessibility and technical depth across the CTF design. Among active teams (n = 217), 170 (78.3%) launched the final challenge (Figure 6); 74 (34.1%) solved it; and 96 (44.2%) launched but did not solve. 47 (21.7%) of active teams never attempted the final at all. Progress tiering explains this: every team with â„10 solves (124/124) attempted the final and 73 (58.9%) solved it, whereas among teams with <10 solves (n = 93), only 46 (49.5%) attempted the final and 1 (1.1%) solved it. 7 Figure 6. FinalâChallenge Attempts by Experience Tier (Active Teams) â Stacked bars show, for teams with â„10 solves (n=124), <10 solves (n=93), and all active teams (n=217): Solved final (73, 1, 74), Launched but not solved (51, 45, 96), and Never launched final (0, 47, 47). These counts separate engagement (launch) from success (solve) on the final challenge. 3.3 Category-wise Performance Challenges fell into two broad categories: â Output Manipulation (2 challenges) â average success: 82.5% of active players. â Data Extraction (9 challenges) â average success: 74.6%. Figure 7. Category-wise Performance â Comparison of Output Manipulation vs. Data Extraction success rates. This figure compares solve rates across two major challenge categories â Output Manipulation and Data Extraction. Participants performed better in manipulation-oriented tasks (82.5%) than in extraction-based ones (74.6%), suggesting greater familiarity with controlling model outputs than exploiting systemic data vulnerabilities. 8 These two broad categories map onto several OWASP LLM Top 10 risks. Output Manipulation challenges align with LLM01: Prompt Injection and LLM09: Overreliance, since they demonstrate how crafted inputs can cause unsafe misclassifications that organizations may trust blindly. Data Extraction challenges map to LLM01: Prompt Injection, LLM02: Data Leakage, and LLM06: Sensitive Information Disclosure; in the case of complex scenarios like Performance Crossroads, they also touch on LLM05: Excessive Agency. Challenge Themes To better contextualize the adversarial behaviors observed in the CTF, we mapped each challenge theme against the OWASP Top 10 for LLM Applications [8] and MITREâs ATLAS framework [9](Table 2). Table 2. Mapping CTF Challenge Themes to OWASP LLM Top 10 and MITRE ATLAS CTF Theme Challenges Description OWASP LLM Top 10 Category MITRE ATLAS Mapping Basic Data Access Bypass Re-Cars, CruiseList Prompt injection to override chatbot restrictions and access hidden data LLM01: Prompt Injection Evasion / Jailbreak â manipulating input to bypass model restrictions Gatekeeper Exploits GrandMonty, Auction Chat Tricking an AI into revealing authentication or secret codes LLM01: Prompt Injection + LLM04: Insecure Output Handling Role Play / Jailbreak â tricking the model into acting outside its role Privilege Escalation Stock Signals, Leaks Premium Extracting paywalled or premium data via crafted prompts LLM06: Sensitive Information Disclosure Model Extraction / Confidentiality Violation â eliciting data beyond authorization Output Manipulation AllSafe Tickets, Cosmic Crisis Forcing AI misclassification (e.g., escalating ticket priority) LLM04: Insecure Output Handling; LLM09: Overreliance Output Manipulation / Misclassification â causing the model to produce incorrect or unsafe outputs 9 Guardrail Evasion GATE Chat, Pump Signals AI Defeating censorship/filters to reveal prohibited or hidden content LLM01: Prompt Injection + LLM02: Data Leakage Jailbreak / Censorship Evasion â defeating safety filters to access blocked knowledge Complex Multi-Step Exploit Performance Crossroads Overriding role-based access controls to extract private performance reviews LLM05: Excessive Agency + LLM06: Sensitive Information Disclosure Multi-Stage Attack / Identity Manipulation â chaining prompts to override role-based access and exfiltrate data Each of these challenges highlighted different facets of AI security. From the summaries above, we see recurring patterns: many tasks required participants to bypass content filters or access controls (no direct code exploitation, since the âvulnerabilitiesâ were in the AIâs training or prompt rules). The high success rates on simpler scenarios show that techniques like formatting a query differently or using out-of-the-box phrasing are widely known and effective. Harder scenarios layered more complex guardrails (e.g., multi-turn requirements, hidden triggers), where fewer participants succeeded. Discussion: Key Insights Insight 1: A Sharp Difficulty Cliff From nearly universal solves on the opening task to just one-third on the final boss (Figures 4 and 5), the data shows how quickly difficulty stratifies the field. This demonstrates that while baseline exploitation skills are widespread, mastering complex multi-step attacks is rare. Insight 2: Social Engineering the Machine Players succeeded more often when tricking the AI into misjudging context than when extracting secrets (Figure 7). This reinforces the notion that AI security must address manipulation just as much as exfiltration risks. For educators and tool developers, this emphasizes focusing on multi-step and chained exploit training to elevate more people to that elite level. Insight 3: Format Obfuscation Works â Too Well Tactics like encoding requests in JSON or base64 reliably bypassed filters. This exposes a systemic weakness: too many AI guardrails rely on pattern recognition instead of deeper semantic defenses. The community may need more practice and techniques specifically for breaking AI security and privacy guardrails, as opposed to just tricking AIâs judgment. It also 10 implies that current AI guardrails can be moderately effective against a large fraction of adversaries, but not all [1, 2, 16]. Insight 4: A Bimodal Skill Distribution Roughly a third of players solved everything, while many solved very little (Figure 8). This gap highlights both the promise of a strong talent pool and the need for more educational on-ramps to close the divide. However, the long tail of partial completions highlights varied expertise levels in AI exploitation and prompt manipulation. Figure 8. Skill Distribution Across Participants â Bimodal performance split between elite solvers and the long tail. This chart visualizes the number of challenges solved per player. The distribution is left-skewed: many participants solved all 11 challenges (n=72), with a long tail of partial completions. Conclusion The ai_gon3_rogu3 CTF revealed a community highly capable at breaking AI guardrails using known techniques but less prepared for multi-step, context-rich defenses. For defenders, the results mean that simple safeguards are trivial to bypass, but layered, adaptive protections still frustrate even skilled players, mimicking real-world attackers. For researchers, the event provides concrete data on attacker tendencies, informing the design of safer AI systems. And for the CTF community, it affirms that adversarial AI security is not just a niche interest â itâs a growing frontier where skills honed today will be critical tomorrow. References [[1] P. Chao et al., âJailbreakBench: An open robustness benchmark for jailbreaking large language models,â arXiv preprint arXiv:2404.01318, 2024. [2] M. Mazeika et al., âHarmBench: A standardized evaluation framework for automated red teaming and robust refusal,â arXiv preprint arXiv:2402.04249, 2024. 11 [3] N. Bozdemir and G. Bertollo, âLLMs adversarial testing with HackerOne: Join the CTF!,â HackerOne Blog, Aug. 19, 2025. [4] E. Perez et al., âRed teaming language models with language models,â arXiv preprint arXiv:2202.03286, 2022. [5] L. Jiang et al., âWildTeaming at scale: From inâtheâwild jailbreaks to (adversarially) safer language models,â arXiv preprint arXiv:2406.18510, 2024. [6] C. Anil et al., âManyâshot jailbreaking,â in Advances in Neural Information Processing Systems (NeurIPS), 2024. [7] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, âInjecAgent: Benchmarking indirect prompt injections in toolâintegrated large language model agents,â arXiv preprint arXiv:2403.02691, 2024. [8] OWASP Foundation, âOWASP Top 10 for Large Language Model Applications v2.0,â 2025. [Online]. Available: owasp.org (accessed Oct. 7, 2025). [9] MITRE Corporation, âAdversarial Threat Landscape for ArtificialâIntelligence Systems (ATLAS),â 2021. [Online]. Available: atlas.mitre.org (accessed Oct. 7, 2025). [10] Anthropic, âCyber competitions,â Anthropic Red Teaming, 2025. [Online]. Available: red.anthropic.com/2025/cyberâcompetitions/ (accessed Oct. 7, 2025). [11] Hack The Box, âAI vs human CTF: Results,â HTB Blog, 2024. [Online]. Available: hackthebox.com/blog/aiâvsâhumanâctfâhackâtheâboxâresults (accessed Oct. 7, 2025). [12] A. K. Zhang et al., âCYBENCH: A framework for evaluating cybersecurity capabilities and risks of language models,â arXiv preprint arXiv:2408.08926, 2024. [13] T. Y. Zhuo, D. Wang, H. Ding, V. Kumar, and Z. Wang, âTraining language model agents to find vulnerabilities with CTFâDojo,â arXiv preprint arXiv:2508.18370, 2025. [14] Z. Ji, D. Wu, W. Jiang, P. Ma, Z. Li, and S. Wang, âMeasuring and augmenting large language models for solving captureâtheâflag challenges,â arXiv preprint arXiv:2506.17644, 2025. [15] S. Longpre et al., âInâhouse evaluation is not enough: Towards robust thirdâparty flaw disclosure for generalâpurpose AI,â arXiv preprint arXiv:2503.16861, 2025. [16] M. Sharma et al., âConstitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming,â arXiv preprint arXiv:2501.18837, 2025. [17] M. Feffer, A. Sinha, W. H. Deng, Z. C. Lipton, and H. Heidari, âRed-teaming for generative AI: Silver bullet or security theater?,â in Proc. AAAI/ACM Conf. on AI, Ethics, and Society (AIES), vol. 7, p. 421â437, 2024. [18] DARPA, âAI Cyber Challenge (AIxCC) â About,â 2024. [Online]. Available: aicyberchallenge.com (accessed Oct. 7, 2025). 12 [19] P. V. Charan, H. Chunduri, P. M. Anand, and S. K. Shukla, âFrom text to MITRE techniques: Exploring the malicious use of large language models for generating cyberâattack payloads,â arXiv preprint arXiv:2305.15336, 2023. [20] A. Sinha, K. Grimes, J. Lucassen, M. Feffer, N. Van Houdnos, Z. S. Wu, and H. Heidari, âFrom Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teaming,â arXiv preprint arXiv:2509.11398, 2025. [21] N. Carlini, J. Rando, E. Debenedetti, M. Nasr, and F. TramĂšr, âAutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defenses,â arXiv preprint arXiv:2503.01811, 2025. [22] G. Deng et al., âPentestGPT: Evaluating and harnessing large language models for automated penetration testing,â in Proc. 33rd USENIX Security Symposium (USENIX Security â24), 2024. [23] R. Fang, R. Bindu, A. Gupta, Q. Zhan, and D. Kang, âLLM agents can autonomously hack websites,â arXiv preprint arXiv:2402.06664, 2024. 13