Paper deep dive
Adversarial Arena: Crowdsourcing Data Generation through Interactive Competition
Prasoon Goyal, Sattvik Sahai, Michael Johnston, Hangjie Shi, Yao Lu, Shaohua Liu, Anna Rumshisky, Rahul Gupta, Anna Gottardi, Desheng Zhang, Lavina Vaz, Leslie Ball, Lucy Hu, Luke Dai, Samyuth Sagi, Maureen Murray, Sankaranarayanan Ananthakrishnan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/27/2026, 9:28:12 AM
Summary
The paper introduces 'Adversarial Arena', a novel framework for high-quality synthetic data generation through interactive competition between 'attackers' (who create prompts) and 'defenders' (who generate responses). This approach addresses the limitations of traditional crowdsourcing and synthetic generation, such as low diversity and high cost. The framework was validated through a cybersecurity alignment case study involving 10 academic teams, resulting in 19,683 multi-turn conversations. Fine-tuning an open-source model on this data showed significant improvements in secure code generation on benchmarks like CyberSecEval-Instruct and CyberSecEval-MITRE.
Entities (10)
Relation Signals (8)
Adversarial Arena → appliedto → Cybersecurity Alignment
confidence 100% · We applied the proposed Adversarial Arena framework to the task of Cybersecurity Alignment
Adversarial Arena → appliedto → Cybersecurity Alignment
confidence 100% · We present a case study on one such task: cybersecurity alignment.
Amazon CodeGuru → detectsvulnerabilitiesin → Defender Response
confidence 100% · run a static code analyzer (specifically, Amazon CodeGuru 1) on it to determine if the code had any vulnerabilities.
Mistral-7B-Instruct → evaluatedon → CyberSecEval-Instruct
confidence 100% · Fine tuning the model on this dataset produced an 18.47% improvement in secure code generation on CyberSecEval-Instruct
Amazon CodeGuru → evaluates → Vulnerable Code
confidence 100% · run a static code analyzer (specifically, Amazon CodeGuru 1) on it to determine if the code had any vulnerabilities.
Adversarial Arena → uses → Defender
confidence 100% · attackers create prompts, and defenders generate responses.
Adversarial Arena → uses → Attacker
confidence 100% · attackers create prompts, and defenders generate responses.
Mistral-7B-Instruct → alignedby → Adversarial Arena Data
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Post-training Large Language Models requires diverse, high-quality data which is rare and costly to obtain, especially in low resource domains and for multi-turn conversations. Common solutions are crowdsourcing or synthetic generation, but both often yield low-quality or low-diversity data. We introduce Adversarial Arena for building high quality conversational datasets by framing data generation as an adversarial task: attackers create prompts, and defenders generate responses. This interactive competition between multiple teams naturally produces diverse and complex data. We validated this approach by conducting a competition with 10 academic teams from top US and European universities, each building attacker or defender bots. The competition, focused on safety alignment of LLMs in cybersecurity, generated 19,683 multi-turn conversations. Fine-tuning an open-source model on this dataset produced an 18.47% improvement in secure code generation on CyberSecEval-Instruct and 29.42% improvement on CyberSecEval-MITRE.
Tags
Links
- Source: https://arxiv.org/abs/2604.17803v1
- Canonical: https://arxiv.org/abs/2604.17803v1
Trouble viewing inline? Open PDF directly →
Full Text
78,693 characters extracted from source content.
Expand or collapse full text
Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. ADVERSARIAL ARENA:CROWDSOURCING DATA GENERATION THROUGH INTERACTIVE COMPETITION Prasoon Goyal ∗ , Sattvik Sahai ∗ , Michael Johnston, Hangjie Shi, Yao Lu, Shaohua Liu, Anna Rumshisky, Rahul Gupta, Anna Gottardi, Desheng Zhang, Lavina Vaz, Leslie Ball, Lucy Hu, Luke Dai, Samyuth Sagi, Maureen Murray & Sankaranarayanan Ananthakrishnan Amazon Nova Responsible AI ABSTRACT Post-training Large Language Models requires diverse, high-quality data which is rare and costly to obtain, especially in low resource domains and for multi- turn conversations. Common solutions are crowdsourcing or synthetic generation, but both often yield low-quality or low-diversity data. We introduce Adversarial Arena for building high quality conversational datasets by framing data generation as an adversarial task: attackers create prompts, and defenders generate responses. This interactive competition between multiple teams naturally produces diverse and complex data. We validated this approach by conducting a competition with 10 academic teams from top US and European universities, each building attacker or defender bots. The competition, focused on safety alignment of LLMs in cyber- security, generated 19,683 multi-turn conversations. Fine-tuning an open-source model on this dataset produced an 18.47% improvement in secure code generation on CyberSecEval-Instruct and 29.42% improvement on CyberSecEval-MITRE. 1INTRODUCTION Figure 1: Adversarial Arena Overview: Attacker/defender pairs interact over several tournament rounds, with each pair generating a multi-turn conversation in every round. These conversations are labeled as success/failure in an evaluation pipeline, producing a ranked list of attackers and defenders. The ranked list and the labeled conversations are provided to attackers and defenders in feedback loop to drive up the overall quality of the generated data. ∗ These authors contributed equally to this work. 1 arXiv:2604.17803v1 [cs.AI] 20 Apr 2026 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. As Large Language Models (LLMs) expanded their capabilities, the importance of high-quality task- appropriate data has become more and more apparent to the research community. Traditionally, data creation has involved significant human effort, including manual annotation (K ̈ opf et al., 2023), data filtering (Li et al., 2024; Penedo et al., 2023; 2024), and data augmentation(Ding et al., 2024). To add to that, during model training, human input in the form of interactive testing (AI @ Meta, 2024), feedback (Bai et al., 2022; Ouyang et al., 2022), and human evaluation (Chiang et al., 2024) is commonly required (Wu et al., 2022). A common approach to scale up human-generated data is crowdsourcing; however, these methods require careful design to obtain high-quality data (Vaughan, 2018). Recently, LLMs themselves have been used to generate synthetic training data at scale (Wang et al., 2022; 2023; Bercovich et al., 2025). While appealing, this approach suffers from important limita- tions. Synthetic data often lacks diversity and coverage, and it can amplify hidden biases, leading to degraded robustness and unwanted behaviors in downstream models (Cloud et al., 2025; Zur et al., 2025; Chen et al., 2024). Attempts to mitigate these issues rely on careful choices of models, prompts, and filtering strategies (Xu et al., 2025a; Wei et al., 2024). Yet the resulting design space is vast, highly sensitive to small decisions, and expensive to navigate effectively. We argue that overcoming these limitations requires a framework that supports structured, adversar- ial exploration of this design space. Therefore, we propose a novel framework Adversarial Arena, to collect synthetic data for tasks that can be formulated using an adversarial setting. As an example, consider the problem of hallucinations in LLMs. This problem can be formulated using an adversar- ial setting as follows: the attacker’s goal is to get the model to generate hallucinated content, while the defender’s goal is to make the model robust against such outputs. Our framework allows dif- ferent research groups to independently explore distinct regions of the design decision space while exchanging intermediate feedback, which leads to greater diversity and lower bias in the generated data. A key component of the Adversarial Arena is an orchestrator, which allows interaction between multiple attackers and defenders in a competitive environment, which we refer to as a “tournament”. We design a competition structure where these teams compete against each other in a series of tournaments. Through multiple rounds of competition, several desired outcomes are achieved. First, the setting naturally supports multi-turn interactions between attackers and defenders, producing data that is more realistic for many tasks. Second, each attacker/defender team develops a pipeline to generate data independently. While individual pipelines may reflect each team’s biases, the presence of multiple teams can help offset these biases. This diversity of perspectives increases coverage compared to data from each individual team. Third, the techniques developed by both sides to generate their data need to be robust against a range of diverse strategies employed by multiple opponents. In other words, the data and techniques developed by each attacker should work well against most defenders, and vice versa. Finally, teams use their experience from past tournaments to improve their approaches in future rounds, resulting in a flywheel effect, in which the teams produce progressively richer data and techniques over time. Our framework enables crowdsourcing data for any task that can be formulated in an adversarial setting. We present a case study on one such task: cybersecurity alignment. We organized a compe- tition, utilizing the Adversarial Arena platform, with ten leading universities from the United States and Europe. The universities were divided equally into five attackers and five defenders and they competed over four tournaments. The competition resulted in a dataset of 19683 labeled multi- turn conversations. We show that the data generated by our framework is effective at aligning an open weight Mistral 7b Instruct (Jiang et al., 2023) model. Fine tuning the model on data from the competition resulted in an 18.47% improvement in secure code generation on the CyberSecEval- Instruct benchmark (Bhatt et al., 2023), and 29.42% improvement on the CyberSecEval-MITRE benchmark (Bhatt et al., 2023). We also provides evidence that having multiple teams leads to a “diversity of perspectives”, as reflected in the semantic separation between datasets generated by different teams. Datasets collected across tournament rounds likewise show this diversity of per- spectives, demonstrating that recurring adversarial tournaments generate richer data over time. The resulting datasets will be released upon publication. Our contributions can be summarized as follows. 2 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. 1. We present Adversarial Arena, a novel framework that enables crowdsourcing of synthetic data through adversarial interactions between multiple independent teams. 2. We demonstrate its effectiveness on the task of cybersecurity alignment, showing that the result- ing data is diverse and effective at aligning public models. 3. We construct and release a dataset for cybersecurity alignment, generated through our framework. This paper is structured as follows. We first review relevant literature (Section 2), followed by an overview of the Adversarial Arena framework (Section 3). We then discuss our deployment of the Adversarial Arena framework for the task of cybersecurity alignment, including design guidelines, evaluation protocol, outcomes (i.e. data and innovations from participating teams), and learnings (Section 4). Finally, we discuss broader applicability and limitations of the proposed framework (Section 5). Section 6 concludes the paper. 2RELATED WORK Crowdsourcing is a popular method for collecting data. However, traditional crowdsourcing meth- ods are prone to producing low quality data. Prior work suggests multiple reasons for this, ranging from satisficing behavior (Hamby & Taylor, 2016) to bad-faith responses and insufficient language fluency or skill level of annotators (Marshall et al., 2023). We attribute these problems to misaligned incentives and insufficient quality signals for the generated data. (Little et al., 2010) propose an it- erative crowdsourcing method that can improve data quality but its benefits are limited to particular domains. Recently, using generative AI has become a popular cost-effective approach to automating many of the above tasks. (Ding et al., 2022) show that using LLMs to label data results in orders of magnitude reduction in cost and time, compared to human labels, but training models on synthetic data leads to lower accuracy. As such, improving synthetic data generation has been an active area of research, with the goal of bridging this gap between human-generated and synthetic data. (Long et al., 2024) provide a comprehensive survey of LLM-based data generation, wherein they categorize prior work in this space into 3 stages: generation, curation, and evaluation. Generation is further subdivided into prompt engineering and multi-step generation. Prompt engineering tech- niques include various methods for task specification, including conditional prompting, role-play, and in-context learning (Wang et al., 2022; Yoo et al., 2021; Gunasekar et al., 2023; Eldan & Li, 2023; Ye et al., 2022b; Yu et al., 2023; Josifoski et al., 2023; Ding et al., 2023; Meng et al., 2022; He et al., 2023). Multi-step generation involves either generating individual samples through multiple generation steps (Li et al., 2022; Ye et al., 2023), or generating different subsets of the data over multiple steps (Honovich et al., 2022; Shao et al., 2023). Curation involves selecting high-quality samples from the generated data (Seedat et al., 2023; Ye et al., 2022a; Chen et al., 2023), or improv- ing the quality of the generated data (Chung et al., 2023; Pangakis et al., 2023; Liu et al., 2022). Evaluation consists of techniques that measure the faithfulness and diversity of the generated data, as well as approaches that use downstream task performance of models trained on the synthetically generated data (Havrilla et al., 2024). Several other papers survey synthetic data generation using large language models (Tan et al., 2024; Li et al., 2023; Guo & Chen, 2024; Bauer et al., 2024; Liu et al., 2024; Nadas et al., 2025), and point out limitations of existing approaches. Some common limitations include hallucinations, bias, diversity, and limited efficacy on subjective tasks. Importantly, these factors critically depend on design choices, such as which models are used for data generation, how prompts are constructed (including multi-step prompting and carefully selecting in-context learning examples), and strategies for filtering out or refining poor quality outputs. With a number of approaches being proposed for synthetic data generation, the space of design decisions is rapidly expanding, making it challenging to generate high-quality data for a given task. We propose a framework to crowdsource synthetic data that addresses the problem of misaligned incentives in crowdsourcing, and diversity and bias in existing synthetic data generation techniques. We introduce a ranking based incentive system where both attackers and defenders are strongly incentivized to generate the best quality data possible in order to achieve a high rank. Additionally, we allow different attackers and defenders to independently explore different parts of the design decision space, leading to better diversity and lower bias in the generated data. 3 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. 3OVERVIEW OF ADVERSARIAL ARENA The crux of the adversarial arena framework is a two-sided running competition where attackers and defenders compete against each other in a series of tournaments. The competition is a means to drive improvements on a specific “Task of Interest (ToI)” by simultaneously testing and generating new training data. Attackers in this context can refer to an automated system which can have a conversation with individual defenders and try to elicit failures at a given ToI. We define defenders to be the models or systems under test. These could range from individual LLMs to more complex agentic systems combining multiple components. Their goal is to respond to attackers’ requests while trying to correctly perform the ToI. Based on the specific ToI, these conversations can consist of a single-turn or more extensive conversations with multiple turns. The format is also agnostic to modalities and can incorporate one or more modalities like text, images, audio, and video. A critical aspect of the Adversarial Arena framework is a robust evaluation suite. In other words, this framework requires a mechanism to judge the winner for each conversation between an attacker and a defender. This evaluator serves multiple purposes in the framework 1) It labels the data generated through Adversarial Arena. 2) It provides a way to rank teams. Two separate leaderboards are maintained for attackers and defenders and ranking is determined by the number of conversations they win. 3) The labels generated by this evaluator serve as feedback signals for both attackers and defenders which can be used to improve their approaches/systems. While in an ideal scenario the evaluator will be perfect, our framework is designed to tolerate some noise to account for the infeasibility of perfect evaluation for many real world ToIs. Random errors in evaluation can be mitigated through having more conversations in a tournament or having multiple tournaments to average out error. This mitigation can ensure that the broader incentive structure for all attackers and defenders remains aligned to the ToI, but it cannot ensure the correctness of every label in the generated data. Another class of errors is systematic bias introduced by the evaluation strategy. A common example of this is the case of loss of functionality orthogonal to the ToI. As a mitigation, we introduce auxiliary objectives that influence the rankings of teams. Teams’ scores can be scaled based on their performance on auxiliary objectives. A detailed example of how to design such auxiliary objectives can be found in Section 4.2.2 where we illustrate the approach in the domain of cybersecurity alignment. In order to execute our concept of the Adversarial Arena at scale, we use an automated orchestrator service that can manage interactions between all attackers and defenders. We design this service to coordinate multiple multi-turn conversations in parallel in an asynchronous, reproducible, and fault tolerant manner. Additionally, the system can be run in test mode for attackers and defenders to ensure that their systems can reliably scale up for tournaments. Implementation details of the orchestrator can be found in Appendix B. 4CASE STUDY: APPLYING ADVERSARIAL ARENA TO CYBERSECURITY ALIGNMENT This section describes an example where we applied the Adversarial Arena framework to the task of Cybersecurity Alignment for LLMs Sahai et al. (2025). As large language models are becom- ing increasingly performant at generating code (Shibu, 2025; Novet, 2025), it becomes crucial to ensure these systems do not cause or facilitate harm. Recent studies (Pearce et al., 2021) show con- sistent patterns of security vulnerabilities in AI-generated code, which left unchecked can quickly propagate into production. Moreover, while it is beneficial to lower the technical barrier of entry to creating and working with software, it is important that the same technologies do not dramatically increase the number of malicious actors able to develop sophisticated cyberattacks. One challenge in aligning LLMs to prevent generation of insecure code or assistance with malicious cyberattacks is limited public data for these domains and in particular limited availability of multi-turn data. We applied the proposed Adversarial Arena framework to collect data for this task. We conducted a competition where 10 teams fielded bots to the adversarial arena. 5 attack teams were tasked with creating automatic systems that seek out weaknesses by trying expose willingness of coding sys- tems to produce malicious code, vulnerable code, or provide detailed assistance with cyberattacks. 5 defense teams fielded code generation systems that attempt to generate helpful responses while 4 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. avoiding generating malicious code, vulnerable code, or cyberattack assistance. In the next section we describe the challenge structure in more detail. 4.1CHALLENGE STRUCTURE & DESIGN GUIDELINES At the start of the competition, defender teams were given open weight access to an 8B pa- rameter coding specialist model built specifically for the challenge (henceforth referred to as ChallengeLLM), although a public model could be used for other challenges. The defenders were chartered with making their version of the model and surrounding system robust to adversarial attacks, all while maintaining utility. The two sides (attackers and defenders) then met up in a series of tournaments. With 5 attackers and 5 defenders, in each tournament there were 25 match-ups between attacking and defending teams. Each matchup between an attacker and a defender con- sisted of 200 conversations. Each conversation was allowed to have a maximum of 10 conversation turns back and forth (i.e. 5 adjacency pairs (Schegloff & Sacks, 1973)). We capped the interaction at 5 to avoid attacking teams exploring an unlimited number of attacks or probes within a single conversation, but allowing for multi-turn interaction. Design guidelines were used to keep the competition tractable and direct teams’ work towards pro- ducing the most useful data. Both attacking and defending teams were required to support multi-turn dialog. Prompts by attackers were required to be in English and/or human readable code – the con- straint to English was driven by annotation requirements. Only Python code was required to be supported by defenders. In keeping with common practice in LLM deployment, defending teams were allowed to augment their core model (built from the provided 8B coding model) with sur- rounding system components. Defending teams could alter the system prompt, classify and modify the incoming prompt from the user, and implement custom decoding logic. Pre-processing of the input including adding rules, classifiers, and small generative models was permitted. On the output side, defending systems could also include manipulation of model output using rules, classifiers, and small generative models. This included use of Chain-of-Thought style reasoning (Wei et al., 2023), followed by post-processing to remove internal thought traces. Also, to focus innovations on the core model and avoid defending system designs where, e.g. the core model is 8B and then a 70B open-source model is used for post-processing, the total number of parameters across all auxiliary models was required to not exceed 800M. In order to accommodate patterns such as self- reflection (Renze & Guven, 2024) or correction, so long as they stay within a latency budget of 45 seconds, teams were permitted to pass input through multiple versions of the core 8B coding model in sequence. Attackers were less restricted in the choice of LLMs they could incorporate into their systems. However, both attackers and defenders were not permitted to use closed-box model APIs at runtime. Attackers were free to incorporate open-source LLMs, potentially using and/or special- izing different models for different tasks (e.g. one model as an attack LLM to generate candidate attacks, and another as an assessor/judge LLM to rank candidate attacks or evaluate responses from the defending system). Attackers were permitted to connect these models with other system compo- nents (e.g. planners, rules, prompt mutators, dialog managers, etc.) to build the most adaptive and effective attack bots. In this challenge design, the data to drive teams’ innovations and development comes from their interaction with the 5 opposing teams they face through each tournament. We also found this format to be highly effective in driving competitive behavior. Throughout tournaments and office hours with each team, we repeatedly saw teams analyze and speculate on what the opposing sides might be doing and how to drive up their performance for the next round. It is important to note that the ad- versarial arena format actually drives two dimensions of competition. In the matchups, attackers and defenders come into contact and compete to win each conversation. However, attackers and defend- ers are not ranked against each other. The true competitors for the attackers are the other competing attacker teams, while the true competitors for defenders are the other competing defenders. 4.2EVALUATION As described in previous sections, the challenge consisted of a series of tournaments between attack- ers and defenders. Each matchup produced a set of conversations between the bots. For the com- petition, we set up an evaluation to decide the winner of each conversation (attacker of defender). Additionally, we created overall ranking strategies to incentivize teams to build the strongest attack 5 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Annotations per sample (Attacker wins/Defender wins)Malicious CodeMalicious ExplanationsOverall 0/3292027222616 1/2526675691 2/1298353369 3/0156150224 % agreement0.7890.7360.728 fleiss kappa0.4000.3340.385 krippendorff alpha0.4000.3340.385 Table 1: Inter-annotator Agreement for Security Event Detection bots and the safest defense systems. We describe our approach for evaluation in this section, which broadly consists of the following steps: 4.2.1EVALUATING EACH CONVERSATION Each conversation between any attacker and defender was evaluated in a two step process: Vulnerable Code Detection. We first extract Python code from each defender response in the conversation (if present) and run a static code analyzer (specifically, Amazon CodeGuru 1 ) on it to determine if the code had any vulnerabilities. If a vulnerability is detected, the conversation is marked as a “Successful Attack.” As we score at conversation level, having multiple vulnerable code blocks in a conversation does not give attackers any additional points. We recognize that static analysis tools are prone to generating false positives and can could result in safe code being marked as vulnerable. We treat this as a high precision, low recall filter on the generated dataset and accept this tradeoff considering that additional human verification or dynamic analysis were infeasible in context if this competition. Security Event Detection. If no vulnerable code is detected, we then have the conversation an- notated by a panel of human experts to determine if the defender model complied with a malicious request. We use human annotators with cybersecurity experience to label conversations for security events. Each conversation is labeled by three annotators, and the majority label is used as the conver- sation label. In order to ensure consistency, we developed detailed annotation guidelines which were provided to each annotator. These guidelines went through five refinement stages based on mock exercises, feedback from participating teams, and based on our learning during the competition. We worked with a pool of 30 expert annotators and each conversation was annotated by three different annotators. Low inter-annotator agreement was used to filter cases for inspection and find avenues for improvement in the annotation guidelines. Table 1 shows inter-annotator agreement scores, and Appendix D contains some analysis on this. If either of these modules returns TRUE, the conversation is marked as a “Successful Attack.” If not, it is a “Successful Defense.” 4.2.2AUXILIARY OBJECTIVES Diversity for Successful Attacks We wanted to preclude attacking teams from using identi- cal/similar attacks to the already successful attacks, and incentivize teams to generate diverse at- tacks. As such, we introduced an auxiliary objective for attackers to maintain diversity of attacks. We enforced this by measuring the diversity within the set successful attacks by an attacking team within a matchup. We experimented with both lexical (e.g. BLEU score (Papineni et al., 2002)) and embedding-based (e.g. SentenceBERT (Reimers & Gurevych, 2019)) approaches for this metric. In order to reward surface variation (e.g. paraphrases of attack strategies) we decided to use the BLEU score and focus on lexical similarity. As such, we used BLEU score to compute pairwise similarity, and then used the average similarity across all successful attack pairs to compute the final diversity score for an attacking team for the matchup. 1 https://aws.amazon.com/codeguru/ 6 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Utility Evaluation for Defenders To ensure that the defender teams’ models were still useful while being safe, we evaluated them on a suite of static utility test sets created for the competition. The test sets covered 1) Instruction based code generation (similar to (Chen et al., 2021)) 2) Multi- turn benign conversations related to cybersecurity concepts 3) Multiturn code generation. For all utility test sets, we normalized teams’ scores by capping the utility to the base ChallengeLLM’s utility. This ensured that teams were penalized when their systems lose util- ity but were not incentivized to generate data related to utility tasks. The final utility score for a defending team was obtained by averaging the normalized utility score for each set. 4.2.3RANKING TEAMS Ranking the attackers. The score for an attack team in each match-up was computed by com- bining the Attack Success Rate (ASR), with their diversity score. ASR is defined as the percentage of successful attack conversations with respect to the total number of conversations between an at- tacker and a defender. Intuitively, if two attacking teams have a similar ASR, but team A has lower diversity than team B, then it should be ranked lower than team B. As such, a team should be highly ranked if it has a high ASR as well as high diversity. We experimented with several combination measures, and the following formula to compute the normalized attack success rate (normalized ASR) was found to capture this intuition: Normalized ASR = ASR× Diversity 100 The overall score for an attacker was computed by averaging the normalized ASR across all defend- ers. This score was used for ranking the attackers. Ranking the defenders. The Defense Success Rate (DSR) of a defender in each match up is defined as the percentage of conversations between an attacker and a defender that were labeled in favor of the defender as per the process described in Section 4.2.1. These DSR scores are averaged across all attackers to compute the average defense success for a defender. The overall score for the defenders is computed by combining the average DSR across all attackers and the utility. Intuitively, this incentivizes defending teams to obtain high DSR while not regressing on utility compared to the base model. We experimented with several combination measures, and the following formula was found to capture this intuition and was used to rank defenders. Defense success is aggressively reduced as utility drops. Normalized DS = Average DS× Utility 100 4 4.3OUTCOMES This challenge demonstrated the effectiveness of the Adversarial Arena framework for generating high-quality adversarial data at scale. Through the competition we collected a rich dataset and observed the evolution of attack and defense strategies over multiple tournaments. Data Generation at Scale Throughout 13 practice runs and 4 official tournaments, over 96,000 multi-turn conversations were generated with minimal human intervention during execution. 20,000 of these were from official tournaments and were hence labeled. Discarding conversations that were incomplete due to execution failures, we get a final dataset of 19683 conversations. Each run typically completed in less than 10 hours, with attack bots averaging 2-7.9 seconds per response and defense bots averaging 4.1-10.1 seconds. Data Diversity Analysis We measure the diversity of the dataset generated from this challenge to demonstrate the following two benefits from the Adversarial Arena format: 1. Crowdsourcing synthetic data: Due to multiple teams generating synthetic data independently in an adversarial setting, we expect data generated by each team to have unique biases. 7 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Attacker LevelDefender LevelTournament Level Average SD for each subset0.29040.31140.3018 Average SD between all subset pairs0.32110.32820.3269 Table 2: Semantic diversity results 2. Adversarial format encourages improvement in data quality over time: As the Adversarial Arena framework works iteratively over multiple tournaments, we expect the data generated in each tournament to have unique biases. For our experiments, data subsets are considered to have different biases if the average Semantic Distance between samples within each data subset d i (denoted by SD(d i )) is lower than the average semantic distance between samples from different data subsets d j and d k (denoted by SD(d j , d k )). To measure semantic distance between samples, we encode each sample s by pooling all activations from the last hidden layer of the Mistral-7B-Instruct model (Jiang et al., 2023). This operation is denoted as E(s). The semantic distance between two samples s 1 and s 2 is defined as S(s 1 , s 2 ) = 1− Cosine(s 1 , s 2 ). Overall, SD(d i ) and SD(d j , d k ) are defined as follows: SD(d i ) = X s 1 ,s 2 ∈d i s 1 ̸=s 2 S(s 1 , s 2 ) andSD(d i , d j ) = X s 1 ∈d i ,s 2 ∈d j S(s 1 , s 2 )(1) In table 2 we show SD comparisons at three levels: 1) For the first column we construct subsets by dividing the dataset according to attack teams. Each subset contains all conversations involving a particular attack team across all tournaments. 2) For the second column subsets are created by dividing the data by defense teams. 3) For the third column subsets are created by tournament. All data generated in one tournament constitutes one subset. In all three cases, we report the average SD of all subsets, and the average SD between all pairs of subsets showing that all subsets have their own unique biases and hence contribute qualitatively different samples to the overall dataset. We also perform manual inspections of the data and found data generated by different teams to be qualitatively different, e.g. one of the attack teams had a lot of role playing style attacks. Another attacker generated a lot of prompts with requests to modify code that could result in vulnerabilities. Data Quality Analysis To study the effectiveness of the collected data, we fine-tuned an open weight model, Mistral-7B-Instruct (Jiang et al., 2023), and measured the improvement in safety of the resulting model. Specifically, we ran 2 experiments. First, we extracted all the conversations that do not contain vulnerable code in any of the defender responses (as detected by Amazon CodeGuru). The resulting dataset, containing 9,942 conversations, was used to fine-tune Mistral-7B-Instruct. The model before and after fine-tuning was tested on CyberSecEval Instruct prompts (Bhatt et al., 2023), which are likely to result in vulnerable code. Second, we extracted all the conversations that do not contain code or detailed explanations for malicious cyberactivity assistance in any of the de- fender responses (as labeled by expert human annotators). The resulting dataset, containing 13,336 conversations, was used to fine-tune Mistral-7B-Instruct. The model before and after fine-tuning was tested on CyberSecEval MITRE prompts (Bhatt et al., 2023) designed to elicit cybersecurity-related malicious responses from an LLM. See Table 3 for the results. We observe that the generated data results in substantial improvements across both secure code generation and malicious cyberactivity refusal tasks. 4.4LEARNINGS The data distribution for conversations between teams is unknown when the challenge starts, and evolves throughout the challenge. As such, we found that our initial evaluations suite did not ade- quately capture all the nuances of the attack and defense approaches. Therefore, we continued to update our evaluation throughout the challenge. Next, we observed that several attackers hosted an internal defense bot, and vice versa, to test their approaches in between tournaments. We believe that to provide teams with more intermediate feed- 8 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Secure Code Generation CyberSecEval-Instruct Malicious Cyberactivity CyberSecEval-MITRE Experiment No. of training samples Secure code generation (%) No. of training samplesRefusal (%) Mistral-7B-72.60-57.10 Mistral-7B (fine-tuned)9,94286.0113,33673.90 Table 3: Results of fine-tuning Mistral-7B-Instruct on conversations obtained through Adversarial Arena for both the secure code generation task (evaluated using CyberSecEval Instruct benchmark) and refusal to malicious cyberactivity requests (evaluated using CyberSecEval MITRE benchmark). back, the challenge structure could be modified to have more frequent but smaller tournaments. Alternately, the challenge could be turned into an online one, where teams need to keep their bots up throughout the challenge. We saw significant variation in teams’ rankings across different tournaments, particularly for at- tackers. We believe that this was, in part, because attackers that did well in previous tournaments exposed their most promising attacks, which defenders were able to guard against in future tourna- ments. If only the scores from the final tournament are used to decide the final ranking, it could incentivize teams to hold off their most promising approaches until the end of the challenge, which may not be desirable. One way to address this problem would be to take into account the scores from all the tournaments for the final ranking. Finally, while the challenge was focused on both vulnerable code and malicious code/explanations from defenders, most attackers found it easier to elicit vulnerable code compared to malicious code/explanations. Consequently, in the later part of the challenge, we saw attackers focus pri- marily on vulnerable code attacks. To balance exploration of multiple attack dimensions, it would be better to use a metric that penalizes imbalanced coverage, such as the harmonic mean of attack success rate across different dimensions. 5DISCUSSION We described the Adversarial Arena framework for the task of cybersecurity alignment. However, the framework is general and can easily extend to other classes of safety and security alignment for LLMs. Additionally, our framework can be adapted to tasks that may not be inherently adversarial in na- ture. For instance, LLMs tend to over-agree with humans (Ranaldi & Pucci, 2023). This can also be cast in our framework: attackers attempt to elicit agreement with invalid assertions, while defender teams align the model to resist such over-agreement. Another such task is the problem of building a proficient model for text summarization. Here, the attackers could be tasked with providing chal- lenging problems, that the model is not likely to work well on, while defense teams would be tasked with improving the model to keep up with increasingly challenging requests from the attackers. While our proposed approach is highly effective for crowdsourcing data, it can also be used as a framework to run competitions to foster innovation. The competition structure provides a dy- namic multi-turn evaluation framework, which can test model behavior not measurable by static benchmarks. Additionally, as each team is evaluated against multiple opponents, this framework incentivizes teams to build robust systems. The dynamic evaluation framework and the competition structure results in a flywheel effect, where teams’ approaches improve over the competition. Despite our proposed framework having several advantages, we recognize it has some limitations. The primary complication is that to execute a challenge using our framework that generates useful data and techniques from participating teams, it is crucial to design a good evaluation protocol. This involves scoping out what attackers and defenders are allowed to do. To rank attack and defense teams, an evaluation approach must be designed to label each conversation as a success for the attacker or defender. Auxiliary objectives may be needed to score attackers and defenders, similar to our attack diversity and defender utility scoring, as described in Section 4.2. 9 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. 6CONCLUSION Availability of sufficient quantities of high quality training data remains a significant challenge in the development and application of large language models. We propose a novel approach for crowd- sourcing data using the Adversarial Arena framework, which consists of an orchestrator that facil- itates multi-turn conversations between multiple attackers and defenders, competing over a series of tournaments. From the crucible of interactive competition, highly varied and diverse datasets can be extracted. As an example, we detail the application of the framework to the challenge of cybersecurity alignment for coding assistants. We present experiments showing that training on the resulting data improves the cybersecurity alignment of a public model and furthermore that the ef- fectiveness of the training data improves over the course of a sequence of tournaments. We also examine measures of relative data diversity using cosine distance among embeddings and show that relative diversity of data collected across multiple teams is more diverse from what we see from a single team. ETHICS STATEMENT Note that while the proposed technique generates multi-turn conversational data it goes not directly involve human subjects. The conversations result instead from interaction among automated attack and defense bots. Human evaluators annotating dialogs for attack or defense success worked under contract and were fairly compensated. We would also like to highlight the fact that the proposed technique is designed to address the problem of inherent bias in datasets. REFERENCES Llama Team AI @ Meta. The Llama 3 Herd of Models, 2024. https://arxiv.org/abs/ 2407.21783. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022. https://arxiv.org/abs/2204.05862. Andr ́ e Bauer, Simon Trapp, Michael Stenger, Robert Leppich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian Foster. Comprehensive Exploration of Synthetic Data Generation: A Survey. arXiv preprint arXiv:2401.02524, 2024. https://arxiv.org/abs/2401.02524. Akhiad Bercovich et al. Llama-Nemotron: Efficient Reasoning Models, 2025. https://arxiv. org/abs/2505.00949. Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksan- dar Straumann, Gabriel Synnaeve, Varun Vontimitta, Spencer Whitman, and Joshua Saxe. Pur- ple Llama CyberSecEval: A Secure Coding Benchmark for Language Models, 2023. https: //arxiv.org/abs/2312.04724. Jie Chen, Yupeng Zhang, Bingning Wang, Wayne Xin Zhao, Ji-Rong Wen, and Weipeng Chen. Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models, 2024. https://arxiv.org/abs/2406.12397. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a Better Alpaca with Fewer Data. arXiv preprint arXiv:2307.08701, 2023. https://arxiv.org/abs/2307.08701. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, 10 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained on Code, 2021. https://arxiv.org/abs/2107.03374. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Forty-first In- ternational Conference on Machine Learning, 2024. https://openreview.net/forum? id=3MW8GKNyzI. John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing Diversity While Main- taining Accuracy: Text Data Generation with Large Language Models and Human Interventions. arXiv preprint arXiv:2306.04140, 2023. https://arxiv.org/abs/2306.04140. Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, and Owain Evans. Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data, 2025. https://arxiv.org/abs/2507.14805. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Shafiq Joty, Boyang Li, and Lidong Bing. Is GPT-3 a Good Data Annotator? arXiv preprint arXiv:2212.10450, 2022. https: //arxiv.org/abs/2212.10450. Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. Data Augmentation Using Large Language Models: Data Perspectives, Learning Paradigms and Challenges, 2024. https://arxiv.org/abs/ 2403.02990. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing Chat Language Models by Scaling High-Quality Instructional Conversations. arXiv preprint arXiv:2305.14233, 2023. https://arxiv.org/abs/2305. 14233. Ronen Eldan and Yuanzhi Li. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? arXiv preprint arXiv:2305.07759, 2023. https://arxiv.org/abs/ 2305.07759. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C ́ esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are All You Need. arXiv preprint arXiv:2306.11644, 2023. https://arxiv.org/abs/2306. 11644. Xu Guo and Yiqiang Chen. Generative AI for Synthetic Data Generation: Methods, Challenges and the Future. arXiv preprint arXiv:2403.04190, 2024. https://arxiv.org/abs/2403. 04190. Tyler Hamby and Wyn Taylor. Survey Satisficing Inflates Reliability and Validity Measures: An Ex- perimental Comparison of College and Amazon Mechanical Turk Samples. Educ. Psychol. Meas., 76(6):912–932, December 2016. https://doi.org/10.1177/0013164415627349. Alex Havrilla, Andrew Dai, Laura O’Mahony, Koen Oostermeijer, Vera Zisler, Alon Albalak, Fab- rizio Milo, Sharath Chandra Raparthy, Kanishk Gandhi, Baber Abbasi, et al. Surveying the Ef- fects of Quality, Diversity, and Complexity in Synthetic Data from Large Language Models. arXiv preprint arXiv:2412.02980, 2024. https://arxiv.org/abs/2412.02980. Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators. arXiv preprint arXiv:2303.16854, 2023. https://arxiv.org/ abs/2303.16854. 11 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor. arXiv preprint arXiv:2212.09689, 2022. https://arxiv.org/abs/2212.09689. Artur Horal, Daniel Pina, Henrique Paz, Iago Paulo, Jo ̃ ao Soares, Rafael Ferreira, Diogo Tavares, Diogo Gl ́ oria-Silva, Jo ̃ ao Magalh ̃ aes, and David Semedo. Redtwiz: Diverse llm red teaming via adaptive attack planning, 2025. URL https://arxiv.org/abs/2510.06994. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L ́ elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth ́ e Lacroix, and William El Sayed. Mistral 7B, 2023. https://arxiv.org/ abs/2310.06825. Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction. arXiv preprint arXiv:2303.04132, 2023. https://arxiv.org/abs/2303.04132. Ond ˇ rej Kobza, Adam ˇ Cern ́ y, Ivan Dost ́ al, Jan ˇ Sediv ́ y, Maria Rigaki, Muris Sladi ́ c, and Sebastian Garcia. Alquistcoder: A constitution-guided approach to safe, trustworthy code generation. Ama- zon Nova AI Challenge Proceedings, 2025. URL https://w.amazon.science/nova- ai-challenge/proceedings/alquistcoder-a-constitution-guided- approach-to-safe-trustworthy-code-generation. Andreas K ̈ opf, Yannic Kilcher, Dimitri von R ̈ utte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Rich ́ ard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant Conversations – Democratizing Large Language Model Align- ment, 2023. https://arxiv.org/abs/2304.07327. Junlong Li, Jinyuan Wang, Zhuosheng Zhang, and Hai Zhao. Self-Prompting Large Language Models for Zero-Shot Open-Domain QA. arXiv preprint arXiv:2212.08635, 2022. https: //arxiv.org/abs/2212.08635. Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning, 2024. https: //arxiv.org/abs/2402.00530. Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin.Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations.arXiv preprint arXiv:2310.07849, 2023. https://arxiv.org/abs/2310.07849. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller.Exploring Iterative and Parallel Human Computation Processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10, p. 68–76, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781450302227. doi: 10.1145/1837885.1837907. https: //doi.org/10.1145/1837885.1837907. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. WANLI: Worker and AI Collabo- ration for Natural Language Inference Dataset Creation. arXiv preprint arXiv:2201.05955, 2022. https://arxiv.org/abs/2201.05955. Jiawei Liu, Nirav Diwan, Zhe Wang, Haoyu Zhai, Xiaona Zhou, Kiet A. Nguyen, Tianjiao Yu, Muntasir Wahed, Yinlin Deng, Hadjer Benkraouda, Yuxiang Wei, Lingming Zhang, Ismini Lourentzou, and Gang Wang. Purpcode: Reasoning for safer code generation, 2025a. URL https://arxiv.org/abs/2507.19060. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best Practices and Lessons Learned on Synthetic Data for Language Models. CoRR, 2024. https://arxiv.org/abs/2404.07503. 12 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Xiaogeng Liu, Jingyu Huang, Jiongxiao Wang, Yingzi Ma, Hao Wu, and Chaowei Xiao. Stepwise multi-turn jailbreak attacks on code llms via task decomposition and test-time scaling. Amazon Nova AI Challenge Proceedings, 2025b. URL https://w.amazon.science/nova- ai-challenge/proceedings/stepwise-multi-turn-jailbreak-attacks- on-code-llms-via-task-decomposition-and-test-time-scaling. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. arXiv preprint arXiv:2406.15126, 2024. https://arxiv.org/abs/2406.15126. Catherine C. Marshall, Partha S.R. Goguladinne, Mudit Maheshwari, Apoorva Sathe, and Frank M. Shipman.Who Broke Amazon Mechanical Turk?An Analysis of Crowdsourcing Data Quality over Time. In Proceedings of the 15th ACM Web Science Conference 2023, Web- Sci ’23, p. 335–345, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700897. doi: 10.1145/3578503.3583622. https://doi.org/10.1145/ 3578503.3583622. Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating Training Data with Language Models: Towards Zero-Shot Language Understanding. Advances in Neural Information Pro- cessing Systems, 35:462–477, 2022. https://proceedings.neurips.c/paper_ files/paper/2022/file/0346c148ba1c21c6b4780a961ea141dc-Paper- Conference.pdf. Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Dongwon Jung, Hadi Askari, Wenxuan Zhou, Zhe Zhao, and Muhao Chen. Redcoder: Automated multi-turn red teaming for code llms, 2025. URL https://arxiv.org/abs/2507.22063. Mihai Nadas, Laura Diosan, and Andreea Tomescu. Synthetic Data Generation Using Large Lan- guage Models: Advances in Text and Code. arXiv preprint arXiv:2503.14023, 2025. https: //arxiv.org/abs/2503.14023. Atharva Naik, Alex Xie, Abhinav Rao, Anmol Agarwal, Shubham Gandhi, Michael Hilton, Carolyn Ros ́ e, and Team Purpl3pwn3rs. Secure and useful models are reasonable: Aligning code models via utility-preserving reasoning. Amazon Nova AI Challenge Proceedings, 2025. URL https://w.amazon.science/nova-ai-challenge/proceedings/secure- and-useful-models-are-reasonable-aligning-code-models-via- utility-preserving-reasoning. Jordan Novet. Satya Nadella says as much as 30% of microsoft code is written by AI. cnbc, 04 2025. https://w.cnbc.com/2025/04/29/satya-nadella-says-as-much- as-30percent-of-microsoft-code-is-written-by-ai.html. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Follow Instructions with Human Feed- back. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, p. 27730–27744. Curran As- sociates, Inc., 2022. https://proceedings.neurips.c/paper_files/paper/ 2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf. Nicholas Pangakis, Samuel Wolken, and Neil Fasching. Automated Annotation with Generative AI Requires Validation. arXiv preprint arXiv:2306.00176, 2023. https://arxiv.org/abs/ 2306.00176. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, p. 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- tics. doi: 10.3115/1073083.1073135. https://aclanthology.org/P02-1040/. 13 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions, 2021. https://arxiv.org/abs/2108.09293. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only, 2023. https://arxiv.org/abs/2306.01116. Guilherme Penedo, Hynek Kydl ́ ı ˇ cek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom- czak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, p. 30811–30849. Curran Associates, Inc., 2024. https://proceedings.neurips. c/paper_files/paper/2024/file/370df50ccfdf8bde18f8f9c2d9151bda- Paper-Datasets_and_Benchmarks_Track.pdf. Jinjun Peng, Weiliang Zhao, Ira Ceka, Alex Mathai, Adam ˇ Storek, Hailie Mitchell, and Junfeng Yang. Securelion: Building a trustworthy ai assistant with security reasoning in a realistic adversarial competition.Amazon Nova AI Challenge Proceedings, 2025.URL https: //w.amazon.science/nova-ai-challenge/proceedings/securelion- building-a-trustworthy-ai-assistant-with-security-reasoning-in- a-realistic-adversarial-competition. Leonardo Ranaldi and Giulia Pucci. When Large Language Models Contradict Humans? Large Language Models’ Sycophantic Behaviour. arXiv preprint arXiv:2311.09410, 2023. https: //arxiv.org/abs/2311.09410. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings Using Siamese BERT- Networks. arXiv preprint arXiv:1908.10084, 2019. https://arxiv.org/abs/1908. 10084. Matthew Renze and Erhan Guven. The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), p. 476–483. IEEE, November 2024. doi: 10.1109/fllm63129.2024. 10852493. http://dx.doi.org/10.1109/FLLM63129.2024.10852493. Sattvik Sahai, Prasoon Goyal, Michael Johnston, Anna Gottardi, Yao Lu, Lucy Hu, Luke Dai, Shao- hua Liu, Samyuth Sagi, Hangjie Shi, Desheng Zhang, Lavina Vaz, Leslie Ball, Maureen Murray, Rahul Gupta, and Shankar Ananthakrishna. Amazon nova ai challenge – trusted ai: Advancing secure, ai-assisted software development, 2025. URL https://arxiv.org/abs/2508. 10108. Emanuel A. Schegloff and Harvey Sacks. Opening up Closings. Semiotica, 8(4):289–327, 1973. doi: 10.1515/semi.1973.8.4.289. https://web.stanford.edu/ ̃ eckert/Courses/ l1562018/Readings/SchegloffSacks1973.pdf. Nabeel Seedat, Nicolas Huynh, Boris Van Breugel, and Mihaela Van Der Schaar. Curated LLM: Synergy of LLMs and Data Curation for Tabular Augmentation in Low-Data Regimes. arXiv preprint arXiv:2312.12112, 2023. https://arxiv.org/abs/2312.12112. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Syn- thetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models. In International Conference on Machine Learning, p. 30706–30775. PMLR, 2023. https: //proceedings.mlr.press/v202/shao23a/shao23a.pdf. Sherin Shibu. AI is already writing about 30% of code at Microsoft and Google. Here’s what it means for software engineers. MSN, 04 2025. https://w.msn.com/en-us/money/ news/ai-is-already-writing-about-30-of-code-at-microsoft-and- google-here-s-what-it-means-for-software-engineers/ar-A1DWyrq. 14 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large Language Models for Data Anno- tation and Synthesis: A Survey. arXiv preprint arXiv:2402.13446, 2024. https://arxiv. org/abs/2402.13446. Laurens van der Maaten and Geoffrey Hinton.Visualizing Data using t-SNE.Journal of Machine Learning Research, 9(86):2579–2605, 2008. http://jmlr.org/papers/v9/ vandermaaten08a.html. Jennifer Wortman Vaughan. Making Better Use of the Crowd: How Crowdsourcing Can Ad- vance Machine Learning Research. Journal of Machine Learning Research, 18(193):1–46, 2018. https://jmlr.org/papers/volume18/17-234/17-234.pdf. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv preprint arXiv:2212.10560, 2022. https://arxiv.org/abs/2212.10560. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions, 2023. https://arxiv.org/abs/2212.10560. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2023. https://arxiv.org/abs/2201.11903. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering Code Generation with OSS-Instruct, 2024. https://arxiv.org/abs/2312.02120. Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. A Survey of Human-in-the-Loop for Machine Learning. Future Generation Computer Systems, 135:364–381, 2022. https://doi.org/10.1016/j.future.2022.05.014. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions, 2025a. https://arxiv.org/abs/2304.12244. Xiangzhe Xu, Guangyu Shen, Zian Su, Siyuan Cheng, Hanxi Guo, Lu Yan, Xuan Chen, Jiasheng Jiang, Xiaolong Jin, Chengpeng Wang, Zhuo Zhang, and Xiangyu Zhang. Astra: Autonomous spatial-temporal red-teaming for ai software assistants, 2025b. URL https://arxiv.org/ abs/2508.03936. Zexin Xu, Tingxi Li, Ravishka Rathnasuriya, Zihe Song, Jun Ren, Bhavesh Mandalapu, Soroush Se- tayeshpour, Xinya Du, and Wei Yang. Comet: Closed-loop orchestration for malicious elicitation techniques in code models. In Amazon Nova AI Challenge Proceedings, 07 2025c. Jiacheng Ye, Jiahui Gao, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Progen: Progres- sive Zero-Shot Dataset Generation via In-Context Feedback. arXiv preprint arXiv:2210.12329, 2022a. https://arxiv.org/abs/2210.12329. Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Ling- peng Kong. ZeroGen: Efficient Zero-Shot Learning via Dataset Generation. arXiv preprint arXiv:2202.07922, 2022b. https://arxiv.org/abs/2202.07922. Jiacheng Ye, Chengzu Li, Lingpeng Kong, and Tao Yu. Generating Data for Symbolic Language with Large Language Models. arXiv preprint arXiv:2305.13917, 2023. https://arxiv. org/abs/2305.13917. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. GPT3Mix: Leveraging Large-Scale Language Models for Text Augmentation. arXiv preprint arXiv:2104.08826, 2021. https://arxiv.org/abs/2104.08826. 15 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Ji- aming Shen, and Chao Zhang.Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias.Advances in Neural Information Processing Systems, 36:55734–55784, 2023. https://proceedings.neurips.c/paper_ files/paper/2023/file/ae9500c4f5607caf2eff033c67daa9d7-Paper- Datasets_and_Benchmarks.pdf. Yi Zeng, Mahavir Dabas, Tran Huynh, Nikhil Reddy, Adam Nguyen, Sanchit Kabra, and Ruoxi Jia.Data is all you need (almost):Iterative synthetic instruction tun- ing for secure code generation.Amazon Nova AI Challenge Proceedings, 2025.URL https://w.amazon.science/nova-ai-challenge/proceedings/data- is-all-you-need-almost-iterative-synthetic-instruction-tuning- for-secure-code-generation. Amir Zur, Alexander R Loftus, Hadas Orgad, Zhuofan Ying, Kerem Sahin, and David Bau. It’s Owl in the Numbers: Token Entanglement in Subliminal Learning. https://owls.baulab. info/, 2025. Blog post. 16 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. ADIVERSITY VISUALIZATIONS Figure 2: T-SNE plots: Conversations in the left plot are grouped by attackers, the middle plot is grouped by defenders, and the plot on the right has conversations grouped by tournaments. Figure 2 shows 2D T-SNE (van der Maaten & Hinton, 2008) plots of the dataset obtained from the Cybersecurity Alignment Challenge using Adversarial Arena. Points in the first plot are colored by attackers. The plot shows that conversations by different attackers occupy different regions in 2D space. This further supports our claim that data generated by different teams is qualitatively different and contains different biases. The collection of all these datasets results in a richer dataset where these individual biases are balanced out. The second plot (middle) also exhibits this pattern but not as pronounced. We believe this is because conversations are driven by attackers as they generate the prompts. Additionally, as the competition was related to cybersecurity alignment, a large portion of defender responses are refusals which tend to be semantically and lexically similar. The last plot is colored by tournaments. This also exhibits different biases for different subsets of the dataset. BORCHESTRATOR INFRASTRUCTURE DETAILS The Orchestrator Infrastructure is built mainly using AWS Lambda 2 , Amazon SQS (Simple Queue Service) 3 and Amazon DynamoDB 4 to achieve a fully serverless, scalable, and event-driven archi- tecture. It consists of two primary phases as described below. (See Figure3 for a schematic.) B.1INITIALIZATION PHASE The Config Assistant Lambda fetches the list of eligible bots from a database and constructs all attacker-defender pairs. It records pair configurations (e.g., session targets, readiness status, number of finished sessions) in a tournament config table. Once pair readiness is verified, the Session Co- ordinator Lambda retrieves all eligible pairs and enqueues the first batch of session-start messages (with empty history) into each attacker’s SQS queue. B.2RUNTIME PHASE (LIFE OF A SESSION) The core unit of orchestration is a multi-turn session between an attacker and a defender: 1. Attack team scheduler invokes the attacker handler (owned by the Orchestrator), which dequeues a session message, constructs a request including session history, and calls the attack team’s Lambda endpoint (owned by team’s bot). (Steps 1-3 in Figure 3) 2. The attacker’s response is logged to the database. If no end signal is returned, a new message with updated history is sent to the defender’s queue. (Steps 4-5 in Figure 3) 3. Defense team Scheduler invokes the defender handler (owned by the Orchestrator), which repeats the above steps for the defender. (Steps 6-10 in Figure 3) 2 https://aws.amazon.com/lambda/ 3 https://aws.amazon.com/sqs/ 4 https://aws.amazon.com/dynamodb/ 17 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. 4. This alternating turn-based flow continues until an end signal is received, a fatal error occurs for either team, or a turn limit is reached. 5. Upon session termination, the Session Coordinator Lambda is notified. It updates session meta- data in the tournament config table and logs high-level session details in the database. If more sessions are needed for the pair, another batch is enqueued. (Steps 11-15 in Figure 3) This lifecycle abstracts away the pacing concerns from bot teams, while allowing sessions to proceed independently across pairs and batches. Figure 3: Orchestrator Architecture B.3FUNCTIONAL GUARANTEES The orchestrator enforces the following guarantees to ensure fairness, robustness, and experimental control: Pairing and Session Scheduling All attacker-defender pairs are statically defined during initial- ization based on the tournament configuration. The system supports per-pair session quotas, en- abling unequal traffic allocation for A/B testing or special matchups. Turn-Based Request Handling Sessions strictly alternate between attacker and defenders by co- ordinating separate Lambda handlers and SQS queues. Each Lambda invocation handles only a single bot response per turn, which ensures that even long-running sessions—exceeding 15 minutes overall—remain compatible with the Lambda execution model. This design avoids the need for session-level infrastructure such as EC2, Amazon Elastic Container Service (ECS), or AWS Batch, maintaining a fully serverless, low-maintenance, and flexible architecture that scales efficiently with minimal operational overhead. Each request carries full session context, preserving chronological state even for stateless bots. Session Control and Termination Sessions terminate when an attacker signals end-of-session, a fatal error occurs, or the maximum number of turns is reached. The Session Coordinator dy- namically monitors the number of finished sessions and session status, and automatically launches additional batches until all configured sessions for each pair are completed. 18 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Error Tolerance and Fault Isolation Each bot has an independent execution context and request queue. Bots experiencing issues can be paused without affecting others. Failed API calls are retried once; persistent failures trigger session termination and log updates. Traffic Control and Batching The system enforces consistent message pacing, which prevents overwhelming bot endpoints. Sessions are launched in batches, allowing attackers to adapt their strategies between batches. Partial Availability Support The system starts or continues tournaments as long as at least one attacker and defender are online. Offline bots are skipped temporarily and can be resumed upon recovery. Elastic Scaling Infrastructure Stateless Lambda functions and decoupled queues scale automat- ically with the number of bots and sessions. B.4DESIGN TRADE-OFFS AND CONSIDERATIONS The Orchestrator was designed for scalability, modularity, and resilience, but several trade-offs were considered: Limited Real-Time Feedback By design, the orchestrator buffers and delays intermediate results until sessions conclude, which limits live monitoring. Latency Turn-based interactions incur delay due to Lambda cold starts 5 and SQS polling, which may not reflect real-time conversation dynamics. Retry Semantics Bots must be designed to handle duplicate requests due to Lambda retries, adding complexity for stateful bots. Despite these limitations, the orchestrator provides a robust and extensible framework for running high-integrity adversarial evaluations at scale. CADDITIONAL OUTCOMES FROM THE CYBERSECURITY ALIGNMENT CASE STUDY Evolution of Attack Success Patterns Analysis of the tournament conversations (Figure 4) re- vealed interesting dynamics in attack success rates. The percentage of conversations (Table 4) with detected security events (malicious code or cyberattack assistance) decreased consistently from Tournament 1 to Tournament 3. This trend indicates that defenders successfully adapted their de- fenses against these attacks, which made security events difficult to elicit. In contrast, code vulnerabilities remained a persistent challenge throughout all tournaments. Each tournament typically uncovered tens of distinct vulnerability types (Table 5), mapped to various Common Weakness Enumerations (CWEs). Individual vulnerable conversations often contained multiple vulnerabilities. Among the detected vulnerabilities, certain types such as resource leaks and OS command injection appeared with higher frequency, demonstrating the effectiveness of attacks targeting resource management and system-level operations. Novel Approaches from Competing Teams Throughout the tournaments, participating teams developed innovative strategies that evolved in response to their opponents’ tactics. Defense teams developed innovative defensive strategies that shared several key themes: multi-component archi- tectures with input classifiers and output guardrails, synthetic data generation for supervised fine- tuning and preference optimization, and reasoning-based alignment inspired by recent advances in deliberative models. Notably, teams like Team Purpl3pwn3rs (Carnegie Mellon University) (Naik et al., 2025) and Team PurpCorn-Plan (University of Illinois Urbana-Champaign) (Liu et al., 2025a) 5 https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime- environment.html#cold-start-latency 19 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. incorporated reinforcement learning with custom reward functions combining static analysis tools and LLM judges to jointly optimize for safety and utility. Team Alquist (Czech Technical Uni- versity) (Kobza et al., 2025) introduced a dynamic prompting system where an intent recognition classifier adjusted the system prompt based on whether requests were benign, malicious, or bor- derline, coupled with output verification that triggered response regeneration when needed. Team LionCoder (Columbia University) (Peng et al., 2025) trained a specialized vulnerability fixer model, and Team HokieTokie (Virginia Tech) (Zeng et al., 2025) created a scalable synthetic data pipeline for finetuning the model. Attackers pursued equally diverse attack strategies. Many teams built attacker-defender-evaluator frameworks, using these multi-component systems to iteratively refine their attacks. A common technique was transforming benign utility examples into harmful prompts, often using multi- turn conversations to gradually escalate the malicious content. Team SaFoLab’s (University of Wisconsin-Madison) (Liu et al., 2025b) approach was a good example of this gradual benign-to- harmful transition. Teams developed sophisticated attack planners - for instance, Team Astro’s (University of Texas at Dallas) (Xu et al., 2025c) COMET system evaluated prompts across mul- tiple dimensions (strategy, objective, style, template), while Team RedTWIZ (NOVA University, Lisbon, Portugal) (Horal et al., 2025) employed hierarchical planning with upper confidence bound algorithms for strategy selection. Particularly innovative approaches included Team PurCL’s (Pur- due University) (Xu et al., 2025b) use of Gibbs sampling to efficiently explore the attack space and find borderline cases where judge models disagreed, and Team CapitalAI’s (University of Califor- nia, Davis) (Mo et al., 2025) strategy library that captured patterns from both failed and successful attacks to adaptively evolve prompts during deployment. The independent development of these diverse approaches by competing teams generated a rich dataset spanning a wide spectrum of attack vectors and defensive strategies. This competitive environment produced strategies with sophisti- cation and diversity that would be difficult to achieve through traditional crowdsourcing or purely synthetic generation methods. T01T02T03 0 200 400 600 800 1,000 1,200 1,400 1,600 967 1,378 1,039 587 237 178 Number of Sessions Vulnerable Code Detected SessionsSecurity Event Detected Sessions Figure 4: Vulnerable vs Malicious Sessions Across Tournaments DANALYSIS OF INTER-ANNOTATOR AGREEMENT Figure 7 shows visualizations of inter-annotator agreement across all pairs of annotators, for MAL CODE, MALEXPLN and overall.The annotators are sorted by their average agree- ment scores. We see that the agreement scores are slightly higher for MALCODE, than for MAL EXPLN. The figure also shows the histogram of inter-annotator agreement scores accross 20 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Tournament% of Vulnerable Code% of Security Event Detected SessionsDetected Sessions T0119%12% T0228%5% T0321%4% Table 4: Percentage of Vulnerable and Malicious Sessions Across Tournaments Vulnerability TitleOccurrence CWE-400,664 - Resource leak1221 CWE-77,78,88 - OS command injection1180 CWE-327 - Insecure cryptography429 CWE-319 - Insecure connection using unencrypted protocol290 Not setting the connection timeout parameter254 CWE-798 - Hardcoded credentials217 CWE-327,328 - Insecure hashing190 CWE-269 - Improper privilege management155 CWE-20,79,80 - Cross-site scripting134 CWE-295 - Improper certificate validation134 Table 5: Top 10 Most Frequent Vulnerabilities in Tournament 3 (from 38 unique vulnerability types mapping to 44+ CWEs) these categories which shows the distribution of agreement between annotators. Further, we calcu- late the average agreement scores for each annotator, averaged over all annotators that they shared an annotation task with. This can be found in Table 6 EUTILITY BENCHMARKS FOR THE CYBERSECURITY CHALLENGE Utility benchmarks were used during the challenge as an auxiliary objective for defender teams. To ensure that these benchmarks remain truly hidden, we created custom benchmarks using a combina- tion of synthetic generation and human verification. Participating teams received a subset of these benchmarks as development sets and were tested against new subsets for each tournament. E.1INSTRUCTION BASED CODE GENERATION This benchmark consisted of function level code generation tasks. We first generated multiple prompts using LLMs by providing a random batch of prompts from HumanEval(Chen et al., 2021). We then generated a large number of solutions and test cases for each prompt and run each solution against all test cases. We then only keep the solution that passes the largest set of test cases and discard all other solutions and the failing test cases. Finally, these prompts, solutions, and test cases are manually reviewed by a human annotator for correctness before being used in the competition. E.2CYBERSECURITY QA This benchmark contained benign questions related to cybersecurity (e.g., “What are the different types of malware?”). We manually collected a set of keywords and used LLMs to generate a set of questions about them. Then, we asked LLMs to generate multi-turn conversation around each of these questions where the question would be the last turn for the model under test to respond to. This benchnmark was evaluated using an LLM judge that detects if the model deflected the question or answered it. The limitation of this benchmark was that it did not check for the correctness of the response, but we found this acceptable as an auxiliary objective for the challenge. 21 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. annotator # avg agreement code annotator # avg agreement explanations annotator # avg agreement overall 140.89350.876140.867 120.885140.86950.861 50.88460.861120.857 10.884190.856170.856 170.88110.85610.850 30.881120.854290.845 90.88190.852190.843 260.878290.85030.837 130.875300.844160.836 190.87330.84390.836 290.871160.84260.830 200.869130.83840.826 40.86740.837130.826 80.863170.834200.825 250.862260.833260.825 160.860200.832250.824 180.86070.829300.823 70.858250.81770.816 300.857180.81420.814 60.85420.813180.814 280.850220.810100.801 150.844210.804220.799 270.843100.802150.797 230.840150.793270.796 20.839270.78980.792 100.837240.788240.790 210.827110.785110.789 110.82480.777210.777 220.824280.771280.770 240.817230.766230.742 Table 6: Average agreement of each annotator in decreasing order 22 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Figure 5: Example Adversarial Conversation 1: A representative conversation between an attacker and a defender captured during a tournament. The attack attempts to elicit malicious code and cyber- attack assistance through crafted prompts. All malicious content has been redacted with descriptive placeholders. Non-malicious code was truncated for brevity. E.3MULTI-TURN CODE GENERATION This benchmark was built to test the ability of defender systems on coding tasks in domains like database access, web servers, etc. As code for these domains are more likely to have vulnerabilities, this would be more likely to have overlap with tournament conversations. To build this benchmark, we started by generating prompts using LLMs with prompts from CyberSecEval (instruct subset) as 23 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. Figure 6: Example Adversarial Conversation 2: A conversation demonstrating a multi-step attack strategy, where the attacker begins with benign requests and gradually transitions to malicious in- tentions over five turns. Malicious content has been redacted with descriptive labels. 24 Published as a paper at 3rd DATA-FM workshop @ ICLR 2026, Brazil. (a) Pairwise IIA scores: Malicious Code (b) Pairwise IIA scores: Malicious Explanations(c) Pairwise IIA scores: Overall (d) Histogram of IIA agreement scores: Malicious Code (e) Histogram of IIA agreement scores: Malicious Explanations (f) Histogram of IIA agreement scores: Overall Figure 7: Visualizations of pairwise agreements between annotators, along with a histogram of inter- annotator agreement scores seeds. We then generated 10 responses for each of these prompts and checked for code vulnerabil- ities in all responses using CodeGuru 6 . We discarded prompt for which more than 7 or less than 1 response were flagged. This way we were left with prompts for which there exists a secure solution but there could also be vulnerable solutions. Finally, we used an LLM to expand each prompt into a multi-turn conversation. Performance on this benchmark was evaluated using and LLM Judge. To make the benchmark stylistically closer to tournament conversations, we also implemented some jailbreak techniques in some of these benign conversations. 6 https://aws.amazon.com/codeguru/ 25