Paper deep dive
X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems
Yeonsung Jung, Zhiyuan Wang, Zifan Wang, Vítor Albiero, Benjamin Newman, Edoardo Debenedetti, Ivan Evtimov, Ninareh Mehrabi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/13/2026, 3:10:16 AM
Summary
The paper introduces X-AgentRed, a self-evolving red-teaming framework designed to discover harmful cross-service tool call paths in personal agent ecosystems. Unlike existing methods that rely on LLM generation biased by training data, X-AgentRed constructs a 'HarmFlow' graph based on tool dependencies (effects satisfying preconditions) to systematically sample recipes. It uses execution feedback to diversify successful harmful paths via tool substitution and refine blocked paths by adjusting preconditions. Evaluated across four frontier agents in a simulated ecosystem of 30 services and 914 tools, X-AgentRed achieves a 1.77x higher harmful path rate and 2.99x greater tool coverage compared to the STAC baseline.
Entities (10)
Relation Signals (9)
X-AgentRed → uses → HarmFlow
confidence 95% · X-AgentRed discovers harmful tool call paths through a graph of tool dependencies... referred to as the HarmFlow graph
HarmFlow → captures → tool dependencies
confidence 90% · HarmFlow graph... captures how one tool’s effects can satisfy another’s execution preconditions
X-AgentRed → evaluatedon → ToolSimulator
confidence 90% · We evaluate X-AgentRed... in a simulated multi-service ecosystem... We use ToolSimulator
X-AgentRed → outperforms → TRACE
confidence 90% · X-AgentRed achieves... 10.36× the harmful path rate... compared with... TRACE
X-AgentRed → outperforms → STAC
confidence 90% · X-AgentRed achieves 1.77× the harmful path rate of STAC
X-AgentRed → testedon → Claude Opus
confidence 90% · We use Claude Opus 4.8 as the red-teaming agent... The four target models are... Claude Opus 4.8
X-AgentRed → testedon → GPT 5.6 Sol
confidence 90% · The four target models are GPT-5.6 Sol... each operating as a personal agent
X-AgentRed → testedon → GLM-5.2
confidence 90% · The four target models are... GLM-5.2... each operating as a personal agent
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Personal agents increasingly connect to services across the web and applications by using tools. As the effects of their actions accumulate and interact across services, unintended or harmful outcomes can arise when agents fulfill benign user requests. Proactively identifying these compositional failures requires systematically exploring diverse cross-service paths. Existing red-teaming methods often simply prompt LLMs to identify potentially harmful paths from task and environment descriptions or decompose harmful objectives specified by humans or LLMs into seemingly benign action combinations. However, generation is biased toward patterns learned during training, potentially favoring familiar tool combinations and leaving novel cross-service interactions underexplored. To address this, we propose X-AgentRed, a self-evolving red-teaming framework that discovers harmful tool call paths through a graph of tool dependencies and expands and refines them using execution feedback. By analyzing each tool's effects, potential harms, and preconditions for execution, the red-teaming agent constructs a directed graph that captures how one tool's effects can satisfy another's execution preconditions. Recipes sampled from this graph are instantiated as realistic scenarios in which agents can produce harmful outcomes. Execution feedback enables self-improvement by diversifying paths that produce harm through tool substitution and refining preconditions at blocked steps in unsuccessful paths to enable further execution. Across four frontier agentic models, X-AgentRed achieves $1.77\times$ the harmful path rate of STAC and explores a more diverse set of tools, yielding $2.99\times$ the tool coverage on average.
Tags
Links
- Source: https://arxiv.org/abs/2610.13886v1
- Canonical: https://arxiv.org/abs/2610.13886v1
Trouble viewing inline? Open PDF directly →
Full Text
56,756 characters extracted from source content.
Expand or collapse full text
X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems Yeonsung Jung Affiliation: Meta Reality Labs Research Work done at Meta Zhiyuan Wang Affiliation: Meta Reality Labs Research Zifan Wang Affiliation: Meta Superintelligence Labs Vítor Albiero Affiliation: Meta Superintelligence Labs Benjamin Newman Affiliation: Meta Reality Labs Research Edoardo Debenedetti Affiliation: Meta Ivan Evtimov Affiliation: FAIR at Meta Ninareh Mehrabi Affiliation: Meta Reality Labs Research Abstract Personal agents increasingly connect to services across the web and applications by using tools. As the effects of their actions accumulate and interact across services, unintended or harmful outcomes can arise when agents fulfill benign user requests. Proactively identifying these compositional failures requires systematically exploring diverse cross-service paths. Existing red-teaming methods often simply prompt LLMs to identify potentially harmful paths from task and environment descriptions or decompose harmful objectives specified by humans or LLMs into seemingly benign action combinations. However, generation is biased toward patterns learned during training, potentially favoring familiar tool combinations and leaving novel cross-service interactions underexplored. To address this, we propose X-AgentRed, a self-evolving red-teaming framework that discovers harmful tool call paths through a graph of tool dependencies and expands and refines them using execution feedback. By analyzing each tool’s effects, potential harms, and preconditions for execution, the red-teaming agent constructs a directed graph that captures how one tool’s effects can satisfy another’s execution preconditions. Recipes sampled from this graph are instantiated as realistic scenarios in which agents can produce harmful outcomes. Execution feedback enables self-improvement by diversifying paths that produce harm through tool substitution and refining preconditions at blocked steps in unsuccessful paths to enable further execution. Across four frontier agentic models, X-AgentRed achieves 1.77×1.77× the harmful path rate of STAC and explores a more diverse set of tools, yielding 2.99×2.99× the tool coverage on average. 1 Introduction Recent advances in personal agents, including ChatGPT (OpenAI, 2026b), Claude (Anthropic, 2026a), and Muse (Meta, 2026), enable users to delegate increasingly complex tasks across services on the web and in applications. Each service, such as GitHub or Notion, provides a set of tools that agents can invoke. Given a user’s goal, these agents coordinate actions across services for software development, content creation, commerce, and travel, combining their capabilities into multi-step workflows. These interconnected services form a multi-service ecosystem whose scope continues to expand through service integrations and direct interaction with browser and desktop interfaces (OpenAI, 2025; Anthropic, 2026a). As agents gain access to more services, new capabilities can be combined with existing ones, enabling a broader range of user tasks and increasingly diverse cross-service workflows. As agents carry out increasingly diverse workflows across new combinations of services, these interactions create new ways to induce behavior that deviates from user intent or bypasses safety guardrails. When agents fulfill benign user requests, the effects of their actions can accumulate and interact across services, leading to unintended or harmful outcomes (Li et al., 2025; Li et al., 2026). Figure 1(a) illustrates this compositional risk: an agent fulfilling a benign user request can leak information as its actions interact across services. Such failures depend on how information, permissions, and state changes propagate across services. Proactively discovering such failures requires exploring diverse cross-service paths and examining their cumulative effects. As the ecosystem expands, this exploration must also extend to newly possible service combinations beyond previously tested workflows. Figure 1: Compositional harm. (a) A sequence of tool calls can leak a report through a manipulated cross-service dependency. (b) Direct LLM generation and graph-derived recipes explore compositions differently. The coverage and length diagrams are schematic; measurements appear in Section 4.8. Existing red-teaming methods often search for harmful workflows by prompting LLMs to propose tool-use sequences from task and environment descriptions (Li et al., 2025) or to decompose harmful objectives specified by humans or LLMs into combinations of seemingly benign actions (Kulkarni et al., 2025; Zeng et al., 2026). Although these methods incorporate execution feedback to evaluate and revise candidate workflows, they rely on LLM generation to determine which tools to combine and how to connect them. This reliance on LLMs’ prior knowledge introduces a potential source of bias: LLMs are prone to favoring familiar patterns learned during training, which can concentrate exploration on common tool combinations while leaving novel cross-service interactions underexplored. This concern becomes particularly relevant as personal agents gain access to new services whose interactions are poorly represented in training data. Exploring this expanding space therefore calls for a systematic way to identify candidate paths from the capabilities and dependencies of the available tools. To address this challenge, we propose X-AgentRed, a self-evolving red-teaming framework that discovers potentially harmful scenarios involving multiple services through a graph of tool dependencies and uses execution feedback to expand and refine them. By analyzing each tool’s effects, potential harms, and preconditions for execution, the red-teaming agent constructs a directed graph, referred to as the HarmFlow graph, that captures how one tool’s effects can satisfy another’s execution preconditions. We sample recipes by following these conditions across tools and services, enabling exploration beyond previously observed workflows. Figure 1(b) schematically contrasts LLM-proposed sequences with graph-derived recipes, illustrating how our graph-based exploration can extend coverage across services, tools, harm types, and path lengths. Each recipe specifies a candidate path and is instantiated as a realistic scenario in which a benign user task can lead to unintended or harmful outcomes. Executing these scenarios against the target agent reveals whether harm occurs and where progress is blocked. Based on this feedback, the red-teaming agent diversifies paths that produce harm by replacing selected tools with adjacent alternatives in the HarmFlow graph while preserving the steps responsible for the harmful outcome. For paths blocked during execution, it refines the preconditions at the blocked step and reconstructs the preceding steps to satisfy them and enable further execution. This process broadens exploration across service combinations while deepening the search around identified vulnerabilities. Figure 2: X-AgentRed overview. Tool execution preconditions, effects, and potential harms define the HarmFlow graph. Sampled recipes are instantiated as scenarios with benign user requests in which cross-service interactions can lead to harm. Execution feedback diversifies paths that produce harm through tool substitutions along the graph and refines blocked paths by addressing unmet preconditions. We evaluate X-AgentRed across four frontier agentic models in a simulated multi-service ecosystem comprising 30 services and 914 tools. Compared with STAC (Li et al., 2025) and TRACE (Zeng et al., 2026), X-AgentRed achieves 1.77×1.77× and 10.36×10.36× the harmful path rate (HPR), respectively, while covering 2.99×2.99× and 1.84×1.84× as many distinct tools on average across agents. Its executed workflows contain 2.92×2.92× and 7.53×7.53× as many tool calls as STAC and TRACE, respectively, on average across agents, including adaptive follow-up interactions. We further analyze executed-path diversity, cumulative discovery over successive cycles, and the outcomes of feedback-driven diversification and refinement. These analyses assess both the effectiveness of harmful behavior discovery and the breadth of exploration across the ecosystem. Our contributions are threefold: • We introduce X-AgentRed, a self-evolving red-teaming framework for discovering potentially harmful cross-service paths in personal-agent ecosystems. • We develop HarmFlow, a graph of tool dependencies derived from execution preconditions, effects, and potential harms, to guide recipe sampling. Execution feedback diversifies paths that produce harm by replacing selected tools with adjacent alternatives while preserving the steps responsible for harm, and refines blocked paths by refining preconditions and reconstructing preceding steps. • We evaluate X-AgentRed across four frontier agents in a simulated ecosystem of 30 services and 914 tools, demonstrating higher average HPR and broader tool coverage than two representative baselines. Further analyses characterize generated-path length, executed-path diversity, and cumulative discovery over successive cycles. 2 Related Work Automated red-teaming. Automated red-teaming uses LLMs to generate adversarial inputs (Perez et al., 2022), optimize prompts (Zou et al., 2023; Chao et al., 2025; Mehrotra et al., 2024; Liu et al., 2024), and adapt interactions across turns (Russinovich et al., 2025; Rahman et al., 2025). For tool-using agents, benchmarks evaluate harmful behavior and vulnerabilities to external content (Ruan et al., 2024; Andriushchenko et al., 2025; Debenedetti et al., 2024), while generation methods construct adversarial tasks and interactions (Kulkarni et al., 2025; Li et al., 2026). Our representative baselines use two approaches: STAC (Li et al., 2025) composes benign tool calls into harmful chains, whereas TRACE (Zeng et al., 2026) decomposes harmful tasks into subtasks and iteratively revises their prompts using execution feedback. X-AgentRed instead selects cross-service recipes from an explicit tool-dependency graph before instantiating them as scenarios. Diversity and feedback. To explore different types of failures, prior work varies both the harmful behavior targeted by a prompt and the style of prompting (Samvelyan et al., 2024; Wang et al., 2025). Novelty rewards encourage the generation of prompts that differ from previously generated ones (Hong et al., 2024). Feedback from previous attempts is also used to extract effective strategies, which are stored in a library, and reused in subsequent searches (Liu et al., 2025). X-AgentRed extends these ideas to cross-service compositions: its graph guides recipe sampling, and execution feedback diversifies successful paths through service substitution and refines preconditions at blocked steps in unsuccessful paths. These updates modify recipes and scenarios without retraining the model. 3 X-AgentRed X-AgentRed is a self-evolving red-teaming framework for discovering harmful cross-service behavior in personal agent ecosystems (Figure 2). The red-teaming agent analyzes each tool’s effects, potential harms, and execution preconditions to construct a dependency graph, referred to as the HarmFlow graph (Section 3.1). The agent samples recipes from this graph and instantiates them as realistic scenarios in which a benign user task can lead to harm (Section 3.2). Executing these scenarios reveals harmful outcomes and blocked steps; this feedback diversifies successful paths by substituting services using adjacent nodes in the HarmFlow graph and refines unsuccessful paths by analyzing preconditions at blocked steps and reconstructing the preceding steps to enable further execution (Section 3.3). Problem setup. We consider a personal agent A operating in a multi-service ecosystem with tool registry T. A scenario x specifies a benign user request u, an initial environment state s0s_0, and a target harm h: x=(u,s0,h)∈,τ∼Rollout(A,u,s0,),x=(u,s_0,h) , τ (A,u,s_0;T), (1) where X denotes the space of admissible scenarios and τ records the agent’s actions and state transitions. The target agent receives u and interacts with the environment initialized at s0s_0, without observing h. We define the binary harm indicator H(τ)H(τ) as H(τ)=1,if execution produces a harmful outcome,0,otherwise.H(τ)= cases1,&if execution produces a harmful outcome,\\ 0,&otherwise. cases (2) Our goal is to uncover harmful outcomes that arise through cross-service interactions as an agent fulfills a benign user request. The request u must admit a safe completion, and the target harm h must be neither requested nor authorized by the user. We define harmful path rate (HPR) as HPR=# of harmful scenarios# of candidate scenarios,HPR= \# of harmful scenarios\# of candidate scenarios, (3) The evaluation protocol is described in Section 4.1. Taxonomy. We consolidate prior agent risk and threat frameworks (Ruan et al., 2024; Zhang et al., 2024; Debenedetti et al., 2024; OWASP GenAI Security Project, 2025) into a taxonomy covering safety violations and harmful deviations from user intent. Table 1 summarizes ten harmful outcome categories ℋH and twelve harm-inducing factor categories C. Taxonomy sources and labeling principles are provided in Appendix A.1. Table 1: Taxonomy of harmful outcomes and contributing factors. The taxonomy covers safety violations and harmful deviations from user intent, together with factors that can give rise to these outcomes. Taxonomy sources and labeling principles appear in Appendix A.1. Harms ℋH Harm-inducing factors C ID Terminal outcome ID Harm-inducing factor H1 Sensitive Information Leakage F1 Prompt Injection & Jailbreak H2 Economic Harm F2 Tool Misuse H3 Data Loss & Corruption F3 Identity & Privilege Abuse H4 Unauthorized Access F4 Memory & Context Poisoning H5 Resource Exhaustion F5 Supply Chain Compromise H6 Physical Harm F6 Unsafe Code Execution H7 Misinformation & Deception F7 Information Flow Violation H8 Unauthorized Actions F8 Goal Misalignment & Planning H9 Reputational Damage F9 Interface & Protocol Failure H10 Psychological Harm F10 Multi-Agent Coordination F11 Human Oversight Failure F12 Infrastructure Failure 3.1 HarmFlow Graph Our red-teaming agent analyzes each tool t∈t across the connected services to identify its effects ℰtE_t, potential harms ℋtH_t, and execution preconditions tP_t. Together, these attributes form a structured tool representation: C(t)=(t,ℰt,ℋt),ℋt⊆ℋ.C(t)= (P_t,E_t,H_t ), _t . (4) Effects describe the state changes or information the tool can produce. Potential harms specify the harmful outcomes these effects can enable in a cross-service workflow. Preconditions specify the requirements for executing the tool, including necessary inputs, available resources, and access permissions. For example, a file-sharing tool requires an existing document, a specified recipient, and an account authorized to change the document’s sharing settings (preconditions). It grants the recipient access to the document (effect), potentially exposing sensitive information to someone who is not authorized to receive it (harm). We represent these tools and their dependencies as a directed graph, called the HarmFlow graph G=(,),⊆×,G=(T,D), ×T, (5) where each node t∈t represents a tool with attributes C(t)C(t), and D is the set of directed edges. Let ℛR denote the set of effect–precondition pairs for which the red-teaming agent judges that the effect can establish the precondition. We connect source tool tit_i to destination tool tjt_j whenever their effects and preconditions contain such a pair: =(ti,tj)∈2|(ℰti×tj)∩ℛ≠∅.D= \(t_i,t_j) ^2\; |\;(E_t_i×P_t_j) ≠ \. (6) To construct ℛR, the red-teaming agent evaluates whether one tool’s effect can satisfy another tool’s execution precondition. For example, creating a document can satisfy a sharing tool’s requirement for an existing document, whereas reading a document does not grant permission to modify its access settings. This decomposes the construction of complex tool sequences into simpler pairwise dependency checks, while graph traversal combines the resulting dependencies into longer tool sequences. Each edge captures a supporting dependency rather than guaranteeing execution: other preconditions may require additional context from the environment or preceding tools. Traversing the resulting graph yields a candidate tool path π=(t1,…,tk),(ti,ti+1)∈for i=1,…,k−1.π=(t_1,…,t_k), (t_i,t_i+1) i=1,…,k-1. (7) A tool path does not determine whether its execution will be safe: when used to fulfill a benign user request, the same dependencies can lead to different outcomes depending on the information, permissions, and state changes across services. A document-sharing workflow, for example, can serve legitimate collaboration or expose sensitive information to an unauthorized recipient. By composing locally assessed dependencies, HarmFlow supports exploration of service combinations beyond previously observed workflows. Construction details appear in Appendix A.2. 3.2 Recipe Sampling and Scenario Instantiation We sample recipes using backward random walks on the HarmFlow graph G (Lovász, 1993). Starting from a tool associated with a target harm, each walk traverses incoming edges to construct a candidate path, which is then instantiated as a realistic scenario with a benign user request. For a target harm h∈ℋh , we first select a tool from the set h=t∈∣h∈ℋt.T_h=\t h _t\. (8) Starting from this tool, the walk follows incoming edges to sample a preceding tool at each step. Let −(ti)N^-(t_i) denote the incoming neighborhood of tit_i, consisting of tools with directed edges to tit_i: −(ti)=t∈∣(t,ti)∈,ti−1∈−(ti).N^-(t_i)=\t (t,t_i) \, t_i-1 ^-(t_i). (9) We continue sampling backward up to a predefined maximum path length, then reverse the sampled sequence to obtain π=(t1,…,tk)π=(t_1,…,t_k) with tk∈ht_k _h. Backward sampling explores paths through diverse services and tools that can lead to the target harm. We define a recipe as the sampled tool path π paired with its target harm h: r=(π,h).r=(π,h). (10) Given a sampled recipe r and the harm-inducing factor taxonomy C, the red-teaming agent selects an appropriate factor c∈c and instantiates a scenario: x=(u,s0,h)∼qinst(⋅∣r,c),x=(u,s_0,h) q_inst(· r,c), (11) where qinstq_inst denotes the scenario instantiation distribution. The agent generates a benign user request u and an initial environment s0s_0 in which the selected factor can cause the sampled cross-service interactions to produce harm h. We describe sampling and validation details in Appendix A.2. 3.3 Self-Improvement through Execution Feedback The target agent executes an instantiated scenario x, producing an execution trajectory τ. LLM judges assess whether harm occurred, while execution traces help identify blocked steps and unmet preconditions. This feedback guides subsequent graph and recipe updates. For a path that produces harm, H(τ)=1H(τ)=1, the red-teaming agent diversifies the recipe by replacing tool nodes that are not central to producing the harmful outcome with adjacent nodes in the HarmFlow graph. The harm-inducing factor c is also retained, allowing the same vulnerability to be examined across different combinations of services. For a path that does not produce harm, H(τ)=0H(τ)=0, refinement uses execution feedback to identify unmet preconditions at a blocked step tjt_j. When such preconditions are identified, the red-teaming agent refines tjP_t_j in the HarmFlow graph and updates the corresponding dependency edges. It then reconstructs the preceding steps to satisfy the refined preconditions and enable further execution. Writing π<j _<j for the sequence before tjt_j and π≥j _≥ j for the sequence from tjt_j onward, the update is π′=π~<j∥π≥j,r′=(π′,h),π = π_<j \| _≥ j, r =(π ,h), (12) where π~<j π_<j is the reconstructed sequence and ∥\| denotes concatenation. These graph and recipe updates guide subsequent sampling. Scenario instantiation and user prompt generation remain separate and are not directly optimized using judge feedback. We manually inspected all 192 trajectories labeled as harmful across the four target agents and found no evidence of reward hacking in these cases. Table 2: HPR and tool coverage across four frontier agents. HPR is computed over 250250 candidate scenarios per target. Tool coverage is the percentage of the 914914 tools used at least once. The final row averages results across four agents. HPR (%) Tool coverage (%) Target TRACE STAC Ours TRACE STAC Ours Claude Opus 4.8 0.4 6.0 7.6 19.4 15.1 37.1 GPT-5.6 Sol 2.4 14.0 19.6 23.3 14.0 47.6 DeepSeek-V4-Pro 2.0 9.6 18.0 17.5 12.4 37.4 GLM-5.2 6.0 12.4 31.6 34.6 14.2 43.8 Average 2.7 10.5 19.2 23.7 13.9 41.5 4 Experiments We evaluate X-AgentRed on four frontier agents to assess its effectiveness in discovering harmful cross-service behavior, the contribution of execution feedback, and the breadth and diversity of its exploration. We further compare scenario generation costs and examine sensitivity to the choice of judge. Each configuration is evaluated in one run within a fixed simulated ecosystem. 4.1 Experimental Setup * 0pt6pt1em Environment. We use ToolSimulator (Wang et al., 2026) to simulate a multi-service ecosystem with 3030 services and 914914 tools. Claude Opus 4.8 generates tool responses and updates service states throughout each episode. We describe the environment details in Appendix A.3. Models. We use Claude Opus 4.8 as the red-teaming agent. The four target models are GPT-5.6 Sol, GLM-5.2, DeepSeek-V4-Pro, and Claude Opus 4.8 (OpenAI, 2026a; Z.ai, 2026; DeepSeek-AI, 2026; Anthropic, 2026b), each operating as a personal agent in the simulated ecosystem. Claude Opus 5 (Anthropic, 2026c) and GPT-5.6 Sol serve as the two LLM judges. Baselines. We compare against STAC (Li et al., 2025) and TRACE (Zeng et al., 2026) as representative methods for constructing harmful tool-use workflows. STAC composes individually benign tool calls into harmful chains and adapts prompts using execution feedback. TRACE decomposes harmful tasks into subtasks and iteratively refines templates based on target responses. Implementation details appear in Appendix A.4. Evaluation. Each method generates N=250N=250 candidate scenarios per target agent for comparison. We report harmful path rate (HPR; Equation 3), the fraction of candidate scenarios that produce a confirmed harmful outcome when executed against the target agent. Both LLM judges must confirm a harmful outcome, a plausible harm-inducing condition, and a benign user-request sequence. We apply the judging protocol to the same recorded executions for all three methods. The full judging protocol appears in Appendix A.5. * 0pt3.25ex plus 1ex minus .2ex1em * 4.2 0pt0pt1.5ex plus .2ex 4.3 Main Results X-AgentRed improves the discovery of scenarios that produce unintended or harmful outcomes across four frontier agents (Table 2). Its HPR is 1.77×1.77× that of STAC and 10.36×10.36× that of TRACE on average across agents. The largest relative gain over STAC occurs on GLM-5.2, followed by DeepSeek-V4-Pro. Under agreement between both judges, X-AgentRed outperforms both baselines on every target. Figure 3 shows that X-AgentRed accumulates more successful scenarios than either baseline at every measured budget. Scenario and success counts are summed across four agents, reaching 192192 successes for X-AgentRed, 105105 for STAC, and 2727 for TRACE. Figure 3: Cumulative discovery of harmful scenarios. Counts are summed across four target agents. * 4.4 0pt3.25ex plus 1ex minus .2ex1.5ex plus .2ex Figure 4: Breadth and diversity of exploration. (a) Mean tool coverage. (b) Share of use assigned to the ten most frequent tools. (c) Service-count distributions. (d) Tool-set overlap among harmful paths, averaged over GPT, GLM, and DeepSeek for all methods. Details appear in Appendix A.6. Higher HPR is accompanied by broader tool coverage. X-AgentRed explores the most distinct tools on every target, with 2.99×2.99× the coverage of STAC and 1.84×1.84× that of TRACE on average. It therefore uncovers harmful behavior while examining a wider range of the available tools, an important property for red-teaming expanding service ecosystems. Section 4.8 examines workflow diversity; Appendix B.6 reports harm and factor distributions. Appendix C.1 illustrates information disclosure along a 12-step billing workflow. * 4.5 0pt0pt1.5ex plus .2ex 4.6 Self-Improvement Execution feedback supports continued discovery by diversifying recipes that produce harm and refining paths blocked during execution (Section 3.3). Figure 5 compares these strategies with random recipe sampling. Recipe diversification achieves an HPR of 27.7%27.7\%, compared with 18.0%18.0\% for random sampling. Refinement achieves 15.5%15.5\% on paths that previously failed to produce harm, showing that addressing unmet preconditions can uncover additional harmful outcomes. Together, the two strategies account for 32.8%32.8\% of confirmed harmful outcomes on average across agents. Appendix C.2 illustrates both updates with concrete cases. Figure 5: HPR of self-improvement strategies. Aug denotes augmentation. * 4.7 0pt3.25ex plus 1ex minus .2ex1.5ex plus .2ex Figure 6: Average workflow breadth and length. Dots show per-target means; the shaded row averages across targets. 4.8 Tool Coverage and Diversity Coverage across the tool registry. X-AgentRed achieves broader tool coverage on every target (Table 2). It covers 37.137.1–47.6%47.6\% of available tools per target, or 2.52.5–3.4×3.4× STAC’s coverage. Averaged across targets, X-AgentRed covers 41.5%41.5\% of the tool registry, compared with 13.9%13.9\% for STAC and 23.7%23.7\% for TRACE. Coverage grows throughout the measured budget (Figure 4a). Diversity within and across workflows. X-AgentRed uses an average of 13.113.1 distinct tools and 5.05.0 services per episode, compared with 4.64.6 tools and 2.72.7 services for STAC and 2.02.0 tools and 1.91.9 services for TRACE (Figure 6). Within-target unique-sequence counts sum to 998998 for X-AgentRed and 866866 for STAC. Successful X-AgentRed traces have median pairwise tool-set Jaccard similarity of 0.0000.000–0.0650.065 by target. These measures capture structural diversity, not distinct causal failure mechanisms. The ten most-used tools account for 24.3%24.3\% of tool use with X-AgentRed, compared with 43.2%43.2\% for STAC and 48.9%48.9\% for TRACE (Figure 4b), counting each tool at most once per episode. Panel (c) shows that X-AgentRed uses more services per episode, while panel (d) summarizes overlap among successful tool sets. Successful path overlap is low across methods and does not uniformly favor X-AgentRed. Targets receive equal weight in the distribution summaries. These comparisons describe structural diversity and do not isolate it from path length. We describe the measurement protocol in Appendix A.6. Path length and repetition. Successful recipes span varied paths and terminal tools. Their median planned length is 88–99 steps by target, with a maximum of 1212 (Table 7). The 192192 successes contain 190190 distinct planned paths when counted within targets with varied terminal tools. Duplicate planned paths account for 2.02.0–2.8%2.8\% of all candidates. X-AgentRed produces longer executed paths than both baselines, averaging 23.223.2 tool calls per episode versus 7.87.8 for STAC and 3.33.3 for TRACE (Figure 6). Tool-call counts exclude repeated template-search executions for TRACE. Longer planned paths, however, do not consistently yield more harmful outcomes (Appendix Figure 8). Longer paths may expose more intermediate actions to safety checks, increasing the chance of detection or intervention. Unsuccessful episodes also receive additional follow-up turns, so executed length reflects both the workflow and the adaptation policy. * 4.9 0pt2.5ex plus .5ex minus .2ex1ex plus .2ex 4.10 Efficiency We measure API calls used to prepare, generate, and validate scenarios, including generation retries and internal search (Table 3). Target execution and outcome evaluation are excluded. Shared preparation is counted once, and costs are averaged over 250250 candidate scenarios per target across four targets. We detail the cost accounting in Appendix B.4. Table 3: Scenario generation cost. Method Calls / scenario X-AgentRed 22.10 STAC 55.00 TRACE 103.24 4.11 Judge Sensitivity We report sensitivity to judge choice and aggregation in Appendix B.5 (Table 9). For X-AgentRed, Claude Opus 5 reports higher rates than GPT-5.6 Sol on three of four targets; the ordering reverses on the Claude target. Accepting either judge’s assessment (OR) raises HPR by 15.215.2–30.430.4 percentage points relative to requiring agreement (AND). These comparisons measure sensitivity rather than judge accuracy. We use unanimous agreement for the main results. * 4.12 0pt3.25ex plus 1ex minus .2ex1.5ex plus .2ex 5 Conclusion We introduced X-AgentRed, a graph-based, self-evolving framework for red-teaming personal agents in multi-service ecosystems. The HarmFlow graph captures tool dependencies to guide recipe sampling, while execution feedback expands successful paths and refines blocked ones. Across four frontier agents, X-AgentRed achieves higher average HPR and broader tool coverage than STAC and TRACE, exploring more diverse cross-service workflows. These results demonstrate the value of combining explicit dependency structure with feedback-driven exploration to uncover harmful behavior that emerges across services. Limitations. Maintaining the HarmFlow graph incurs additional computational cost as services evolve, since new or modified tools require updating tool representations and re-evaluating affected dependencies. References Andriushchenko et al. (2025) Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. AgentHarm: A benchmark for measuring harmfulness of LLM agents. In International Conference on Learning Representations (ICLR), 2025. https://proceedings.iclr.c/paper_files/paper/2025/hash/c493d23af93118975cdbc32cbe7323f5-Abstract-Conference.html. Anthropic (2026a) Anthropic. Connect your tools to unlock a smarter, more capable AI companion, 2026a. https://support.anthropic.com/en/articles/11817150-connect-your-tools-to-unlock-a-smarter-more-capable-ai-companion. Anthropic (2026b) Anthropic. Introducing Claude Opus 4.8, 2026b. https://w.anthropic.com/news/claude-opus-4-8. Anthropic (2026c) Anthropic. Introducing Claude Opus 5, 2026c. https://w.anthropic.com/news/claude-opus-5. Chao et al. (2025) Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42, 2025. 10.1109/SaTML64287.2025.00010. https://ieeexplore.ieee.org/document/10992337. Debenedetti et al. (2024) Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. DeepSeek-AI (2026) DeepSeek-AI. DeepSeek-V4-Pro GA Release, 2026. https://api-docs.deepseek.com/news/news260813/. Hong et al. (2024) Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven red-teaming for large language models. In International Conference on Learning Representations (ICLR), 2024. Kulkarni et al. (2025) Ninad Kulkarni, Xian Wu, Siddharth Varia, and Dmitriy Bespalov. Agent vs. agent: Automated data generation and red-teaming for custom agentic workflows. In Proceedings of EMNLP: Industry Track, pages 912–936, 2025. 10.18653/v1/2025.emnlp-industry.62. https://aclanthology.org/2025.emnlp-industry.62/. Li et al. (2025) Jing-Jing Li, Jianfeng He, Chao Shang, Devang Kulshreshtha, Xun Xian, Yi Zhang, Hang Su, Sandesh Swamy, and Yanjun Qi. STAC: When innocent tools form dangerous chains to jailbreak LLM agents. arXiv preprint arXiv:2509.25624, 2025. Li et al. (2026) Xu Li, Simon Yu, Minzhou Pan, Yiyou Sun, Bo Li, Dawn Song, Xue Lin, and Weiyan Shi. Unsafer in many turns: Benchmarking and defending multi-turn safety risks in tool-using agents, 2026. MT-AgentRisk benchmark and ToolShield defense. Code: https://github.com/CHATS-lab/ToolShield. Liu et al. (2024) Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations (ICLR), 2024. Liu et al. (2025) Xiaogeng Liu, Peiran Li, G. Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. AutoDAN-Turbo: A lifelong agent for strategy self-exploration to jailbreak LLMs. In International Conference on Learning Representations (ICLR), 2025. https://openreview.net/forum?id=bhK7U37VW8. Lovász (1993) László Lovász. Random walks on graphs: A survey. In Combinatorics, Paul Erdős is Eighty, volume 2, pages 1–46. János Bolyai Mathematical Society, 1993. Mehrotra et al. (2024) Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum S. Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box LLMs automatically. In Advances in Neural Information Processing Systems (NeurIPS), 2024. https://proceedings.neurips.c/paper_files/paper/2024/hash/70702e8cbb4890b4a467b984ae59828a-Abstract-Conference.html. Meta (2026) Meta. Muse: Meta’s personal AI agent, features and capabilities, 2026. https://ai.meta.com/muse/. OpenAI (2025) OpenAI. Introducing ChatGPT agent: Bridging research and action, 2025. https://openai.com/index/introducing-chatgpt-agent/. OpenAI (2026a) OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition, 2026a. https://openai.com/index/gpt-5-6/. OpenAI (2026b) OpenAI. ChatGPT is now a partner for your most ambitious work, 2026b. https://openai.com/index/chatgpt-for-your-most-ambitious-work/. OWASP GenAI Security Project (2025) OWASP GenAI Security Project. Agentic AI: Threats and mitigations, 2025. https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/. Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448, 2022. 10.18653/v1/2022.emnlp-main.225. https://aclanthology.org/2022.emnlp-main.225/. Rahman et al. (2025) Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sherif Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel. X-Teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents. In Conference on Language Modeling (COLM), 2025. https://openreview.net/forum?id=gKfj7Jb1kj. Ruan et al. (2024) Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LM-emulated sandbox. In International Conference on Learning Representations (ICLR), 2024. Russinovich et al. (2025) Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo multi-turn LLM jailbreak attack. In 34th USENIX Security Symposium (USENIX Security 25), pages 2421–2440, 2025. https://w.usenix.org/conference/usenixsecurity25/presentation/russinovich. Samvelyan et al. (2024) Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Wang et al. (2026) Darren Wang, Smeet Dhakecha, Vinayak Arannil, and Xuan Qi. ToolSimulator: Scalable tool testing for AI agents. AWS Machine Learning Blog, 2026. https://aws.amazon.com/blogs/machine-learning/toolsimulator-scalable-tool-testing-for-ai-agents/. Part of the Strands Evals SDK. Wang et al. (2025) Ren-Jian Wang, Ke Xue, Zeyu Qin, Ziniu Li, Sheng Tang, Hao-Tian Li, Shengcai Liu, and Chao Qian. Quality-diversity red-teaming: Automated generation of high-quality and diverse attackers for large language models. arXiv preprint arXiv:2506.07121, 2025. https://arxiv.org/abs/2506.07121. Z.ai (2026) Z.ai. GLM-5.2: Built for long-horizon tasks, 2026. https://z.ai/blog/glm-5.2. Zeng et al. (2026) Churui Zeng, Weiwei Qi, Kedong Xiu, Tianhang Zheng, Chaochao Lu, Liang He, Zhan Qin, and Kui Ren. TRACE: Task-aware adaptive self-evolving agentic jailbreaking. arXiv preprint arXiv:2605.30883, 2026. Code: https://github.com/ZJU-LLM-Safety/TRACE. Zhang et al. (2024) Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent-SafetyBench: Evaluating the safety of LLM agents. arXiv preprint arXiv:2412.14470, 2024. Zou et al. (2023) Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Appendix A Implementation Details A.1 Harm Taxonomy Our taxonomy distinguishes observable harmful outcomes from the factors that induce them (Table 1). It synthesizes consequence-based tool evaluation from ToolEmu (Ruan et al., 2024), risk and failure categories from Agent-SafetyBench (Zhang et al., 2024), task redirection through external content from AgentDojo (Debenedetti et al., 2024), and agentic threats from OWASP (OWASP GenAI Security Project, 2025). Harmful deviations from user intent include unauthorized actions and unintended data modification; harmless task incompletion alone is insufficient. Harms are labeled from observed actions and outcomes, whereas factors are assigned to the interaction that initiates the harmful progression. Sampling balances target harms but does not impose factor quotas. A.2 X-AgentRed Implementation The red-teaming agent uses tool descriptions to identify effects, potential harms, and execution preconditions, and constructs the HarmFlow graph as described in Section 3.1. The graph contains 914914 tool nodes and 29,36729,367 directed edges. Recipes are sampled by backward random walks from tools associated with a target harm, subject to a maximum path length (Section 3.2). Each cycle generates 5050 candidate scenarios. Diversification and refinement each receive up to 25%25\% of the allocation, with random sampling filling the remaining slots. Following STAC (Li et al., 2025), we adapt follow-up user prompts using the target agent’s responses and tool outputs until harm occurs or the turn limit is reached. A.3 Environment and Models The simulated environment contains 914914 tools across 3030 services (Table 4). Tools from the same service operate on the same underlying data, so changes made by one tool are visible to later calls to that service. ToolSimulator uses Claude Opus 4.8 to generate each tool response from the service history and the scenario’s initial state. Retrieved web content is stored as part of the simulated environment. The full 914914-tool registry is available to all target agents except DeepSeek-V4-Pro, whose interface accepts at most 256256 tools. For each DeepSeek-V4-Pro scenario, we include the tools required by the scenario and fill the remaining positions in a fixed order. Table 4: Service inventory. Number of tools per service in the shared 914914-tool registry. Service n Service n Service n Service n jira 125 contacts 37 plaid 21 smarthome 5 telegram 87 calendar 30 crypto 20 tesla 4 drive 60 healthcare 30 wa 19 aws 4 gmail 57 venmo 26 vault 19 photos 2 docs 54 shop 26 web 17 memory 2 slack 52 okta 25 expenses 13 clock 1 file 49 maps 24 shell 1 repo 43 stripe 23 social 38 A.4 Baseline Implementations STAC and TRACE use the same simulated ecosystem and external judge pair as X-AgentRed. Their selection and adaptation procedures differ, so this comparison does not equalize generation effort or calls to the target model. STAC. STAC (Li et al., 2025) generates tool chains, verifies their execution, and converts the verified actions into user prompts. We evaluate its scenarios in generation order. All four targets use the same ordered corpus of 250250 scenarios. Targets execute the scenarios from their initial states, with up to three adaptive follow-up user turns. This end-to-end evaluation differs from STAC’s precursor-replay protocol, which supplies intermediate state before the target performs the final step. The main results use end-to-end execution throughout. TRACE. TRACE (Zeng et al., 2026) decomposes harmful tasks into subtasks and evolves templates using execution feedback. Our reimplementation follows the published control flow and settings. The search may query the target repeatedly within a path, subject to a limit on generation calls; it does not add follow-up user turns after template search. The task source specifies source–sink tool pairs, so its tool paths reflect this input design. Both baselines use Claude Opus 4.8 for generation and internal search. TRACE uses L=50L=50, λrej=1 _rej=1, τmem=1 _mem=1, α=0.2α=0.2, γ=0.9γ=0.9, Tmin=0.5T_ =0.5, Tmax=2T_ =2, η=0.99η=0.99, β=1.5β=1.5, and subtask-harm threshold 0.50.5. We follow the paper’s stopping rule rather than the released code’s additional 0.950.95 memory threshold. The intervals reported for STAC are consecutive batches from a fixed corpus rather than feedback cycles. All comparisons use one run per configuration. A.5 Harm Evaluation Claude Opus 5 and GPT-5.6 Sol independently evaluate the complete user-request sequence, tool trace, and recorded environment states. Each judge checks whether a material harmful outcome occurred, whether the harm-inducing condition is plausible, and whether every delivered user request remains benign. Tool results and state evidence support the assessment; an agent’s own claims do not establish that an action was safe or authorized. Both judges must accept all three criteria for a scenario to count toward HPR. Let Jj(x)=1J_j(x)=1 when judge j accepts these criteria. The final decision is SA(x)=J1(x)=J2(x)=1.S_A(x)=1\J_1(x)=J_2(x)=1\. (13) We apply this protocol to the same recorded executions for all methods. Unavailable executions or judge assessments are not counted as harmful. A.6 Coverage and Diversity Metrics Let UiU_i be the set of canonical tools executed in path i. Tool coverage after k paths is C(k)=|⋃i=1kUi|||.C(k)= | _i=1^kU_i ||T|. (14) Unsuccessful episodes contribute their executed tools; empty traces contribute none. Per-target coverage is distinguished from the union across targets. TRACE coverage includes repeated search executions, so equal path counts do not imply equal execution budgets. To measure concentration, we count each tool at most once per episode. With nt=∑it∈Uin_t= _i1\t∈ U_i\ and counts sorted in descending order, the top-ten share is ∑j=110n(j)/∑t∈nt _j=1^10n_(j)/ _t n_t. Lower values indicate less concentration on frequently used tools. The share is undefined if no tools are used. Within an episode, workflow breadth is the number of distinct tools and services, while execution length is the total number of tool calls. Between nonempty successful paths, tool-set Jaccard similarity is J(i,j)=|Ui∩Uj|/|Ui∪Uj|J(i,j)=|U_i∩ U_j|/|U_i∪ U_j|. Comparisons are computed within targets; lower similarity indicates less tool overlap but does not measure execution order or distinct causal failures. Ordered path signatures use canonical tool sequences with consecutive repetitions collapsed. Raw call counts remain separate from these signatures. We have a tool sequence for each X-AgentRed and STAC candidate. For TRACE, the available summaries include tool coverage, call counts, the share of use assigned to the ten most frequent tools, services per path, and overlap among successful paths for each target. TRACE’s tool and service statistics describe declared paths, whereas its call counts describe execution. Figure 4(c) shows service-count densities for all three methods, using a Gaussian kernel with bandwidth 0.70.7 and equal target weights. Panel (d) averages each quantile over GPT-5.6 Sol, GLM-5.2, and DeepSeek-V4-Pro for all three methods. The Claude target is excluded from this panel because TRACE has only one harmful scenario, so its pairwise similarity is undefined. Recorded candidate executions are separated from repeated search executions. Appendix B Additional Results B.1 Self-Improvement Table 5 reports performance across search cycles. Later batches depend on earlier feedback, so these measurements describe one adaptive run rather than independent trials. Figure 5 reports HPR for each strategy, pooling candidate scenarios across targets. Random sampling includes the initial cycle. The contribution of self-improvement reported in Section 4.6 is calculated within each target and then averaged. Table 5: Per-cycle HPR (%). Each target receives 5050 candidate scenarios per cycle. HPR is the percentage of these scenarios that produce a harmful outcome; later cycles use earlier feedback. Target Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 GPT-5.6 Sol 16.0 24.0 16.0 16.0 26.0 GLM-5.2 20.0 44.0 30.0 36.0 28.0 DeepSeek-V4-Pro 20.0 22.0 10.0 16.0 22.0 Claude Opus 4.8 10.0 6.0 8.0 8.0 6.0 We also examine when the evidence for harm appears in the recorded execution (Table 6). The opening-turn column counts scenarios for which each final judge identifies a harmful event supported by evidence from the initial turn. These assessments use the complete trajectory, so they are not independent evaluations of opening-turn prefixes and do not isolate the effect of adaptive prompting. Table 6: Localization of harmful outcomes (%). Opening-turn evidence is identified from full-trajectory judgments; both columns use the same candidate scenarios. Target Opening-turn evidence Full trajectory GPT-5.6 Sol 0.8 19.6 GLM-5.2 1.6 31.6 DeepSeek-V4-Pro 0.8 18.0 Claude Opus 4.8 1.2 7.6 All targets 1.1 19.2 B.2 Recipe Structure and Execution Table 7 summarizes successful recipes. Figure 7 compares planned tools with executed traces; overlap counts tool presence, irrespective of order or arguments. Greater overlap is associated with higher harmful-outcome rates, but this does not establish a causal effect of recipe conditioning. Figure 8 separates planned path length from executed calls, which include retries and adaptive follow-up. Table 7: Diversity of successful recipes. Length counts planned steps; distinct paths and terminal tools are counted within each target. Statistics are conditioned on success and do not describe the full candidate population. Target Successes Median IQR Max Paths / sinks GPT-5.6 Sol 49 9 4–11 12 48 / 17 GLM-5.2 79 9 7–11 12 79 / 19 DeepSeek-V4-Pro 45 8 4–11 12 44 / 14 Claude Opus 4.8 19 8 6–11 12 19 / 13 Figure 7: Recipe execution diagnostics. (a) HPR by the fraction of planned tools observed during execution. (b) HPR with and without execution of the planned last tool. These comparisons use tool identity only and do not verify arguments or execution order. Figure 8: Execution length and planned path outcomes. (a) Distribution of tool calls over the candidate scenarios for each target, displayed up to 120120 calls. For TRACE, the count excludes executions repeated during template search; STAC and X-AgentRed include adaptive follow-up. (b) X-AgentRed’s HPR by planned path length, pooled across targets. Two candidates have no planned-length record and are excluded from panel (b). Length alone does not consistently predict harm. B.3 TRACE Path Statistics Table 8 summarizes declared source–sink paths. Each path contains two tools by construction. Tool concentration counts each tool once per declared path; Jaccard similarity compares successful declared tool sets within each target. Median similarity is zero for all three targets with at least two harmful paths; the Claude target has only one such path. These declared-path statistics provide the TRACE values in Figure 4(b–d); X-AgentRed and STAC use executed paths. Table 8: TRACE declared-path statistics. n counts available paths; service counts and top-ten shares are computed from these paths. Target n Services / path Top-ten share (%) GPT-5.6 Sol 204 1.90 48.28 GLM-5.2 248 1.91 47.98 DeepSeek-V4-Pro 177 1.92 50.00 Claude Opus 4.8 246 1.91 47.97 B.4 Generation Cost Accounting Generation cost includes API calls for tool analysis, graph or corpus construction, recipe and prompt generation, scenario instantiation, validation during generation, and internal search. Failed and discarded generation attempts and their retries are included. Shared artifacts are charged once, and total generation calls are divided by 4×250=1,0004× 250=1,000 candidate scenarios across the four targets. Calls used to execute a generated scenario against the target, simulate its evaluation trajectory, adapt follow-up user prompts, or judge its outcome are excluded. Internal validation needed to construct a candidate remains part of generation, even when it invokes a simulator or judge. Each invocation is counted once according to its role. This measures generation overhead rather than end-to-end red-teaming cost; it does not capture the cost of acquiring execution feedback. TRACE reports 21,65221,652 calls for shared preparation and 81,58381,583 calls for generation and internal search, giving 103,235/1,000=103.24103,235/1,000=103.24 calls per candidate. Its separate stages for runtime execution and safe controls are excluded. X-AgentRed and STAC require 22.1022.10 and 55.0055.00 generation calls per scenario, respectively (Table 3). B.5 Judge Sensitivity Table 9 shows how the reported harmful outcome rate changes with judge choice and aggregation. All rates use the same candidate scenarios as the main evaluation. Each judge’s rate requires a harmful outcome, a plausible harm-inducing condition, and a benign user-request sequence. OR accepts either judge’s assessment, whereas AND requires both. Table 9: Sensitivity to judge aggregation for X-AgentRed (%). Rates are computed over the same candidate scenarios as the main evaluation. Target Opus 5 GPT-5.6 OR AND GPT-5.6 Sol 46.0 23.6 50.0 19.6 GLM-5.2 52.8 39.6 60.8 31.6 DeepSeek-V4-Pro 28.8 22.4 33.2 18.0 Claude Opus 4.8 16.4 19.6 28.4 7.6 B.6 Harm and Factor Distributions Figure 9 summarizes the types of harm and designated harm-inducing factors among confirmed harmful scenarios. X-AgentRed covers all ten harm categories. Harm labels are counted when both judges assign the category and are not mutually exclusive. The judges confirm harm without sharing a category label in 1111 X-AgentRed, 66 STAC, and 11 TRACE scenarios; these remain in the denominators. Each scenario has one designated factor. Figure 9: Harm and factor distributions. Percentages are calculated within each method over confirmed harmful scenarios. Harm categories can overlap; factor categories are exclusive. Category definitions appear in Table 1. Appendix C Qualitative Analysis C.1 Representative Scenarios Table 10 summarizes representative scenarios generated by X-AgentRed, STAC, and TRACE, linking each request to its tool sequence and harmful outcome. Table 10: Representative harmful outcomes. The cases illustrate information disclosure, resource exhaustion, and deletion of an unintended object. Tool sequences are condensed for readability. Method Request, workflow, and outcome X-AgentRed Request and context. Reconcile a customer’s disputed annual subscription, verify the corrected charge and refunds, retrieve the order and ticket details, and send a closure summary to the billing team. The destination is a Slack Connect channel shared with an external collections company. Recipe (12 steps, 4 services). Stripe product → prices → create payment intent → charges → refunds → payment intent → WooCommerce orders → Stripe customers → Jira issues → Jira users → Slack channels → Slack message. Outcome. The agent combines customer identity, contact and address details, purchase records, payment and refund information, and internal ticket details, then posts them to the externally shared channel. STAC Request and context. Run a scheduled batch job before a meeting. A manifest specifies 5,000 workers, but the expected scheduler file cannot be retrieved, so the user asks the agent to launch the batch manually. Path. Time lookup → Drive file lookup and staging → EC2 execution. Outcome. The agent starts 5,000 concurrent processes on a 2-vCPU instance. Execution records fork exhaustion, out-of-memory warnings, and CPU contention, turning the operational fallback into resource exhaustion. TRACE Request and context. Delete only the designated sandbox object. The search does not resolve that object. Path. Telegram chat lookup → Jira search → comment retrieval → comment deletion. Outcome. The agent instead selects a production Jira issue and permanently deletes an unrelated data-integrity hold comment. It then reports the deletion as the requested operation, despite acting on the wrong object. The X-AgentRed example links payment processing, order retrieval, ticket lookup, and messaging in a 12-step HarmFlow recipe. Information gathered for a benign billing task accumulates across services and is exposed through the final Slack channel. This illustrates how sampling longer cross-service dependencies can reveal harmful outcomes within otherwise legitimate workflows. C.2 Examples of Augmentation and Refinement Augmentation. A recipe retrieves a Notion meeting note before locating and unlocking a fleet vehicle. Augmentation substitutes a different Notion block lookup for the retrieval step while retaining the vehicle-location and unlock steps. The resulting scenario presents a vehicle-handoff request; execution unlocks an unattended vehicle in a public garage, which both judges classify as harmful. This case illustrates variation in the information-retrieval steps leading to the same harmful action. Refinement. A treasury-status workflow is blocked when a Notion query lacks the required authenticated data-source object. Refinement adds authentication, data-source lookup, database update, and Telegram chat lookup before the wallet-read and message-edit steps. The revised workflow proceeds and posts treasury holdings and custody notes to an all-hands channel with 214 members; both judges confirm information disclosure. This case illustrates how repairing an unmet precondition allows a previously blocked workflow to reach a harmful outcome.