Paper deep dive
Towards Risk-free AI Agent Deployment
Yintong Huo, Rangeet Pan, Abhik Roychoudhury
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/23/2026, 2:04:55 AM
Summary
This paper addresses the risks of deploying LLM-based AI agents in organizational workflows, advocating for a systematic approach to testing and debugging based on agent trajectories. It identifies key challenges in testing, including the oracle problem, non-determinism, trajectory validation, and lack of adequacy metrics. It also discusses debugging methodologies such as trajectory management, failure attribution, and self-evolution, culminating in a deployment-readiness checklist to ensure trustworthy and risk-free agent deployment.
Entities (20)
Relation Signals (18)
Yintong Huo → affiliatedwith → Singapore Management University
confidence 100% · YINTONG HUO, Singapore Management University, Singapore
Rangeet Pan → affiliatedwith → IBM T.J. Watson Research Center
confidence 100% · RANGEET PAN, IBM T.J. Watson Research Center, USA
Abhik Roychoudhury → affiliatedwith → National University of Singapore
confidence 100% · ABHIK ROYCHOUDHURY, National University of Singapore, Singapore
Agent Trajectory → usedfor → Testing
confidence 95% · Trajectories are available for any agent, and many failures are visible only in the trajectory. To make agents deployable and sustainable, we advocate agent testing and debugging
Agent Trajectory → usedfor → Debugging
confidence 95% · The analogous artifact for an LLM-based agent is its trajectory... building the debugging layer on top of them yields methodologies
ReAct Architecture → introduces → Testing Challenges
confidence 90% · The ReAct architecture introduces profound testing and debugging challenges.
ReliabilityBench → applies → Chaos Engineering
confidence 85% · ReliabilityBench [14] applies chaos engineering to single-agent systems
MAST → catalogs → Behavioral Failure Modes
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM-based agents are rapidly moving from research prototypes into the core business processes of organizations, but these agents pose deployment risks to security, compliance, and functionality. In this article, we argue that risk-free deployment must be grounded in the agent's trajectory: the recorded sequence of reasoning steps, tool invocations, and environmental observations. Trajectories are available for any agent, and many failures are visible only in the trajectory. To make agents deployable and sustainable, we advocate agent testing and debugging as a systematic research direction for detecting and mitigating these risks. This article begins with the challenges of testing agents, including the oracle problem, non-determinism, trajectory validation, and the absence of adequacy metrics. We then turn to debugging agents, from automated failure attribution to repair and self-evolution. We distill these directions into a practical deployment-readiness checklist covering the full deployment lifecycle. Finally, we identify open problems, i.e., formal adequacy metrics, root-cause attribution over long-horizon trajectories, and the reliability of self-evolving agents, that the community must address to enable trustworthy agent deployment.
Tags
Links
- Source: https://arxiv.org/abs/2608.16411v1
- Canonical: https://arxiv.org/abs/2608.16411v1
Trouble viewing inline? Open PDF directly →
Full Text
48,765 characters extracted from source content.
Expand or collapse full text
Towards Risk-free AI Agent Deployment YINTONG HUO, Singapore Management University, Singapore RANGEET PAN, IBM T.J. Watson Research Center, USA ABHIK ROYCHOUDHURY, National University of Singapore, Singapore LLM-based agents are rapidly moving from research prototypes into the core business processes of orga- nizations, but these agents pose deployment risks to security, compliance, and functionality. In this article, we argue that risk-free deployment must be grounded in the agent’s trajectory: the recorded sequence of reasoning steps, tool invocations, and environmental observations. Trajectories are available for any agent, and many failures are visible only in the trajectory. To make agents deployable and sustainable, we advocate agent testing and debugging as a systematic research direction for detecting and mitigating these risks. This article begins with the challenges of testing agents, including the oracle problem, non-determinism, trajectory validation, and the absence of adequacy metrics. We then turn to debugging agents, from automated failure attribution to repair and self-evolution. We distill these directions into a practical deployment-readiness checklist covering the full deployment lifecycle. Finally, we identify open problems, i.e., formal adequacy metrics, root-cause attribution over long-horizon trajectories, and the reliability of self-evolving agents, that the community must address to enable trustworthy agent deployment. CCS Concepts:• Computing methodologies→Artificial intelligence; Artificial intelligence;• Software and its engineering→ Software creation and management. Additional Key Words and Phrases: AI Agent, Testing, Debugging, Deployment Risk ACM Reference Format: Yintong Huo, Rangeet Pan, and Abhik Roychoudhury. 2018. Towards Risk-free AI Agent Deployment. J. ACM 37, 4, Article 111 (August 2018), 12 pages. https://doi.org/X.X 1 Industrial Transformation Agents have had a significant impact not only on computing, but also on society at large. Agents provide autonomous decision-making via the use of tools, memory, and planning. It is thus quite significant that many manual or human-guided activities in computing, such as coding, code review, and system validation, are being taken over by agents. In software engineering per se, interest in agents is a few years old at this point. The main interest started with the design of coding agents, as it was found that Large Language Models (LLMs) are capable of vibe-coding and generating code for natural language requirements. Soon afterwards, there were suggestions for invoking tools such as command-line utilities (bash tools) and program analysis tools to support code generation, program repair, and feature-addition tasks, i.e., generative tasks. A commentary on coding agents and the role of trust in using coding agents appears in a recent Communications of the ACM article [29]. Authors’ Contact Information: Yintong Huo, Singapore Management University, Singapore, Singapore, ythuo@smu.edu.sg; Rangeet Pan, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA, rangeet.pan@ibm.com; Abhik Roychoudhury, National University of Singapore, Singapore, Singapore, abhik@nus.edu.sg. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-735X/2018/8-ART111 https://doi.org/X.X J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. arXiv:2608.16411v1 [cs.SE] 17 Aug 2026 111:2Trovato et al. Agent TestingAgent DebuggingDeployment Readiness Test Oracle Problem Trajectory Validation End-to-end Testing Test Adequacy Metrics Non-deterministic Behavior Non-functional Requirements Agent-specific Test Framework Trajectory Management Failure Diagnosis Automated Failure Attribution Debugging Techniques Self-evolving Agents Phase 1: Pre-deployment Phase 2: Deployment Phase 3: Post-deployment Internal Reasoning Response/ Action Tool Invocation Environment Observations Agent Trajectory 1 2 3 Reasoning Synergizing testing, debugging, and lifecycle management to enable trustworthy agent deployment and risk mitigation Fig. 1. The overview of this article. The roadmap to risk-free deployment includes agent testing, agent debugging, and deployment readiness checklists. While coding agents from software engineering have evoked significant interest in the broader generative AI and agentic AI communities, they are only one class of agents. The broader automation possibilities from agentic AI are not restricted to coding agents. Any organization with a significant business process or workflow can obtain potential productivity gains by automating parts of that workflow. As a simple example, a bank can automate part of its operational workflow by automating the loan approval task via a curated AI agent. However, such agentification of organizational workflows is currently ad hoc. If loan approval is done by an agent, it can introduce security, compliance, or functionality-breaking risks into the rest of the bank’s processes. Currently, such risks of agentification are not being actively studied or understood; instead, specific components of an organization’s workflow are being agentified in isolation. The aim of this article is to present a suite of ideas, techniques, and technologies that prompt us to consider the risk-free deployment of agents. To achieve such risk-free deployment, one of the first steps is to systematically test and debug AI agents. Such testing and debugging methodologies should be applicable to any agent, not only coding agents. Furthermore, ideally, the testing and debugging methodologies should work with agents using both open-weight and closed models. This is possible if the methodologies are based on agent trajectories. As shown in Fig. 1, in this paper, we thus present a suite of agent testing and debugging methodologies to enable the risk-free deployment of agents in organizations. We also discuss the impact of agentification on the rest of the organizational workflow. The techniques and ideas presented can provide an evaluation layer that supports the risk-free agentification of an organizational workflow. 2 Agents as a Reactive System Traditional software is deterministic: given the same input, the software produces the same output. Agents, by contrast, are fundamentally reactive and non-deterministic systems. They perceive an environment, update their internal state based on observations and reasoning, and generate actions that change the environment, creating a feedback loop that can persist for dozens or hundreds of steps. Many agents realize this loop through a ReAct (Reason-and-Act) architecture, which interleaves reasoning with tool-based actions. The ReAct architecture introduces profound testing and debugging challenges. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Towards Risk-free AI Agent Deployment111:3 An agentic system’s execution can be represented as a trajectory, i.e., a sequential record of the agent’s thinking process, tool invocations, environmental observations, and responses. Each step in a trajectory contains four critical types of information: the agent’s internal reasoning, its response or proposed action, the concrete actions taken (such as code modifications or tool use), and the resulting observations from the environment. This trajectory captures the agent’s decision-making process and provides insight into its behavior. Many agent failures are visible only in the trajectory, making the trajectory itself a primary resource for testing and debugging [33]. The reactive nature of agents makes their failure modes different from traditional software. A single flawed reasoning step early in an agent’s execution can cascade through subsequent steps, leading to compounded errors that manifest only at the trajectory’s end. For example, an agent might misunderstand a user’s instruction in its first step, adopting an incorrect implementation that goes undetected until it falls into an infinite loop. This implicit relationship between cause and effect makes agents harder to debug than classical systems, and it obscures which step to hold responsible for an observed failure. Furthermore, agents operate under continuous uncertainty about their environment. Tool outputs contain noise, incomplete information, and unexpected results, and agent actions may produce side effects on external services and the state. The sheer length of agentic trajectories (for example, often exceeding 40 steps in project issue resolution) introduces additional strain on the agent’s reasoning capability. Research shows that even state-of-the-art LLMs struggle to maintain reasoning quality when processing extended contexts, compounding the challenges of both testing and debugging. 3 Challenges in Testing Agents Testing LLM-based agent applications introduces challenges that go beyond those encountered in conventional software testing. These challenges stem from the various properties of agentic systems: non-deterministic behavior, multi-step reasoning, tool use, interaction with external environments, etc. Together, these properties complicate how developers define correctness, observe failures, measure coverage, and automate regression testing. In this section, we summarize the key challenges that arise when testing agentic systems. 3.1 Test Oracle Problem The test oracle problem—determining whether an observed behavior is correct—is especially acute in LLM-based agent applications. In conventional software, testers can often specify an expected output for a fixed input and compare the observed result against it. In agentic systems, however, correctness is often distributional rather than point-valued. The same query may produce different, yet equally acceptable, responses across executions. Acceptability may also depend on context, phrasing, intermediate reasoning paths, and the persona or role of the end user, such as a product manager versus a technical lead. As a result, assertions must be flexible enough to admit legitimate variation while still being precise enough to detect real failures. 3.2 Testing Non-deterministic Behavior Non-determinism is one of the most fundamental obstacles to reliable agent testing. Because LLM inference is stochastic, repeated executions of the same test may produce different final outputs, intermediate tool calls, or reasoning paths. This undermines traditional testing assumptions, where a fixed input is expected to produce a fixed and reproducible output. A test may pass in one run and fail in another without any change to the system under test, producing behavior analogous to flaky tests in conventional software [10]. Non-determinism also complicates debugging and test isolation. When a failure occurs, devel- opers may be unable to determine whether it reflects a genuine defect, a sampling artifact, an J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:4Trovato et al. environmental condition, or an interaction among these factors. Reproducing failures requires controlling, replaying, or at least logging the stochastic elements of execution, but current testing frameworks provide limited support for doing so. Several approaches from software testing offer useful foundations. Property-based testing [16] and metamorphic testing [8] shift the focus from exact output equivalence to relational properties that should hold across executions. Techniques from probabilistic program testing [4,21] and model checking [23] provide formal foundations for reasoning about stochastic systems. Recent work has begun adapting these ideas to agents: ToolFuzz [24] fuzzes tool runtimes, ChainFuzzer [35] targets multi-tool workflows, and agentic property-based testing [22] explores bug finding across LLM-powered components. However, a comprehensive framework for managing non-determinism across prompts, models, tools, memory, and external environments remains absent. 3.3 Testing Trajectories Agent behavior is not fully captured by final outputs. Many failures arise from the trajectory of execution: the sequence of reasoning steps, tool selections, delegation decisions, intermediate states, and environment transitions that lead to a final result. Testing these trajectories is therefore one of the most distinctive challenges in agent testing. Observability platforms such as LangSmith, LangFuse, Arize AI, and Maxim AI expose execution traces, intermediate decisions, tool calls, and environment transitions. However, these artifacts are typically external to the main test suite and are not well integrated with standard testing frameworks such as JUnit, PyTest, Unittest, etc. Consequently, even when developers inspect trajectories during debugging, those checks are rarely encoded as automated, reusable assertions. This creates a gap between what can be observed and what is systematically validated during routine development. Bridging this gap requires testing infrastructure that supports first-class assertions over trajec- tories, including tool-selection paths, delegation patterns, reasoning traces, and multi-step task execution [18]. Such assertions may need to operate at different levels of granularity: individual tool calls, sub-task plans, full execution traces, or resulting environment states. Formalizing what constitutes an acceptable trajectory, and defining adequacy criteria based on trajectory coverage rather than only code coverage, remain important open problems. 3.4 Testing Non-functional Requirements Non-functional requirements (NFRs)—including security, performance, resilience, privacy, and pol- icy compliance—are critical for agentic systems because agents increasingly interact with external tools, sensitive data, and real-world environments. Yet NFR testing remains underrepresented in practice. Empirical evidence from open-source agent projects [28] shows that only a small fraction of tests, roughly 7–8%, target NFRs at all. Among those tests, most focus on security scenarios, while performance, memory, and resilience receive substantially less attention. This scarcity reflects several compounding difficulties. Performance and latency are difficult to evaluate without realistic workloads and deployed environments, which are often unavailable during unit testing. Security properties, such as prompt-injection resistance, PII anonymization, and privilege-escalation prevention, require adversarial inputs that are difficult to construct system- atically. Resilience testing, which evaluates behavior under partial failures such as tool timeouts, schema drift, or unreachable services, requires fault-injection capabilities that current agent frame- works rarely expose. Established software engineering techniques provide a useful starting point. Fault injection [1], chaos engineering [3], and metamorphic testing [9] are well-established approaches for evaluating behavior under failures and adversarial conditions. Recent work has begun extending these ap- proaches to agents: MAST [6] catalogs behavioral failure modes in multi-agent execution traces, J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Towards Risk-free AI Agent Deployment111:5 ReliabilityBench [14] applies chaos engineering to single-agent systems using infrastructure-level faults, and MAS-FIRE [17] defines a fault taxonomy for multi-agent systems. However, prior work has not yet provided a unified fault taxonomy or injection mechanism that spans the full agentic stack, including tool integration, memory, inter-agent communication, security, and fault propagation across architectural layers. 3.5 End-to-end Testing End-to-end (E2E) testing exercises a complete flow from user input through all system layers to the final outcome. For agentic systems, E2E testing is especially important because correctness often depends on the composition of agents, tools, memory, external services, and environment side effects across a full task. At the same time, E2E testing is difficult to perform in practice. It typically requires a deployed or near-deployed environment, realistic user inputs, and mechanisms for observing and asserting on outcomes that may span multiple turns, tool invocations, and external state changes. A central difficulty is distinguishing integration testing from true E2E testing when the application is not fully deployed. E2E tests also face the full force of the oracle problem (Section 3.1) and non- determinism (Section 3.2), because both are amplified when the entire system is exercised together. Validating environment side effects, such as files written, database entries modified, or external services called, adds further complexity. Recent benchmarks such as휏-bench [38] and OSWorld [37] move toward E2E evaluation by assessing resulting system state rather than only final outputs. However, systematic support for E2E testing of agent workflows in development settings remains limited. Developers still lack practical tools for constructing realistic scenarios, controlling external dependencies, validating side effects, and replaying failures with sufficient determinism for regression testing. 3.6 Test Adequacy Metrics Measuring the adequacy of a test suite—how much of the system has been exercised and how well— is a long-standing concern in software testing. Conventional criteria such as statement coverage, branch coverage, and mutation score provide well-defined measures for traditional software. These criteria, however, do not capture the higher-level abstractions that characterize agentic systems, including tool-selection paths, reasoning traces, delegation patterns, multi-step task execution, memory interactions, and environment state transitions. In practice, no widely accepted adequacy standard exists for agent testing. Practitioners instead rely on ad hoc approaches such as edge-case coverage, domain-driven test design, trajectory sampling, query clustering, and acceptable failure thresholds—all of which are project-specific judgments rather than formal criteria [28]. This creates uncertainty about when a test suite is sufficiently comprehensive and makes it difficult to compare testing practices across projects. Developing adequacy metrics for agentic systems likely requires moving beyond code coverage toward behavioral and interaction coverage. Examples include coverage over tool-use patterns, prompt and input distributions, execution trajectories, memory states, delegation structures, and environment-side effects. These metrics must also account for stochastic behavior, since a single execution may not adequately characterize an agent’s behavior under a given scenario. 3.7 Agent-specific Test Frameworks Despite the rapid growth of LLM-based agent applications, the tooling and infrastructure for testing them remain immature. General-purpose frameworks such as PyTest and Unittest were designed for conventional software and do not natively support assertions over execution trajectories, dele- gation behavior, reasoning steps, non-deterministic outputs, or environment side effects. Although J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:6Trovato et al. observability platforms expose many of these artifacts, they are often separate from the test suite and are not designed for automated, regression-style validation. Agent-specific test infrastructure is therefore needed to treat trajectories, tool calls, intermediate reasoning states, memory interactions, and environment transitions as first-class test targets. Such infrastructure should provide: (1) reusable test doubles that simulate not only tool execution behav- ior but also the effects of tool descriptions on agent reasoning [15]; (2) distributional assertions for validating stochastic behavior across repeated executions; (3) mechanisms for replaying execution traces with sufficient determinism for regression testing; and (4) integration with observability data so that trajectory-level assertions can be embedded within standard testing workflows. Developing this infrastructure, grounded in a rigorous theory of agent testability that extends classical notions of controllability and observability [11,31] to the agentic setting, is a key prerequisite for systematic and rigorous testing of LLM-based agent applications. 4 Challenges in Debugging Agents Another central requirement for risk-free agent deployment is the ability to debug agents. In classical software engineering, debugging proceeds by inspecting the execution state of a program using execution traces. The analogous artifact for an LLM-based agent is its trajectory: the recorded sequence of observations, reasoning steps, and tool calls. Because trajectories can be produced by any agent, building the debugging layer on top of them yields methodologies that generalize across coding agents, enterprise workflow agents, and any other deployment class an organization might consider. We organize the body of work on trajectory-based agent debugging into three areas: (1) trajectory management, where trajectories must be captured, structured, and stored so that they can be inspected and reasoned over, (2) failure analysis that detects a failure and attributes it to a specific component or step, and (3) the diagnosis have to be turned into an actual fix, either by correcting the ongoing run or by ensuring the agent does not repeat the mistake in the future. We discuss each area in depth below. 4.1 Capturing Agent Trajectories Capturing agent trajectories is the foundational step for any post-hoc analysis, failure diagnosis, or debugging effort. However, trajectory capture is not simply logging; it requires a systematic approach to recording, standardizing, and managing execution data. The need for robust trajectory management becomes significant with the scale of modern agentic runs. A single SWE-bench coding task may involve dozens of file reads, shell commands, and LLM reasoning steps. An enterprise workflow agent processing a multi-document loan application might execute hundreds of API calls across several sub-agents. Storing, indexing, and retrieving this data with enough fidelity to support debugging is a research problem. Standardized trajectory format. Raw agent logs are unstructured flat text and lack standards, whereas failure analysis tools have to operate on a standard representation. On the infrastructure side, the practical implementation layer is served by open-source tools such as Langfuse, to capture LLM calls, tool invocations, and latency as structured, queryable traces. At the standardization level, OpenTelemetry defines cross-framework semantic conventions for LLM and agent tracing, with adoptions already appearing in the industry. Graphectory [19] is a directed graph constructed automatically from raw trajectories in which nodes represent agent actions and edges encode both temporal sequencing and structural navigation. Noise and context management. Raw trajectories are noisy: a long-horizon task can exceed one hundred thousand tokens, most irrelevant to any given failure. The community has converged on memory distillation as the primary solution. Agent Workflow Memory [34] induces compact, J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Towards Risk-free AI Agent Deployment111:7 reusable workflow subroutines from prior trajectories, improving task accuracy through smarter trace utilization. ReasoningBank [27] distills generalizable reasoning strategies via a self-judging mechanism, establishing experience scaling as a new performance dimension. 4.2 Failure diagnosis Failure analysis asks: given a trajectory, did the agent fail, why did it fail, and at what point did the failure become inevitable? This area encompasses failure characterization and attribution. The difficulty of this problem should not be underestimated. Unlike a traditional program that throws an exception at the point of failure, an LLM-based agent can fail silently and finally lead to wrong or unsafe output. The failure may be owed to an early bad decision whose consequences only become visible many steps later. Characterizing Agent Failures. Before failures can be detected or attributed, they must be understood empirically. Failure characterization studies help practitioners identify which instru- mentation to deploy, which guardrails to implement, and where human oversight is most needed. The most systematic single-agent effort is Bouzenia et al. [5], identifies recurring behavioral anti- patterns across 120 trajectories, including incoherent reasoning chains, failure to integrate tool feedback, and action-repetition loops. Liu et al. [20] isolate plan deviation as a distinct failure class across 10K trajectories, finding that a poorly specified plan inflicts more damage than no plan at all. For multi-agent systems, MAST [7] summarizes 14 failure modes across 1,600+ traces, highlighting that coordination breakdowns and system design deficiencies are as harmful as the individual model error. Automated Failure Attribution. Even when a failure is detected, attributing it to the right step in a long trajectory is a challenging research problem because of its long and noisy causal chains. A wrong decision at step 7 may not surface until later steps fail with test suites. Naive approaches that inspect only the final few steps, or treat all steps as equally likely candidates, fall short in uncovering the failure root causes. The field has developed with early benchmarks and several strategies. Who&When [39] benchmarks automated attribution directly, introducing three baseline strategies—all-at-once, step-by-step, and binary-search attribution—and finding that even frontier models achieve only a miserable step-level localization accuracy. AgentRx [2] advances step localization through constraint synthesis: it prompts an LLM to generate behavioral constraints a correct execution should satisfy, evaluates each constraint against each trajectory step, and passes the resulting step-indexed validation log to a judge—substantially reducing the reasoning burden over raw traces. RootSE [32] is the first failure diagnosis benchmark on agentic coding trajectories, characterized by noise from redundant program structure and verbose code. It follows up a diagnostic solution where the failure-irrelevant content is pre-folded via pattern matching, and an investigator iteratively retrieves necessary execution information on demand. 4.3 Debugging: End-to-end validation The focus closes the loop from diagnosis to improvement. Existing methods can be categorized into two directions: Intra-task repair uses the failure diagnosis to correct the agent by re-executing. Inter-task self-evolution persists the lessons learned across tasks, building up a growing base of experience to improve itself over time. For organizational deployment, end-to-end debugging is where the practical value of the entire pipeline materializes. An agent that can detect its own mistakes and recover from them is safer to deploy in a consequential workflow. An agent that learns from its failures over time progressively reduces its error rate, which is the necessary condition for sustainable automation. Intra-task debugging. The immediate challenge for intra-task debugging is translating a failure diagnosis into an effective correction. If the diagnosis is too vague, the agent has insufficient J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:8Trovato et al. information to do better on a retry; if the feedback is too specific, it may not generalize to related failures. The quality of the corrective signal is the primary determinant of whether repair succeeds. Reflexion [30] established the foundational mechanism - an agent writes a verbal self-reflection after each failed attempt and stores it in a memory buffer - reaching an advanced performance without any model weight updates. AgentDebug [40] integrates the step-level attribution with targeted feedback injection, re-executing the agent from the identified fault point with error-grounded context. Its recovery gains demonstrate that attribution precision is a direct predictor of repair quality. Wink [25] classifies live trajectories’ misbehaviors across pre-defined failure categories, and intervenes in agent runtime behavior with course-correction guidance. Self-evolving through trajectory learning. Self-evolving agents go beyond one-shot repair; they extract reusable knowledge from past trajectories and accumulate it in persistent memory or skill libraries that shape future behavior. The key insight is that trajectories are not merely execution logs but a form of implicit knowledge: by extracting and reusing what has worked and what has not, an agent can improve across tasks over time without retraining. An early survey of this line of work argues that self-evolution through trajectory learning is the defining long-term challenge for autonomous agent systems [12]. Specifically, SE-Agent [13] realizes this by revising, recombining, and refining prior reasoning traces, yielding consistent improvements on multi-step coding tasks. SkillRL [36] formalizes the distillation step: a teacher model converts raw trajectories into reusable skills, enabling the agent to generalize learned behaviors to new tasks rather than re-solving from scratch. Trace2Skill [26] provides direct trajectory-to-skill distillation: a set of sub-agents extracts trajectory-level lessons through inductive reasoning and consolidates them into a unified, reusable skill directory for future use. Ensuring the quality of an evolving skill library remains the central open challenge: incompetent skills introduce noise and contradiction into agent execution, ultimately risking performance degradation. 5 Deployment Readiness: A Checklist for Organizations The testing and debugging methodologies discussed above establish the research foundation for safer agent deployment. This section provides a checklist to help organizations assess their readiness to deploy an agent into core business processes. The checklist is organized around three deployment phases: pre-deployment, deployment, and post-deployment. 5.1 Pre-Deployment Readiness • Have you established a trajectory capture system? Without systematic trajectory logging, it is impossible to debug failures after deployment and to learn from agent behavior at scale. How: Use infrastructure like LangFuse, Langsmith, or OpenTelemetry to record all agent reasoning steps, tool calls, and environment observations in a queryable format. Trajectory data should be retained for at least the duration of your rollout plan, and indexed by task, date, and outcome. • Have you characterized failure modes specific to your domain? The failure patterns discovered in one domain (e.g., improper reasoning chains in coding) may not be the primary failure modes in yours (e.g., policy compliance checking). Your testing strategy should target your most likely failure modes first. How: Conduct a sampled trajectory review with domain experts and the agent team to identify recurring behavioral anti-patterns. Document at least critical failure scenarios as in [5] and [20]. •Have you defined acceptance criteria that your agent must meet? Acceptance criteria translate your business requirements into measurable assertions. This provides an objective basis for determining whether the agent is ready for deployment. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Towards Risk-free AI Agent Deployment111:9 How: Define criteria across multiple dimensions: functional correctness, safety, and compliance rate. Frame criteria in terms of trajectory properties or distributional outcomes rather than single-run outputs. •Have you executed an end-to-end test in a staging environment that mirrors production? Unit and integration tests exercise components in isolation. End-to-end tests run the full workflow, including all tools, external services, and data flows. Failures discovered in staging are orders of magnitude cheaper to fix than failures discovered in production. How: Run your agent through realistic, complete workflows in an environment as close to production as possible. Validate both the final output and procedural actions (files written, database records modified, external services called). Document latency, error rates, and recovery behavior. •Have you established human oversight and rollback procedures? Even well-tested agents fail in production under conditions not anticipated during development. Human oversight ensures that high-stakes decisions are reviewed, and rollback procedures ensure that failures do not cascade. How: Define which agent decisions require human review before being executed, which require review after the fact, and which are fully autonomous. 5.2 Early Deployment and Monitoring •Are you monitoring the execution process in trajectory? Many agent failures are implicit in the final output but are visible in the trajectory (incorrect reasoning chains, failure to use available tools, or infinite loops). Trajectory monitoring catches failures before they cause business damage. How: Set up automated alerts on trajectory properties: unusual tool-call sequences, repeated failed steps, or reasoning traces that deviate from expected patterns. Sample trajectories regularly for manual inspection by domain experts. •Have you established a feedback loop for capturing user-reported failures? Users will encounter edge cases and contexts that your test suite did not anticipate. A structured process for capturing and diagnosing these failures ensures that real-world experience feeds back into improvements. How: Create a reporting mechanism for users to flag unexpected behavior. Correlate each report with the corresponding trajectories and task environments. Prioritize reports by frequency and business impact, and route high-priority failures to your debugging team with full trajectory context attached. •Are you executing intervention on failed trajectories? Many can be fixed by correcting the agent mid-execution or by providing targeted feedback that helps the agent recover. Intra-task debugging is faster and less risky than redeployment. How: When a task failure is detected, before rolling back or escalating, attempt one or more recovery strategies: ask the agent to reflect on its error and retry [30], re-execute from the fault point with corrected context [32], or inject course-correction guidance [25]. Log the outcome and the triggering error for later analysis. 5.3 Sustained Deployment and Improvement • Are you storing lessons learned from trajectories into reusable knowledge? Over time, agents should become more reliable through learned experience. Self-evolution through trajectory learning allows agents to internalize patterns and avoid repeating mistakes. How: Periodically review successful and failed trajectories to extract patterns. Consolidate these into reusable skills or decision heuristics that the agent can reference in future runs. Update the J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:10Trovato et al. agent’s system prompt, memory, or tool set to incorporate these lessons. Validate that the agent improves on similar tasks in subsequent runs. • Have you measured and reported your adequacy of testing? Stakeholders need to under- stand beyond test acceptance metrics, for example, how comprehensively we have exercised the agent and its failure modes? How: Report metrics beyond pass/fail rate, such as trajectory diversity, failure-mode coverage, consistencies, and distributional coverage across different scenarios or prompts. 6 Open Research Questions for the Community While trajectory-based testing and debugging provide a systematic foundation for agent deployment, three fundamental research challenges remain open: Formal Test Adequacy Metrics for Agentic Systems. Software testers have long relied on metrics like statement coverage and branch coverage. But these do not work for agents. What matters for agents is different: tool selection diversity, reasoning quality, trajectory variation, and memory interactions. Today, practitioners rely on intuition to choose edge cases based on domain knowledge. Therefore, there exists a gap in understanding when testing is enough. A rigorous, formal adequacy metric for agents is needed. Such a metric must account for non-determinism and multi-step behavior. It would let practitioners answer the fundamental question: is my test suite comprehensive enough? It would also let researchers compare testing methodologies fairly. Root-cause attribution in Long-horizon Trajectories. When a traditional program fails, we see an exception and investigate it. When an agent fails, the diagnosis is much harder. In a trajectory spanning 50 to 200 steps, a wrong decision at step 7 may only become visible at step 50. By then, many other decisions have cascaded onto it. Current approaches treat attribution as a search problem. Systems like AgentRx [2] and RootSE [32] show promise. Yet even the best models achieve only modest accuracy in pinpointing the actual failure point. The field needs a capable framework for causal reasoning over noisy, stochastic trajectories. With better attribution, we can build debugging tools that practitioners can trust. Safety and Trustworthiness of Agents. Agents that learn from their own trajectories show real promise. They accumulate skills over time and improve themselves without retraining. But it also introduces new risks. Learned skills can be wrong. They can contradict each other. Agent behavior can drift away from what the organization intended. When an agent changes its own behavior, who audits that change? In this direction, the fundamental questions are still open. How do we verify that a skill library maintains safety invariants? How do we detect when learned skills degrade before they break production? When should humans intervene in an autonomous learning loop? 7 Conclusion The transition from human-guided workflows to agentic automation is underway across industries, but deploying an agent into a consequential business process without the means to audit it is a critical risk. This article has argued that the path to risk-free deployment runs through the observable agent trajectory: a model-agnostic artifact for building systematic testing methodologies and a debugging pipeline for fixing issues. The deployment-readiness checklist we presented translates these research advances into concrete organizational practice across the deployment lifecycle. We also identify several open research questions for the community, including formal adequacy criteria for stochastic multi-step systems, root-cause attribution in long-horizon trajectories, and mechanisms to keep self-evolving agents aligned with organizational intent. Addressing these challenges will require sustained collaboration among the software engineering and AI communities. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Towards Risk-free AI Agent Deployment111:11 References [1]Jean Arlat, Martine Aguera, Louis Amat, Yves Crouzet, J-C Fabre, J-C Laprie, Eliane Martins, and David Powell. 1990. Fault injection for dependability validation: A methodology and some applications. IEEE Transactions on software engineering 16, 2 (1990), 166–182. [2]Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal. 2026. AgentRx: Diagnosing AI Agent Failures from Execution Trajectories. arXiv preprint arXiv:2602.02475 (2026). [3]Ali Basiri, Niosha Behnam, Ruud De Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos engineering. IEEE software 33, 3 (2016), 35–41. [4] Mateus Borges, Antonio Filieri, Marcelo d’Amorim, and Corina S. Păsăreanu. 2015. Iterative distribution-aware sampling for probabilistic symbolic execution. In FSE. [5]Islem Bouzenia and Michael Pradel. 2025. Understanding software engineering agents: A study of thought-action-result trajectories. arXiv preprint arXiv:2506.18824 (2025). [6]Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al.2025. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657 (2025). [7]Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al.2026. Why do multi-agent llm systems fail? Advances in Neural Information Processing Systems 38 (2026). [8]Tsong Y. Chen, Shing C. Cheung, and Siu Ming Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. Technical Report. CS Department, Hong Kong University of Science and Technology. [9]Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR) 51, 1 (2018), 1–27. [10]Saikat Dutta, August Shi, Rutvik Choudhary, Zhekun Zhang, Aryaman Jain, and Sasa Misailovic. 2020. Detecting flaky tests in probabilistic and machine learning applications. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. 211–224. [11] Roy S Freedman. 1991. Testability of software components. IEEE transactions on Software Engineering 17, 6 (1991), 553–564. [12]Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al.2025. A survey of self-evolving agents: On path to artificial super intelligence. arXiv e-prints (2025), arXiv–2507. [13] Yifu Guo, Jiaye Lin, Huacan Wang, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, and Mingguang Chen. 2026. SE-agent: Self-evolution trajectory optimization in multi-step reasoning with LLM-based agents. Advances in Neural Information Processing Systems 38 (2026), 116314–116341. [14]Aayush Gupta. 2026. ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions. arXiv preprint arXiv:2601.06112 (2026). [15] Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. 2026. Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions. arXiv preprint arXiv:2602.14878 (2026). [16]John Hughes. 2007. QuickCheck testing for fun and profit. In International Symposium on Practical Aspects of Declarative Languages. Springer, 1–32. [17] Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng. 2026. MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems. arXiv preprint arXiv:2602.19843 (2026). [18]Wonjoong Kim, Sangwu Park, Yeonjun In, Sein Kim, Dongha Lee, and Chanyoung Park. 2025. Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents. arXiv preprint arXiv:2510.02837 (2025). [19] Shuyang Liu, Yang Chen, Rahul Krishna, Saurabh Sinha, Jatin Ganhotra, and Reyhaneh Jabbarvand. 2026. Process- centric analysis of agentic software systems. Proceedings of the ACM on Programming Languages 10, OOPSLA1 (2026), 1961–1988. [20]Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand. 2026. Evaluating Plan Compliance in Autonomous Programming Agents. arXiv preprint arXiv:2604.12147 (2026). [21]Kasper Luckow, Corina S. Păsăreanu, Matthew B. Dwyer, Antonio Filieri, and Willem Visser. 2014. Exact and approximate probabilistic symbolic execution for nondeterministic programs. In ASE. [22]Muhammad Maaz, Liam DeVoe, Zac Hatfield-Dodds, and Nicholas Carlini. 2025. Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem. arXiv preprint arXiv:2510.09907 (2025). [23]Piotr Mardziel, Stephen Magill, Michael Hicks, and Mudhakar Srivatsa. 2013. Dynamic enforcement of knowledge-based security policies using probabilistic abstract interpretation. Journal of Computer Security 21, 4 (2013). [24]Ivan Milev, Mislav Balunović, Maximilian Baader, and Martin Vechev. 2025. ToolFuzz–Automated Agent Tool Testing. arXiv preprint arXiv:2503.04479 (2025). J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:12Trovato et al. [25]Rahul Nanda, Chandra Maddila, Smriti Jha, Euna Mehnaz Khan, Matteo Paltenghi, and Satish Chandra. 2026. Wink: Recovering from misbehaviors in coding agents. arXiv preprint arXiv:2602.17037 (2026). [26] Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. 2026. Trace2skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158 (2026). [27]Siru Ouyang, Jun Yan, I Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T Le, Samira Daruki, Xiangru Tang, et al.2025. Reasoningbank: Scaling agent self-evolving with reasoning memory. arXiv preprint arXiv:2509.25140 (2025). [28]Rangeet Pan, Tyler Stennett, Divya Sankar, Bridget McGinn, Alessandro Orso, Raju Pavuluri, Saurabh Sinha, and Maja Vukovic. 2026. Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications. In 2026 41th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE. [29] Abhik Roychoudhury, Corina Păsăreanu, Michael Pradel, and Baishakhi Ray. 2026. Agentic AI software engineers: Programming with trust. Commun. ACM 69, 5 (2026), 56–58. [30]Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv. org/abs/2303.11366 8 (2024). [31] Jeffrey M. Voas and Keith W Miller. 2002. Software testability: The new verification. IEEE software 12, 3 (2002), 17–28. [32]Minxing Wang, Xiaofei Xie, and Yintong Huo. 2026. TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems. arXiv preprint arXiv:2605.26563 (2026). [33]Zexin Wang, Changhua Pei, Yuanhao Liu, Jingjing Li, Yintong Huo, Quan Zhou, Haotian Si, Hang Cui, Zihan Liu, Gaogang Xie, et al.2026. Agent System Operations: Categorization, Challenges, and Future Directions. arXiv preprint arXiv:2606.01581 (2026). [34] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. 2025. Agent Workflow Memory. In International Conference on Machine Learning. PMLR, 63897–63911. [35]Jiangrong Wu, Zitong Yao, Yuhong Nan, and Zibin Zheng. 2026. ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents. arXiv preprint arXiv:2603.12614 (2026). [36]Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, et al.2026. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234 (2026). [37] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al.2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37 (2024), 52040–52094. [38]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024.휏-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045 (2024). [39]Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al.2025. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. Proceedings of Machine Learning Research 267 (2025), 76583–76599. [40]Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, et al.2025. Where llm agents fail and how they can learn from failures. arXiv preprint arXiv:2509.25370 (2025). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.