Paper deep dive

Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis

Penny Chong, Harshavardhan Abichandani, Jiyuan Shen, Atin Ghosh, Min Pyae Moe, Yifan Mai, Daniel Dahlmeier

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 106

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/22/2026, 5:23:06 AM

Summary

The TED (Talk, Evaluate, Diagnose) framework is a novel, scalable evaluation approach for LLM agents that addresses the challenges of heterogeneous domains and user-aware performance assessment. It utilizes reusable persona templates for interaction, natural language grading notes for automated evaluation via LLM-as-a-judge, and an automated error analysis tool to identify inconsistencies and provide actionable feedback.

Entities (5)

TED framework · evaluation-framework · 100%LLM-as-a-judge · evaluation-method · 98%Grading Notes · evaluation-artifact · 95%MaxProgressRate@k · metric · 95%User Persona Templates · tool · 92%

Relation Signals (4)

TED framework → includesstage → Talk

confidence 100% · We introduce the TED framework (Talk, Evaluate, Diagnose).

TED framework → includesstage → Evaluate

confidence 100% · We introduce the TED framework (Talk, Evaluate, Diagnose).

TED framework → includesstage → Diagnose

confidence 100% · We introduce the TED framework (Talk, Evaluate, Diagnose).

LLM-as-a-judge → uses → Grading Notes

confidence 95% · grading notes serve as assertion criteria for LLM-as-a-judge

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Agent applications are increasingly adopted to automate workflows across diverse tasks. However, due to the heterogeneous domains they operate in, it is challenging to create a scalable evaluation framework. Prior works each employ their own methods to determine task success, such as database lookups, regex match, etc., adding complexity to the development of a unified agent evaluation approach. Moreover, they do not systematically account for the user's role nor expertise in the interaction, providing incomplete insights into the agent's performance. We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors. To address this, we introduce the TED framework (Talk, Evaluate, Diagnose). (1) Talk: We leverage reusable, generic expert and non-expert user persona templates for user-agent interaction. (2) Evaluate: We adapt existing datasets by representing subgoals-such as tool signatures, and responses-as natural language grading notes, evaluated automatically with LLM-as-a-judge. We propose new metrics that capture both turn efficiency and intermediate progress of the agent complementing the user-aware setup. (3) Diagnose: We introduce an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement. We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels. We also demonstrate potential gains in agent performance with peaks of 8-10% on our proposed metrics after incorporating the identified error remedies into the agent's design.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

105,933 characters extracted from source content.

Expand or collapse full text

Accepted as a conference paper at ICLR 2026 TALK, EVALUATE, DIAGNOSE: USER-AWARE AGENT EVALUATION WITH AUTOMATED ERROR ANALYSIS Penny Chong 1∗ , Harshavardhan Abichandani 1 , Jiyuan Shen 1 , Atin Ghosh 1 , Min Pyae Moe 1 , Yifan Mai 2 , Daniel Dahlmeier 1 1 SAP, 2 Stanford University penny.chong, harshavardhan.abichandani, jiyuan.shen@sap.com atin.ghosh, min.pyae.moe, d.dahlmeier@sap.com yifan@cs.stanford.edu ABSTRACT Agent applications are increasingly adopted to automate workflows across diverse tasks. However, due to the heterogeneous domains they operate in, it is chal- lenging to create a scalable evaluation framework. Prior works each employ their own methods to determine task success, such as database lookups, regex match, etc., adding complexity to the development of a unified agent evaluation approach. Moreover, they do not systematically account for the user’s role nor expertise in the interaction, providing incomplete insights into the agent’s performance. We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors. To ad- dress this, we introduce the TED framework (Talk, Evaluate, Diagnose) 1 . (1) Talk: We leverage reusable, generic expert and non-expert user persona templates for user-agent interaction. (2) Evaluate: We adapt existing datasets by representing subgoals—such as tool signatures, and responses—as natural language grading notes, evaluated automatically with LLM-as-a-judge. We propose new metrics that capture both turn efficiency and intermediate progress of the agent comple- menting the user-aware setup. (3) Diagnose: We introduce an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement. We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels. We also demonstrate potential gains in agent performance with peaks of 8-10% on our proposed metrics after incorporat- ing the identified error remedies into the agent’s design. 1INTRODUCTION Large Language Models (LLMs) agents (Liu et al., 2023; Jang et al., 2025; Koh et al., 2024) are in- creasingly being adopted for many real-world tasks in various domains due to their potential of fully automating mundane workflows and enhancing productivity. However, evaluation of agents remains a challenge today due to the heterogeneous domains the agents operate in. As every domain comes with its own goals, creating a scalable unified evaluation framework which reliably assesses agent performance across diverse tasks is non-trivial. Existing works (Qian et al., 2024; Lu et al., 2024; Barres et al., 2025; Chang et al., 2024) each propose their own evaluation methods, e.g., checking database states, tool signatures, or exact matches which differ in scope and assumptions, making unification challenging. Moreover, since agent behavior is heavily influenced by the conversation trajectory with the user, current assessment methods that overlook the user’s role in the interaction may fail to comprehensively capture agent’s performance. Given that agents are non-deterministic and it is difficult to craft reference conversations, a common practice to interact with the agent is to dynamically simulate the user responses in the conversation loop with the agent (Yao et al., 2024). This has been adopted as a common practice for agent ∗ Corresponding author. 1 Code and dataset are available in the repository https://github.com/SAP-samples/agent-quality-inspect. 1 arXiv:2603.15483v1 [cs.AI] 16 Mar 2026 Accepted as a conference paper at ICLR 2026 evaluation because static user setups, where user messages are predetermined, do not work. This is because the agent’s responses to earlier predetermined user inputs may diverge from the reference conversation for which the static messages were curated. However, most works employing dynamic conversation have limitations because they do not systematically separate user persona from task instructions, thus failing to account for the impact of user behavior (independent of the task) on agent performance, providing incomplete insights. This is important as good agents ask clarifying questions when given incomplete input, while poor agents do not; thus, systematic testing is essential for fair comparison across agents and tasks. Despite the complexity of agent trajectories, existing works (Qiao et al., 2024; Xiao et al., 2024; Qian et al., 2025) often stop at metric reporting. To address these shortcomings, we propose the TED framework (Talk, Evaluate, Diagnose). (1) In the Talking stage, we decouple user personas from task instructions and introduce a user-aware agent evaluation framework based on reusable, generic persona templates enabling diverse and systematic creation of test scenarios. (2) In the Evaluation stage, we adapt existing datasets by representing subgoals—such as tool signatures, and responses—as natural language grading notes, and evaluate them with LLM-as-a-judge. We propose new metrics that capture not only partial progress and task success, but also the efficiency of task progression—measured in conversational turns. (3) In the Diagnosis stage, we introduce an automated error analysis tool that examines inconsistencies of both agents and LLM-as-a-judge, automatically identifies errors, and offers actionable feedback. We summarize our contributions as follows: i) Propose an agent evaluation framework applicable across heterogeneous agent domains that is built on reusable, generic expert and non-expert persona templates that systematically assess the impact of users’ role on agent performance. i) Introduce a benchmark by adapting existing datasets to grading notes—natural language check- lists of subgoals. Grading notes serve as assertion criteria for LLM-as-a-judge, which scores the agent performance based on its trajectory log without requiring access to the environment. i) Introduce new metrics to accompany the user-aware evaluation setup, which are essential for capturing an agent’s progress with respect to the number of conversational turns. iv) Propose an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement. 2RELATED WORKS Conversation simulation. A majority of the agents today are conversational and involve invok- ing multiple tools to solve a task. With complex tasks requiring human interaction, the literature (Yao et al., 2024; Wang et al., 2023; Xiao et al., 2024) has adopted a dynamic setup using a LLM- simulated user (user proxy), for automated testing of agents. However, existing dynamic evaluation methods face several limitations: some rely on user instruction prompts that are tightly coupled with specific agents, scenarios, and personas (Yao et al., 2024), while others omit user personas alto- gether (Lu et al., 2024)—both of which limit the reusability of evaluation methods across different domains. Although agent performance is influenced by the behavior of user proxy, this dependency is rarely analyzed systematically due to user personas being inconsistently defined across samples (Huang et al., 2025). While prior work (Barres et al., 2025) introduced a systematic evaluation of “easy” and “hard” personas for one of the domains, their telco-specific user prompt templates are not generic and limits reusability across domains. Our TED framework differs from prior work by allowing end-user to systematically test the agent with reusable, generic expert and non-expert personas that are agent- or task-agnostic. We demonstrate this in our experiments on the τ 2 -bench (Barres et al., 2025) and ToolSandbox (Lu et al., 2024) datasets, which span various domains such as airline booking, messaging, setting reminders, etc., all evaluated using the same user persona templates without any tuning. Metrics and error analysis. To evaluate agent performance, most prior work (Wang et al., 2023; Xie et al., 2024) relies on success rate. However, the metric focuses solely on the final outcome and provides only a coarse-grained assessment of agent behavior. This is first addressed by AgentBoard (Chang et al., 2024) that introduced progress rate as a fine-grained metric but in a multi-step agent- environment setting without conversation simulation. We extend this to multi-turn settings and propose metrics that combine turn-level efficiency and progress rate. Unlike MINT (Wang et al., 2 Accepted as a conference paper at ICLR 2026 2023), which measures only final success after t interactions, our turn-aware evaluation captures per-turn progress and efficiency, offering a richer measure of agent performance in complex tasks. Given the non-deterministic behavior of agents, Yao et al. (2024) reports the pass@k and passˆk metrics. In line with the pass@k metric used to assess the chance of whether at least one out of k trials is successful, we also report new metrics that capture the best-case performance under the stochastic runs. Instead of checking goal attainment via direct database lookups, tool signatures, etc., we represent all subgoals as grading notes. This approach abstracts complex goals, is user- friendly, and does not require system-state access, making our evaluation applicable to both agents that modify the system state and those that do not. While prior work uses natural language for only some assertions (Barres et al., 2025), we extend this to cover tool calls and end responses. Similar to Cui et al. (2025), we identify common errors made by LLM agents; however, our approach discovers these errors in an unsupervised manner via automatic analysis of real-time logs rather than relying on predefined categories. 3TALK, EVALUATE, DIAGNOSE: TED FRAMEWORK We define a LLM agent as an automated system that performs tasks via interactions with users, tools, and the environment. Its action space includes tool use, responses to users, and internal reasoning. After each action, the agent receives partial state information, such as API responses, or a subse- quent user utterance. To systematically evaluate agents, we introduce the TED framework—Talk, Evaluate, and Diagnose—as complementary and interdependent stages. In the Talking stage, diverse user-agent interactions are simulated, to study how robust agents complete tasks, for the different type of users, such as non-expert users who require more conversational turns. Traditional metrics like success or progress rates often fail to capture subtleties of turn efficiency, motivating metrics that consider both task progress and turn-efficiency during the Evaluation stage. Moreover, evalu- ation using LLM-as-a-judge are subject to stochasticity and potential errors. The Diagnosis stage helps extract meaningful insights from inconsistencies and errors made by both the agent and LLM- as-a-judge. Together, these stages form a unified framework as detailed in the following subsections. 3.1THE TALKING STAGE Dynamic evaluation with expert and non-expert user personas. Existing methods that use LLM- simulated user also known as user proxy (Yao et al., 2024; Lu et al., 2024) are constrained by either tightly coupled or missing user personas, hindering systematic analysis of the effect of user behavior on agent performance. A tightly coupled task complexity and user persona, makes it challenging to isolate their individual impacts on agent performance. For instance, when an agent answers technical legal questions, the outcome may differ depending on whether the user is an expert or a layperson, even if the task complexity remains constant. However, if both the task and user expertise as determined by the user persona vary simultaneously, it becomes difficult to determine which factor is driving performance differences. In this work, we propose a scalable, dynamic agent evaluation framework that leverages reusable, generic expert and non-expert user personas to simulate realistic user interactions across a wide range of scenarios. Let P = p expert ,p non−expert denote the set of persona prompts with different user expertise level, I be the set of task instructions, and U be the set of full user prompt consumed by the LLM-simulated user. We abstract the full user prompt templating process as a function f , combining user persona prompt p, with a task instruction i : u = f(p,i),(1) where p∈ P , i∈ I , and u∈ U . The function f includes general rules for the user proxy, along with a two-step process—reflection followed by response. For each agent and task instruction sample i, we vary only the persona prompt p to generate u expert and u non−expert . Refer to Appendix A.3 for the prompt f and user persona template p. An example of task instruction i is shown in Fig. 5. 3.2THE EVALUATION STAGE We define the set of grading notes G as natural language text used as assertion-based ground truths by LLM-as-a-judge. Each subgoal is represented by one such grading note 2 . Unlike prior work 2 Subgoal is represented by grading note which is a natural language text. 3 Accepted as a conference paper at ICLR 2026 that uses keypoints (Hao et al., 2025) or limited natural language assertions (Barres et al., 2025), we expand coverage to include tool calls, their order, and key agent responses in G. While we adopt the notion of milestones (key events that must happen) (Lu et al., 2024) for the set G, we do not follow their DAG-based construction method. An example of grading note is: Agent should enable Wifi. More examples are in Appendix A.12. 3.2.1LLM-AS-A-JUDGE AND MAXPROGRESSRATE@k LLM-as-a-judge. We extend beyond the multi-step agent-environment setting and exact match metric (Chang et al., 2024) by evaluating agents in a multi-turn user-agent setup, where grading notes serve as subgoals to assess both intermediate and final states, tool calls, as well as the agent’s output responses. Let D = (i,G i )| i ∈ I be the test dataset, where i ∈ I is a task instruction, G i =g i,1 ,g i,2 ,...,g i,n i be the set of grading notes associated with the task instruction i, and|G i | be the number of subgoals, i.e., grading notes. We denote the corresponding agent trajectory, which includes information on tool calls, agent responses and user utterances for the entire conversation up to the final conversational turn, as τ i . For a task sample (i,G i ), the progress of the agent given its trajectory τ i , is defined as the proportion of subgoals achieved: progress(i,G i ,τ i ) = 1 |G i | |G i | X j=1 LLM judge (i,g i,j ,τ i ),(2) where LLM judge (·) returns 1 if the subgoal g i,j is achieved, and 0 otherwise. We define the progress rate as the average progress across all samples in the dataset D, i.e., progressrate = E (i,G i )∼P D [ progress(i,G i ,τ i ) ]. Using LLM-as-a-judge with grading notes reduces the need for custom dataset-specialized evaluation harnesses and infrastructure. In this formulation, the judge is queried once for every subgoal. However, to ensure reliability, we run the judge multiple times and take a majority vote as the final score. We discuss the stability of the judge further in Section 3.3. The LLM judge (·) prompt is provided in Appendix A.4. From pass@k to MaxP rogressRate@k.Given the non-deterministic nature of agent behavior, a commonly used evaluation metric is pass@k = E P task 1− n−c k / n k (Yao et al., 2024), which measures the probability that at least one trial succeeds when sampling k out of n total trials. The notation c denotes the number of trials that are successful. Each trial represents a complete multi-turn conversation, consisting of multiple back-and-forth user-agent exchanges. By this definition, when n = k, the pass@k metric evaluates to 1 if at least one of the k trials for a given task is successful, and 0 otherwise. The metric then corresponds to the expected maximum success per task, averaged over all tasks, measuring the agent’s best performance across the trials: pass@k = E (i,G i )∼P D maxsuccess(i,G i ,τ l i )| l = 1,...,k , where success(·)∈0, 1. (3) The notation success(i,G i ,τ l i ) for a given sample (i,G i ) represents whether the agent with trajec- tory τ l i successfully completes the task on the l-th trial, with a value of 1 for success and 0 for failure. By taking the maximum success over k trials via the max· operator, we capture the agent’s best performance across these trials. We then relax the strict success condition in equation 3 by defining a thresholded progress-based success criterion: pass@k = E (i,G i )∼P D max 1 progress(i,G i ,τ l i )≥ threshold| l = 1,...,k ,(4) where 1· is the indicator function and the threshold ∈ [0, 1] defines the minimum progress for a trial to be considered successful. Setting threshold = 1 counts only trials with full sub- goals completion (i.e., progress(i,G i ,τ l i ) = 1) as successful, and treats any partial progress (i.e., progress(i,G i ,τ l i ) < 1) as failure. Nonetheless, equation 4 applies a hard threshold—treating all progress below the threshold as fail- ure—and discards agent’s fine-grained progress. To retain this information, we define a soft ver- sion, MaxProgressRate@k to evaluate agent’s best performance based on the maximum progress achieved at the final conversational turn, across k trials, averaged over all samples: MaxProgressRate@k = E (i,G i )∼P D max progress(i,G i ,τ l i )| l = 1,...,k .(5) 4 Accepted as a conference paper at ICLR 2026 3.2.2PROGRESS AND TURN-LEVEL EFFICIENCY The turns within each conversational trial are interdependent where errors in the earlier turns can propagate and impact task success. While the MaxProgressRate@k metric in equation 5 cap- tures non-determinism by measuring agent’s best performance across the k trials and evaluates fine- grained progress only at the final conversational turn, it does not assess how quickly progress is made throughout the conversation. This gap in evaluation leads us to consider two distinct scenarios: i) where making early progress matters, and i) where it does not. i) Early progress matters. In this subsection, we view progress as a function of conversational turns and for notational simplicity, we denote the progress at turn t by p(t) := progress(i,G i ,τ l i [1 : t]), where τ l i [1 : t] denotes the segment of the agent trajectory τ l i from the first turn up to turn t. Let p(t) : [0,T] → R represents the discrete progress values at each turn. For computing AUC, we treat the discrete values p(t) as a continuous, monotonically increasing function obtained via linear interpolation. The function measures the agent’s progress at turn t by the proportion of achieved subgoals, i.e., grading notes, assuming previously completed milestones cannot be undone. The AUC of the continuous progress function is then defined as AUC = R T 0 p(t)dt where T is the maximum turns of a conversation. For a given task sample (i,G i ) ∈ D, we define p 1 (t) and p 2 (t) to be the progress functions of two agents, respectively. Consider the case where both agents starts from 0 progress, i.e., p 1 (0) = p 2 (0) = 0 and first agent is strictly more efficient than the second, i.e., p 1 (t)≥ p 2 (t),∀t∈ (0,T], we have: AUC 1 = Z T 0 p 1 (t) dt > Z T 0 p 2 (t) dt = AUC 2 .(6) In this scenario, an efficient agent—compared to a less efficient one—will achieve a higher AUC score. The AUC rewards agent for achieving subgoals early which is crucial for long-horizon tasks such as navigation (Shridhar et al., 2020; Chevalier-Boisvert et al., 2018), where finding the right room or object early often reduces downstream confusion. Likewise, in multi-step planning tasks, like web browsing (Zhou et al., 2023), early retrieval of relevant results significantly narrows the search space, increasing likelihood of success. i) Early progress does not matter. While AUC metric favors early progress, one may argue that this is unnecessary in tasks like booking a trip, where reserving a plane and hotel are interchangeable subtasks, and order should not affect the outcome. In such cases, completing the simpler subtask with fewer subgoals first, followed by the more complex one (or vice-versa) should not affect the final score, i.e., case where two agents start with zero progress and reach the same progress within the same number of conversational turns, despite the differences in trajectories. To handle scenarios where early progress is not vital, one can weight the increase in progress uniformly by computing the progressperturn (PPT), forming a telescoping series: PPT = 1 T T−1 X t=0 p(t + 1)− p(t) = p(T) T ,(7) where p(t) is the discrete progress value at turn t, T is the minimum number of conversational turns to reach the final achieved progress p(T), and p(0) = 0. To align with the MaxProgressRate@k metric from equation 5, we report both the MaxAUC@k and MaxPPT@k, averaged over the task samples, while setting n = k. Further details are in the Appendix A.5. 3.3THE DIAGNOSIS STAGE Automated Error Analysis. Although a majority of existing works (Xiao et al., 2024; Qian et al., 2025) stop at reporting final dataset metrics, we argue that evaluation should also include error anal- ysis and actionable improvements. While using grading notes and LLM-as-a-judge simplify our evaluation, the inherent non-determinism of LLMs remains a challenge. Our proposed metrics ag- gregate results using a majority vote from judge runs and the best agent performance across k trials. However, the aggregation overlooks consistency—an essential aspect of robust agent evaluation. To address this, we further introduce an automated error analysis tool that analyzes both judge and agent inconsistencies by plotting sample-level progress expectations and variances, offering deeper insights on top of the final aggregated metrics. 5 Accepted as a conference paper at ICLR 2026 Error 5:Required Cancellation Tool Not Utilized Error 4: MissingBooking ToolInvocation Error 6: IncorrectArgumen- update_reservation_f lights Cluster 1: Missing Tool Usage - cancel_reservation Cluster 3: Incorrect Flight Selection or Argument in Booking Tool Error 1: Missing cancel_reservationtool invocation Error 3: Booking Tool Payment Allocation Failure Error 2: update_reservation_flight s-incorrect_arguments All samples i) Low-level Error Identification Cluster 2: Missing Tool Usage - book_reservation i) Semantic Clustering of Error Types 퓒=풇 clus 퓧,푮 풙=풇 iden 휺 Figure 1: Our proposed two-step automated error discovery approach that automatically identifies common errors of the agent based on judge and agent inconsistencies. Identical error colors indicate that similar low-level errors are clustered into the same high-level category. For each subgoal g i,j ∈ G i , we define a binary r.v. Z i,j , where Z i,j = 1 if the agent achieves the j-th subgoal under the given trajectory, and 0 otherwise. Let the probability of achieving the subgoal g i,j be Pr(Z i,j = 1) = z i,j . The progress for the sample (i,G i ) is defined as the proportion of subgoals the agent achieved, i.e., progress(i,G i ,τ l i ) = P j Z i,j |G i | . Its expectation and variance are given by: E[progress(i,G i ,τ l i )] = P j z i,j |G i | ;Var[progress(i,G i ,τ l i )] = P j z i,j (1− z i,j ) |G i | 2 ,(8) where z i,j = 1 Q P Q q=1 z (q) i,j is estimated by averaging over Q judge runs per subgoal, generalizing the single binary judge output in equation 2 to a probabilistic estimate. Plotting E[progress(i,G i ,τ l i )] and Var[progress(i,G i ,τ l i )] for each task (i,G i ) ∈ D, capture judge’s inconsistency through the variance, while agent’s inconsistency is reflected in the different expected progress values across the k trials. Building on this, we propose an automated error discovery approach that automatically identifies the common errors of the agent based on judge and agent inconsistencies. Our approach consists of two steps : (1) low-level error identification, and (2) semantic clustering of error types. For every binary score z (q) i,j from the judge, there is a corresponding explanation e (q) i,j . We define e i,j = e (1) i,j ,...,e (Q) i,j and the error candidate set E = (g i,j , e i,j ) | Pr(Z i,j = 1) < 1 to be a tuple of subgoals and corresponding explanations where the judge’s prediction is inconsistent or indicates the subgoal may not have been achieved. For each candidate error ε ∈ E , we first perform the low-level error identification step, followed by a semantic clustering step: x = f iden (ε); C = f clus (X,G),(9) where f iden (·) and f clus (·) is the error identification, and clustering prompt functions, respectively, and x ∈ X is the low-level error, andC is the cluster label. This clustering step will merge seman- tically similar errors into the same group and provide a high-level error summary. We illustrate this two-step process in Fig. 1. Note that the errors with the same color are merged into one cluster label. We also show preliminary results demonstrating agent improvement by leveraging the identified er- rors. For a detailed algorithm of our automated error analysis method, and the f iden (·) and f clus (·) prompt templates, refer to Appendix A.6. 4DATASETS AND EXPERIMENTAL SETUP We use two agent benchmarks: τ 2 -bench (Barres et al., 2025) and ToolSandbox (Lu et al., 2024). For τ 2 -bench, we utilize 21 and 25 samples from the airline and retail domains, respectively. For the airline domain, we further divide the samples into “easy” and “hard”. For ToolSandbox, we select 37 base scenarios and exclude variants with different initial messages or multi-turn conversations, as these can be effectively simulated using our dynamic user proxy—where both initial and subsequent 6 Accepted as a conference paper at ICLR 2026 Table 1: Overall performance of different agent models on τ 2 -bench airline and ToolSandbox, using gpt-4.1 as user proxy and LLM-as-a judge. Results are displayed with scores for Expert Persona| Non-expert Persona. For metrics with @k, the number of trials is n = k = 20 for τ 2 -bench and n = k = 8 for ToolSandbox. MaxProgressRate@k is abbreviated as MaxProg@k. Agent Model MeanProg@k MaxProg@k MaxAUC@k MaxPPT@k pass@k τ 2 -bench Airline Domain (Easy) gpt-4.10.95| 0.821.00| 1.000.99| 0.810.80| 0.501.00| 1.00 gpt-4o0.79| 0.861.00| 1.000.96| 0.860.70| 0.531.00| 1.00 gpt-4o-mini0.70| 0.610.90| 0.900.85| 0.730.60| 0.370.80| 0.80 gpt-50.92| 0.921.00| 1.000.97| 0.880.67| 0.541.00| 1.00 mistral-nemo0.87| 0.491.00| 0.800.97| 0.670.67| 0.481.00| 0.60 mistral-large0.65| 0.531.00| 1.000.96| 0.790.60| 0.421.00| 1.00 ToolSandbox Dataset gpt-4.10.91| 0.870.98| 0.970.96| 0.920.84| 0.730.92| 0.92 gpt-4o0.95| 0.940.99| 1.000.98| 0.960.94| 0.81 0.95| 0.97 gpt-4o-mini0.91| 0.850.95| 0.930.94| 0.900.89| 0.770.89| 0.84 gpt-50.78| 0.780.97| 0.910.95| 0.840.83| 0.66 0.95| 0.84 mistral-nemo0.72| 0.710.92| 0.960.88| 0.870.76| 0.650.84| 0.92 mistral-large0.82| 0.790.94| 0.950.93| 0.910.87| 0.750.89| 0.89 messages are generated dynamically, and the non-expert user persona effectively simulates multi- turn conversations. Our setup offers greater variability than the original variants with fixed initial messages. The base scenarios consist of a variety of task-oriented domains ranging from contact updates and messaging to reminders, currency conversion, etc. We use only milestones (key events that must happen) and convert them into grading notes. Importantly, any existing benchmark can be adapted to fit into our evaluation framework by converting the ground truths into grading notes. We set the maximum number of turns to 15 for τ 2 -bench and 8 for ToolSandbox. Each sample is evaluated over multiple agent trials, n = 20 trials for τ 2 -bench and n = 8 trials for ToolSandbox. We report metrics at k = n trials. We use the gpt-4.1 model as LLM-as-a-judge for grading the subgoals and for error identification and clustering in our experiments. Unless specified, the user proxy also uses the gpt-4.1 model. More details are in Appendix A.12. 5RESULTS AND DISCUSSION 5.1MAIN RESULTS Table 1 summarizes the overall performance of various agent models on τ 2 -bench airline domain and ToolSandbox, with gpt-4.1 serving as the user proxy. Additional results for τ 2 -bench re- tail domain are provided in Appendix A.10, Table 15. On easy airline samples, metrics such as MaxProgressRate@k and pass@k tend to saturate, with most models achieving near-perfect scores. MeanProg@k, which measures the average progress rate across all k trials, captures how consistently agents can achieve the subgoals. However, even MeanProg@k can remain high for strong models making it less effective at distinguishing between top-performing agents. While MaxProgressRate@k gives us the best agent performance over k trials, it fails to give any mean- ingful distinction between models, especially for easy samples. By incorporating MaxAUC@k and MaxPPT@k, we obtain a more comprehensive evaluation of agent performance. For example, on τ 2 -bench, gpt-4o-mini (expert) and mistral-large (expert) achieve similar MeanProgress@k scores (differing by only 5%). However, MaxAUC@k shows a larger difference of 10% (0.96 vs 0.85) and a change in rankings. Further comparison of the MaxAUC@k with MaxPPT@k scores for the two models, suggests that mistral-large achieves greater turn-level efficiency and faster progress in the initial turns, but both models have equal aver- age progress over turns as indicated by the identical MaxPPT@k scores. Similar pattern persists in the ToolSandbox dataset, where models such as gpt-5 and mistral-nemo have larger differences 7 Accepted as a conference paper at ICLR 2026 12345678 Turn 0.00 0.25 0.50 0.75 1.00 progress ( i , G i ) gpt-5 mistral-nemo (a) 12345678 Turn 0.00 0.25 0.50 0.75 1.00 progress ( i , G i ) gpt-5 mistral-nemo (b) 12345678 Turn 0.00 0.25 0.50 0.75 1.00 progress ( i , G i ) gpt-4.1 mistral-large (c) Figure2:ProgresscurvesforselectedToolSandboxsamples.(a) searchreminderwithrecencyupcoming:mistral-nemo(non-expert,pur- ple; AUC=0.88, PPT =0.20) vs.gpt-5 (non-expert, blue; AUC=0.61, PPT =0.20).(b) find currentcitylowbatterymode:mistral-nemo (expert, purple;AUC=0.77) vs. gpt-5 (non-expert, blue; AUC=0.64). (c) addremindercontentanddateandtime: gpt-4.1 (non-expert, green; AUC=0.50) vs. mistral-large (non-expert, red; AUC=0.34). Sample 14Sample 18Sample 22 0.0 0.2 0.4 0.6 0.8 1.0 [ progress ( i , G i , l i )] Trajectory T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 T20 Figure 3: E[progress(i,G i ,τ l i )] (dot) and Var[progress(i,G i ,τ l i )] (error bar) on τ 2 -bench gpt-4o- mini agent using non-expert user proxy gpt-4.1. Each dot is an agent trajectory from a single trial and each task sample from the airline domain is evaluated using n = k = 20 agent trials. We display only three example samples here. Sample 14 belongs to the hard split and others in the easy split. on the MaxAUC@k and MaxPPT@k metrics, when interacting with expert user, but a smaller difference on MaxProgressRate@k. We also examine the impact of user persona on agent performance. The non-expert user simulates an inexperienced user, resulting in agents taking more conversational turns to complete the task. This is consistently reflected in the MaxAUC@k scores, which are lower for the non-expert compared to the expert user persona across all models and datasets. It is because the expert persona provides clearer and more informative input, enabling the agents to complete tasks faster. The baseline metric MaxProgressRate@k which measures the agent best performance at the end of the conversation, overlooks turn count and thus show similar agent performances when interacting with expert versus non-expert user. The τ 2 -bench agent with gpt-4o-mini achieves the same MaxProgressRate@k score of 0.9 for expert and non-expert users, but more conversational turns are required during the interaction with non-expert user as shown in the lower MaxAUC@k and MaxPPT@k scores when compared to expert user. Another interesting observation is in some cases (e.g., gpt-5 vs mistral-large on τ 2 -bench), both models achieve the same MaxProgressRate@k = 1 for both user personas. However, when examining the MaxAUC@k metric, we see the performance gap between the two models is notably larger for agents interacting with non-expert user as compared to expert user. This highlights that the type of user interaction can significantly influence agent performance and should be considered as an important dimension when evaluating agents. Moreover, we know that the AUC metric emphasizes early progress, while PPT weights the in- creases in progress uniformly across turns. To illustrate this, we analyze the performance at the sample level. In Fig. 2, we showcase instances where two agents reach the same final progress 8 Accepted as a conference paper at ICLR 2026 but differ in the number of subgoals achieved at various turns. For example, in the ToolSandbox sample search reminderwithrecencyupcoming, both gpt-5 (non-expert) and mistral-nemo (non- expert) achieve a PPT of 0.20, yet their AUC scores are 0.61 and 0.88, respectively. The progress rate curves in Fig. 2a demonstrate that mistral-nemo makes rapid early progress, while gpt-5’s progress is more gradual. A closer examination of the agent trajectories (see Appendix A.13) re- veals that mistral-nemo executes tool calls in the initial turn and then seeks clarification, whereas gpt-5 begins by clarifying questions before invoking tools. Since the AUC is sensitive to agents that achieve subgoals earlier in the interaction, mistral-nemo has a much higher AUC score than gpt-5, as compared to the PPT metric which simply averages the increase uniformly. Hence, in scenarios where early progress is less relevant, the PPT metric may be more suitable. 5.2COMPARISON OF DIFFERENT AGENT MODELS USING THE EVALUATION PARADIGMS FROM THE ORIGINAL τ 2 -BENCH AND TOOLSANDBOX PAPERS Beyond demonstrating the additional insights provided by our metrics, we also compare the TED framework as a whole—including metrics and user proxy—with the evaluation paradigms from τ 2 - bench (Barres et al., 2025) and ToolSandbox (Lu et al., 2024). 5.2.1 τ 2 -BENCH ORIGINAL PAPER EVALUATION APPROACH In the Tables 2 and 3, we evaluated the τ 2 -bench agent on the airline and retail domains across different LLM models using the τ 2 -bench (Barres et al., 2025) original paper evaluation approach. For a fair comparison with our TED framework, the same set of samples and the same number of trials n = k = 20 are used in the experiments. Comparing Barres et al. (2025)’s approach (Table 2, left column) with our TED framework (Ta- ble 1) on the easy split of the τ 2 -bench airline domain, we observe that metrics such as pass@k (Yao et al., 2024) and passˆk (Barres et al., 2025) saturate for several models in Table 2, where performances are clustered into two different groups, revealing minimal information about the per- formance ranking of the different models. On the passˆ4 metric which is a form of consistency or reliability measure, model gpt-4.1 performs best, followed by gpt-4o, gpt-4o-mini, mistral-large, and finally mistral-nemo when evaluated using Barres et al. (2025)’s original approach on the airline easy split. Comparing this performance ranking with TED-evaluated results in Table 1, we observe change in rankings for several models. For instance, gpt-4o-mini on some metrics and setups in Table 1 performs worse than mistral-nemo which contradicts with the ranking in Table 2. In Table 1, mistral-nemo performs better than gpt-4o-mini across the different metrics for the expert user setup. For the non-expert user setup, we observe that mistral-nemo performs worse than gpt-4o-mini on the MaxAUC@k metric (0.67 vs 0.73) but better on the MaxPPT@k metric (0.48 vs 0.37). This suggests that under the non-expert user interaction, the agent with gpt-4o-mini exhibits a steeper per- formance improvement in the early interaction turns before plateauing (reflected by a higher AUC), but a lower average performance gain (indicated by a lower PPT ) compared to mistral-nemo. This finding highlights the importance of systematically analyzing agent’s performance not only based on the underlying LLM models but also based on the conversation quality between user and agent as determined by the user expertise. Additionally, we see the benefit of a joint interpretation of multiple metrics (AUC and PPT ) for a better understanding of the progression of agent performance which existing coarse-grained metrics like pass@k and passˆk fail to provide. Similarly, a change in model rankings can be observed for the τ 2 -bench agent on the full airline (easy+hard) and retail domains. In Table 2 (right column), the best performing model on the full airline domain is gpt-4.1 but this is not true for TED-evaluated results in Table 14, where the mistral models when interacting with the expert user generally show better or on par performance with the gpt-4.1 model on the MaxAUC@k, MaxPPT@k, and pass@k metrics. A similar behavior can be observed for the mistral-nemo and gpt-4o-mini models. As shown in the right column of Table 2, the agent using gpt-4o-mini outperforms mistral-nemo. This finding is consistent with our results for the non-expert user persona setting in Table 14, but it does not align with the expert user setting, where we observe a reversal in the relative ranking, with mistral-nemo outperforming gpt-4o-mini. We further illustrate this with an example in Fig. 4a and 4b by plotting the metric values for the two LLM models on a given sample from the airline domain. Based on Fig. 4a, we observe that the agent using mistral-nemo model, when interacting with an expert user, it achieves the maximum progress value of 1 in the first turn, resulting in an AUC = 1 which surpasses the performance of 9 Accepted as a conference paper at ICLR 2026 Table 2: Comparison of different agent models using the evaluation paradigm from the τ 2 -bench (Barres et al., 2025) original paper and their performances on the pass@k, passˆ4, and passˆk metrics in the airline domain. The number of trials used is n = k = 20. Agent Model τ 2 -bench Airline Domain (Easy)τ 2 -bench Airline Domain (Easy + Hard) pass@k passˆ4 passˆkpass@k passˆ4passˆk gpt-4.11.000.600.200.760.290.10 gpt-4o0.800.400.20 0.570.130.05 gpt-4o-mini1.000.280.200.670.100.05 mistral-nemo0.800.060.00 0.430.010.00 mistral-large1.000.160.000.710.070.00 Table 3: Comparison of different agent models using the evaluation paradigm from the τ 2 -bench (Barres et al., 2025) original paper and their performances on the pass@k, passˆ4, and passˆk metrics in the retail domain. The number of trials used is n = k = 20. Agent Model τ 2 -bench Retail Domain pass@k passˆ4 passˆk gpt-4.11.000.540.20 gpt-4o1.000.430.04 gpt-4o-mini0.920.230.04 mistral-nemo0.760.080.00 mistral-large0.960.330.08 gpt-4o-mini (i.e., AUC = 0.982) that completes all the defined subgoals only in the second turn. On the other hand, Barres et al. (2025)’s original evaluation approach in Fig. 4b aligns with our results on the non-expert user setting where the gpt-4o-mini model performs better than mistral- nemo model (AUC = 0.714 vs AUC = 0.4892). This further justify the importance of considering the user-agent conversation quality during evaluation. Likewise when comparing results on the retail domain, the model gpt-4.1 using the evaluation paradigm from the τ 2 -bench (Barres et al., 2025) original paper appears to be the best perform- ing model based on Table 3 but is not necessarily the case when comparing with the results obtained from our TED framework in Table 15. These observations show the significance of reporting mul- tiple metrics, e.g., AUC and PPT , to fully understand the task progression of the agent and the impact of user’s role on the agent performance. 123456789 0.0 0.5 1.0 progress ( i , G i , i ) mistral-nemo-expert gpt-4o-mini-expert 123456789 Turn 0.0 0.5 1.0 progress ( i , G i , i ) mistral-nemo-nonexpert gpt-4o-mini-nonexpert (a) pass@kpass^2pass^4pass^k Metric Name 0.0 0.2 0.4 0.6 0.8 1.0 Metric Value 1.0 0.00.00.0 1.0 0.15 0.01 0.0 mistral-nemo gpt-4o-mini (b) Figure 4: (a) τ 2 -bench airline sample 8: mistral-nemo expert (AUC=1.0, PPT =1.0) vs. gpt- 4o-mini expert (AUC=0.982, PPT =0.5) and mistral-nemo non-expert (AUC=0.482, PPT =0.25) vs. gpt-4o-mini non-expert (AUC=0.714, PPT =0.125) (b) τ 2 -bench airline sample8: Barres et al. (2025)’s original evaluation approach with n = k = 20 trials. 10 Accepted as a conference paper at ICLR 2026 5.2.2TOOLSANDBOX ORIGINAL PAPER EVALUATION APPROACH We evaluated the ToolSandbox agent across different LLM models using the original evaluation approach from ToolSandbox (Lu et al., 2024) and report the models’ performance in Table 4. For a fair comparison with our TED framework in Table 1, the same set of samples and the same number of trials n = k = 8 are used in the experiments. We do not report the final graph similarity score, which is a composite of milestone and minefield scores, because minefields are not considered in our setup. Instead, we consider only the milestone similarity score Score M+ and report the best performance of the model by taking the maximum milestone similarity score across the k trials as denoted by MaxScore M+ @k. Table 4: Comparison of different agent models using the evaluation paradigm from the ToolSandbox (Lu et al., 2024) original paper and their performances on the milestone similarity Score M+ metric. The number of trials used is n = k = 8. Agent ModelToolSandbox Dataset (Base Scenarios)(Base Scenarios + Variants) MaxScore M+ @kMaxScore M+ @k gpt-4.10.920.90 gpt-4o0.940.91 gpt-4o-mini0.900.88 mistral-nemo0.870.85 mistral-large0.910.89 In Table 4, we present the performance of different agent models on base scenarios (left column), as well as on both base scenarios and variants (right column). Based on the table, we observe that both setups exhibit the same performance ranking across models, with gpt-4o achieving the best perfor- mance, followed by gpt-4.1, and mistral-nemo performing the worst. However, this performance ordering differs from our main observation in Table 1 that is obtained using the TED framework. Our TED-based evaluation on the ToolSandbox dataset in Table 1 also shows gpt-4o outperforming gpt-4.1 across different metrics and user personas, but reveals a different performance ranking be- tween mistral-nemo and gpt-4o-mini. Based on Table 1, mistral-nemo outperforms gpt-4o-mini on the pass@k metrics for the non-expert user persona setting (0.92 vs 0.84) and vice-versa (0.84 vs 0.89) for expert user. For these two models, only the expert user setting in Table 1 aligns with the ranking observed in Table 4. This finding suggests that Lu et al. (2024)’s evaluation paradigm and the Score M+ metric in Table 4 may not always paint a complete story, as they do not effectively disentangle user behavior from the evaluation process. These observations highlight the importance of agent evaluation frameworks that account for user behavior, and demonstrate the additional values provided by our TED framework in which existing evaluation frameworks are lacking. 5.3ERROR ANALYSIS Besides reporting the results on the dataset-level, we analyze the expectations and variances of the multiple judge runs and the different k agent trials, as shown in Fig. 3. The different color points for each sample represent the different agent trial runs, which we refer to as trajectories in our discussion. For sample 14, we can clearly see that half of the agent runs have an expected per-sample progress of 0.6, while 35% of them are at 0.4, suggesting that there are one or more subgoals, where the agent consistently fails or succeeds. Upon examining the trajectories, we identify the problematic subgoal to be “Agent should cancel reservation K1NW8N.” Notably the trials in these two clusters has zero variance in the judge runs, indicating that the judge is highly confident in its assessment. For this sample, only trajectories 5 and 18 show relatively large variance bars, suggesting inconsistency in the judge model itself. We then apply our automated error analysis to identify the common errors made by the agent. Our tool identifies four distinct errors for sample 14 as shown in Fig. 5. In the trajectory 6, the agent did not check the details of the existing flight, which was supposed to be basic economy. Thus, the agent did not cancel the previous flight when attempting to reschedule causing a discrepancy in the final payment output. This error was consistently captured by our 11 Accepted as a conference paper at ICLR 2026 Figure 5: τ 2 -bench airline sample 14. The blue box shows a truncated task instruction i∈ I for the non-expert user proxy gpt-4.1 model. The green boxes contain the truncated dialogue for trajectory 6 (left) and agent’s trajectory 18 (right). The agent model is gpt-4o-mini. The top-right box shows the errors identified. Zoom in for a larger view. judge as indicated by the zero variance bar. On the other hand, trajectory 18 involves a different payment-related error whereby the agent hallucinates the value $2613.00, that exceeds the actual cost. This spurious value prevented the agent from calling book reservation, triggering a cascade of three subsequent errors. 5.4INCORPORATING IDENTIFIED ERRORS INTO AGENT’S DESIGN To demonstrate the effectiveness of our identified errors in improving the agent, we incorporate these errors into the design of the agent using several strategies on τ 2 -bench’s airline special split (selected due to their low progress rate and progress-per-turn) and ToolSandbox in Table 5. We examine several in-context learning approaches to incorporate the found errors into the agent instruction. We explore simple strategies such as Human Notes (HN) 3 where TED errors are manually refined before using them in the agent instructions, and the Error Insertion (EI) approach where TED errors are directly used without modification. Moreover, to demonstrate the effectiveness of the TED errors with other in-context learning approach such as the HiTEC-ICL (abbreviated as HTC) method (Cui et al., 2025), we conduct a comparison of the authors’ generic global errors versus the variant with our TED errors, denoted as HTC † in Table 5. The standard HTC method constructs global and local error checklists, which are injected as meta- data into the user instruction, simulating an artificial conversation loop to better guide the agent. The predefined global errors are highly generic, such as Empty Parameter Value Error or Missing Required Parameter Error. After the first round of user prompting with the global error checklist, the user model is also prompted again in the second round with the local error checklist, enabling the agent to recognize local errors for the relevant tools. For the HTC † variant, the generic global errors are replaced with TED errors, while the use of local error checklist remain consistent with the original method where detailed tool info are provided to the LLM to generate tool-specific error patterns. Based on Table 5, all four approaches show positive gains in agent performance for a majority of the metrics on ToolSandbox dataset. For the τ 2 -bench special split, we observe mixed trends, with both positive and negative performance gains over the baseline agents among the four approaches. Specifically for gpt-4o-mini, we observe a significantly larger performance gain on τ 2 -bench for the HTC † variant using TED errors as compared to the standard HTC method that uses predefined 3 E.g., “You must strictly check and double confirm all requirements for change flight actions before calling the tool. If you are unsure or confused, always ask clarifying questions to the user.” 12 Accepted as a conference paper at ICLR 2026 Table 5: Agent improvement results over the baseline agent for Expert Persona| Non-expert Persona, using gpt-4.1 as user proxy and LLM-as-a judge. Blue↑ shows an improvement over the baseline agent, and red↓ shows a decrease. We compare the gain in performance for several agent models using the Error Insertion (EI), Human Notes (HN), HiTEC-ICL (HTC) (Cui et al., 2025), and HTC † variant methods. Agent ModelMeanProg@k MaxProg@k MaxAUC@k MaxPPT@k τ 2 -bench Airline Domain (Special Split: Samples 7, 14, 21, 23, and 29) gpt-4o-mini + EI 0.37↑0.09| 0.31↓0.01 0.66↑0.06| 0.61±0.00 0.59↑0.05| 0.39↓0.02 0.27↑0.03| 0.11↓0.01 gpt-4o-mini + HN0.30↑0.02| 0.33↑0.01 0.63↑0.02| 0.61±0.00 0.53↓0.01| 0.45↑0.04 0.24±0.00| 0.14↑0.02 gpt-4o-mini + HTC 0.29↑0.01| 0.26↓0.06 0.54↓0.06| 0.51↓0.10 0.50↓0.04| 0.41±0.00 0.32↑0.08| 0.13↑0.01 gpt-4o-mini + HTC † 0.33↑0.05| 0.33↑0.01 0.61↑0.01| 0.65↑0.04 0.57↑0.03| 0.43↑0.02 0.29↑0.05| 0.13↑0.01 gpt-4.1 + EI0.52±0.00| 0.38±0.00 0.78↑0.04| 0.77↓0.08 0.66↑0.02| 0.41↓0.06 0.26↓0.01| 0.10↓0.02 gpt-4.1 + HN0.53↑0.01| 0.41↑0.03 0.78↑0.04| 0.85±0.00 0.66↑0.02| 0.55↑0.08 0.27±0.00| 0.14↑0.02 gpt-4.1 + HTC0.54↑0.02| 0.39↑0.01 0.82↑0.08| 0.73↓0.12 0.67↑0.03| 0.41↓0.06 0.27±0.00| 0.10↓0.02 gpt-4.1 + HTC † 0.55↑0.03| 0.40↑0.02 0.71↓0.03| 0.82↓0.03 0.63↓0.01| 0.51↑0.04 0.24↓0.03| 0.12±0.00 ToolSandbox Dataset gpt-4o-mini + EI0.87↓0.04| 0.89↑0.04 0.97↑0.02| 0.98↑0.05 0.94±0.00| 0.91↑0.01 0.85↓0.04| 0.66↓0.11 gpt-4o-mini + HN0.88↓0.03| 0.91↑0.06 0.96↑0.01| 0.96↑0.03 0.95↑0.01| 0.92↑0.02 0.90↑0.01| 0.74↓0.03 gpt-4o-mini + HTC 0.93↑0.02| 0.92↑0.07 0.97↑0.02| 0.98↑0.05 0.96↑0.02| 0.95↑0.05 0.94↑0.05| 0.83↑0.06 gpt-4o-mini + HTC † 0.92↑0.01| 0.91↑0.06 0.98↑0.03| 0.98↑0.05 0.97↑0.03| 0.95↑0.05 0.93↑0.04| 0.82↑0.05 gpt-4.1 + EI0.95↑0.03| 0.93 ↑0.06 0.99↑0.01| 0.99 ↑0.02 0.97↑0.01| 0.93 ↑0.01 0.87↑0.03| 0.76 ↑0.03 gpt-4.1 + HN0.95↑0.03| 0.97 ↑0.10 0.98±0.00| 0.99 ↑0.02 0.97↑0.01| 0.95 ↑0.03 0.91↑0.07| 0.83 ↑0.10 gpt-4.1 + HTC0.95↑0.03| 0.95↑0.08 0.98±0.00| 0.99↑0.02 0.96±0.00| 0.95↑0.03 0.89↑0.05| 0.75↑0.02 gpt-4.1 + HTC † 0.95↑0.03| 0.95↑0.08 1.00↑0.02| 0.99↑0.02 0.99↑0.03| 0.96↑0.04 0.92↑0.08| 0.82↑0.09 generic global errors. However, for gpt-4.1 on τ 2 -bench samples, there is no clear trend suggesting that one performs better than the other. We also find that the Errors Insert strategy improves several setups—notably gpt-4o-mini on τ 2 - bench (+9% in MeanProg@k, +5% in MaxAUC@k)—implying that awareness of common fail- ures helps the agent perform better, with only a few showing declines. In contrast, the Human Notes strategy gives a more consistent improvement for more setups as compared to Error Insert. We ob- serve in particular a significant gain in MaxPPT@k of 7-10% for gpt-4.1 using Human Notes on ToolSandbox dataset. We want to emphasize that, our work only evaluates the usefulness of the TED errors—as indicated by improvements over baseline agents (blue upward arrows)—and does not propose a new in-context learning method to construct better agent instruction prompts. 5.5EXTENDED EXPERIMENTS AND HUMAN VALIDATION We also present results on the full split of τ 2 -bench airline and retail domains in Appendix A.10, and include an ablation on user model variation in Appendix A.11. To validate our evaluation, we conduct human studies on the correctness of the user proxy, LLM-as-a-judge, and the identified TED errors as reported in Appendices A.7, A.8, and A.9, respectively. Results show that the user proxy behaves correctly in most cases, with only 6–12% errors due to instruction-following issues. The LLM-as-a-judge human study also shows high agreement between human rater and the LLM- as-a-judge prediction. These observations suggest both components in the TED framework remain reliable and cost-effective. More details are discussed in Appendices A.7 to A.9. 13 Accepted as a conference paper at ICLR 2026 6CONCLUSION AND FUTURE WORK In this work, we introduced the TED framework that redefines agent evaluation. We showed that including error insights into the agent’s design leads to gains, with peaks of 8% for MaxAUC@k and 10% for MaxPPT@k metrics. In the future, we plan to explore the applicability of our metric to non-task-oriented domains, such as open-ended dialogue with conversational agents, where the expected responses of the agent can be assessed using our grading notes. Limitation of our approach and LLM usage are discussed in Appendix A.1 and A.2, respectively. REFERENCES Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Ling- peng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37:74325–74362, 2024. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272, 2018. Yue Cui, Liuyi Yao, Shuchang Tao, Weijie Shi, Yaliang Li, Bolin Ding, and Xiaofang Zhou. En- hancing tool learning in large language models with hierarchical error checklists. arXiv preprint arXiv:2506.00042, 2025. Yupu Hao, Pengfei Cao, Zhuoran Jin, Huanxuan Liao, Yubo Chen, Kang Liu, and Jun Zhao. Evalu- ating personalized tool-augmented llms from the perspectives of personalization and proactivity. arXiv preprint arXiv:2503.00771, 2025. Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal, Prafulla Kumar Choubey, Yixin Mao, Silvio Savarese, Caiming Xiong, and Chien-Sheng Wu. Crmarena-pro: Holistic assessment of llm agents across diverse business scenarios and interactions. arXiv preprint arXiv:2505.18878, 2025. Kyochul Jang, Donghyeon Lee, Kyusik Kim, Dongseok Heo, Taewhoo Lee, Woojeong Kim, and Bongwon Suh. Dice-bench: Evaluating the tool-use capabilities of large language models in multi-round, multi-party dialogues. arXiv preprint arXiv:2506.22853, 2025. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023. Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. arXiv preprint arXiv:2408.04682, 2024. Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, et al. Escapebench: Towards advancing creative intelli- gence of language model agents. arXiv preprint arXiv:2412.13549, 2024. Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-T ̈ ur, Gokhan Tur, and Heng Ji. Smart: Self-aware agent for tool overuse mitigation. arXiv preprint arXiv:2502.11435, 2025. Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation. arXiv preprint arXiv:2410.07869, 2024. 14 Accepted as a conference paper at ICLR 2026 Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆ ot ́ e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309.10691, 2023. Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flowbench: Revisiting and benchmarking workflow-guided planning for llm-based agents. arXiv preprint arXiv:2406.14884, 2024. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv. org/abs/2406.12045, 2024. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307.13854, 2023. 15 Accepted as a conference paper at ICLR 2026 APPENDIX CONTENTS A.1 Limitation of our approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 A.2 LLM Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 A.3 Prompt templates for user proxy . . . . . . . . . . . . . . . . . . . . . . . . . . .17 A.4 Prompt template for LLM-as-a-judge . . . . . . . . . . . . . . . . . . . . . . . . .20 A.5 Additional details on progress and turn-level efficiency metrics . . . . . . . . . . .22 A.6 Additional details on automated error analysis . . . . . . . . . . . . . . . . . . . .22 A.7 Human validation on the correctness of user proxy . . . . . . . . . . . . . . . . .26 A.8 Human validation on the correctness of LLM-as-a-judge . . . . . . . . . . . . . .26 A.9 Human validation on the correctness of the identified TED errors . . . . . . . . . .27 A.10 Additional experiments on the τ 2 -bench dataset . . . . . . . . . . . . . . . . . . .28 A.11 Ablation study on the different user models . . . . . . . . . . . . . . . . . . . . .34 A.12 Additional information on dataset and experimental setup. . . . . . . . . . . . .35 A.13 Additional details on agent trajectories . . . . . . . . . . . . . . . . . . . . . . . .40 16 Accepted as a conference paper at ICLR 2026 AAPPENDIX A.1LIMITATION OF OUR APPROACH Our approach, which uses grading notes and LLM-as-judge, simplifies evaluation by relying solely on the agent’s trajectory, without requiring access to the underlying environment. However, this approach has certain limitations. While we show that multiple judge runs improves reliability and our automated error analysis tool helps in debugging, the method cannot verify whether the database state has actually changed when such modifications are not reflected in the trajectory. Consequently, this limits our ability to capture silent failures that produce no observable outputs. A.2LLM USAGE LLM is used to assist the writing of UI code for the automated error analysis tool. In addition to that, we use LLM to refine the prompt templates that are used in our experiments. LLM is also used to refine and polish the text in the paper to improve clarity and presentation. A.3PROMPT TEMPLATES FOR USER PROXY The following are the reusable, generic expert and non-expert user prompt templates, followed by the templates for the two-step function f in equation 1. Theuser tasksummary placeholder corresponds to the task instruction i ∈ I , and the agentdesc placeholder corresponds to the agent description. For the two-step generation process f -reflection followed by response, the placeholderschathistory andterminationmsg represent the user-agent chat history up to the current stage of conversation and the termination message that the user should produce at the end of the dialogue, respectively. The placeholder reflectionhistory represents the user reflection history up to the current stage of conversation. Generic expert user persona prompt template: You are acting as an expert LLM-simulated user who fully understands the AI assistant system and goal. Always respond naturally in clear, concise language that fits the expert user role and goal. Provide complete and precise information in your responses. Generate one line at a time. Do not give away all the instructions at once. Only provide the information that is necessary for the current step. You are provided with the following user task summary: [user tasksummary] usertasksummary You understand the system well and will provide thorough, accurate responses using only the information provided in the [usertasksummary] section. If the AI assistant returns output in JSON format, respond only to the content inside the JSON as if the format does not matter. --- The following provides an overview of the AI assistant if available. [AI Assistant Description] : agent desc --- When you as an expert LLM-simulated user is analysing the real-time chat history, carry out a two-step process as the user: first, a Reflection Phase, followed by a Response Generation Phase. 17 Accepted as a conference paper at ICLR 2026 Generic non-expert user persona prompt template: You are simulating a clueless, casual NON-expert user who is interacting with an AI assistant. You don’t fully understand how the AI system works, and you tend to give vague or incomplete instructions — often leaving out key steps or context. When you respond: Speak naturally, casually, like someone who’s unsure how to talk to an AI. Be brief and only provide part of the needed information. Do not give a full picture unless the assistant directly asks for it. Only share details that are directly related to what was just asked or prompted — not more. Never proactively explain your reasoning or provide background info unless the assistant digs into it. You are working toward the following general task: [User Task Summary] user tasksummary But since you’re not an expert, you’l just sort of ”feel your way through it” and leave lots of gaps in your instructions.NEVER provide COMPLETE instructions.ALWAYS OMIT some variables and missing key context. If the assistant returns something in structured formats like JSON, you can just react casually to the content. Treat the format like it doesn’t matter. --- The following provides an overview of the AI assistant if available. [AI Assistant Description]: agent desc --- When you as a clueless, casual NON-expert user is analysing the real-time chat history, carry out a two-step process as the user: first, a Reflection Phase, followed by a Response Generation Phase. When simulating your process during the conversation: You go through two internal steps each time: 1. Reflection Phase (internal thought): Take a quick look at the current chat history. Think to yourself: “Okay, what did the assistant just say or ask? What should I probably say next without overexplain- ing?” Remember: you’re not confident in how this system works, so don’t try to be precise. 2. Response Generation Phase (your reply): Now write a short, casual message that gives only partial information based on what the assistant asked. Leave things unclear unless the assistant is persistent. 18 Accepted as a conference paper at ICLR 2026 The reflection-step prompt template in the two-step functionf equation 1: --- The following [Chat History] (if available) provides context and indicates the CURRENT stage of your conversation as a LLM-simulated user with the AI assistant. [Chat History] chat history --- Step 1: Reflection Phase Given the [Chat History] REFLECT carefully on the AI assistant’s last response and what the LLM-simulated user is trying to accomplish based on the [user tasksummary]. Briefly address: - Your role as the LLM-simulated user. - The current stage of the conversation. You SHOULD NOT skip any user instructions as mentioned in the [user tasksummary]. - The assistant’s last reply in the [Chat History]. IMPORTANT CLARIFICATION: - Review the entire [Chat History] and the [usertasksummary] and see what should be your next response as a LLM-simulated user. - At times, the AI assistant’s last message may overlap with or anticipate a future user turn. In such cases, treat it strictly as the AI assistant response, not a replacement of the user message Do NOT generate the LLM-simulated user response yet. RESPOND only with a REFLECTION. **IMPORTANT** remember your user persona as written in the system prompt (eg: expert user or non-expert) and respond with appropriate reflection. TERMINATE ONLY IF the conversation is at its FINAL STAGE where the agent has completed all the tasks wanted by the user as shown in the [user tasksummary]. If the conversation has concluded, prepare to respond withtermination msg in the next re- sponse generation phase. Otherwise, DO NOT consider termination if the current conversation is not at its final stage. 19 Accepted as a conference paper at ICLR 2026 The response-step prompt template in the two-step functionf equation 1: --- The following [Chat History] (if available) provides context and indicates the CURRENT stage of your conversation as a LLM-simulated user with the AI assistant. [Chat History] chat history --- The following is the LLM-simulated user reflection. [Reflection] reflection history --- Step 2: Response Generation Phase Given the [Chat History] and [Reflection], GENERATE the LLM-simulated user NEXT RE- SPONSE that: i) Naturally continues the conversation WITHOUT ADDING NEW TASK that is NOT found in the [user tasksummary]. You SHOULD NOT skip any tasks for the LLM-simulated user. i) Avoids revealing or repeating the AI assistant’s answers. iv) Responds appropriately to the assistant’s actual reply, even if vague or off-track. If the AI assistant’s last message echoes or resembles any part of a user message, it’s the AI assistant response, NOT a new user turn. Note that suggestions or recommendations by the AI assistant should NEVER be MISTAKEN for actual actions taken. GENERATE the LLM-simulated USER RESPONSE based on the [Reflection].Return ONLY the LLM-simulated user response. **IMPORTANT** remember your user persona as written in the system prompt (eg: expert user or non-expert) and respond with appropriate response. TERMINATE ONLY IF the conversation is at its FINAL STAGE where the agent has com- pleted all the tasks wanted by the user as shown in the [user tasksummary]. If the conversation has concluded, prepare to respond withterminationmsg in the next response generation phase. Otherwise, DO NOT consider termination if the current conversation is not at its final stage. A.4PROMPT TEMPLATE FOR LLM-AS-A-JUDGE The following is the LLM judge (i,g i,j ,τ i ) prompt template from equation 2.The usertasksummary placeholder corresponds to the task instruction i ∈ I , the gradingnote placeholder corresponds to the j-th grading notes g i,j for the task instruction i. Theremainingplaceholders, trajectory, agentresponsesand dynamicDialogue represent agent’s trajectory, responses output, and the user-agent dialogue, respectively, as extracted from τ i . 20 Accepted as a conference paper at ICLR 2026 Prompt Template for LLM-as-a-judge equation 2: You are provided with a sample that contains several key components centered around an interaction between an agent and a simulated user, referred to as the user proxy. The user proxy represents a human-in-the-loop, engaging with the agent by posing questions and guiding the conversation throughout the dialogue. The [User Summary Instructions] section outlines the user’s goals, expectations, and the over- all task the agent is expected to complete. The [Agent Responses Submission] section captures the agent’s actual responses to the user proxy at each turn of the interaction. The [Agent Intermediate Trajectories] section provides a detailed step-by-step reasoning and actions taken by the agent. Finally, the [Dynamic Dialogue] section presents the full conversation between the agent and the user proxy. As a judge, your task is to determine whether the subgoal specified in the [Ground Truth Sub- goal] has been achieved. To make this assessment, evaluate the [Agent Intermediate Trajectories] and [Agent Responses Submission] to ascertain whether the subgoal has been successfully completed. When the completion of a subgoal depends on the availability of a specific tool, and that TOOL is ABSENT, the subgoal must be marked as INCOMPLETE, REGARDLESS of the correctness of the agent’s responses in [Agent Responses Submission]. Tool availability is a hard constraint—no amount of valid reasoning can compensate for its absence. Suggesting or recommending a tool is acceptable only if the subgoal does not require the tool to be used. However, CONCEPTUALIZING or HALLUCINATING a tool call—i.e., falsely implying the tool was invoked—is NOT acceptable, and the subgoal must be marked as INCOMPLETE. Additionally if the agent intermediate trajectory are left BLANK or DOES NOT contain rele- vant tools to the subgoal, it means that the ground truth subgoal is NOT MET and the subgoal is considered as INCOMPLETE, REGARDLESS of the correctness of the agent’s responses in [Agent Responses Submission]. The thought in [Agent Intermediate Trajectories] CANNOT be considered as a substitute for a tool call and the ground truth subgoal is considered to be INCOMPLETE. [BEGIN DATA] ************ [User Summary Instructions]: user tasksummary ************ [Ground Truth Subgoal]: grading note ************ [Agent Intermediate Trajectories]: trajectory ************ [Agent Responses Submission]: agentresponses ************ dynamicDialogue [END DATA] During assessment focus solely on the factual content and the goal completion while disregarding any differences in style, grammar, punctuation, or syntax. Begin by presenting a concise argument to confirm the validity of your conclusion.Avoid simply stating the correct answers at the outset. Decide what type of tools is required and then end with your answer formatted as ’GRADE: $LETTER’ (without quotes) where LETTER is one of C or I. Reply with ’GRADE: C’ (without quotes) to indicate COMPLETE if the agent has successfully achieved the subgoal. Otherwise, reply with ’GRADE: I’ (without quotes) to indicate INCOMPLETE if the agent did not achieved the subgoal. 21 Accepted as a conference paper at ICLR 2026 A.5ADDITIONAL DETAILS ON PROGRESS AND TURN-LEVEL EFFICIENCY METRICS To align with the MaxProgressRate@k metric from equation 5, which evaluates the agent’s best performance across k trials, we report both the max AUC and max PPT of the k trials averaged over the task samples: MaxAUC@k = E (i,G i )∼P D [ maxAUC l | l = 1,...,k].(10) MaxPPT@k = E (i,G i )∼P D [ maxPPT l | l = 1,...,k].(11) We show in our experiments that the proposed metrics provide interesting insights into the agent behavior that existing metrics failed to capture. A.6ADDITIONAL DETAILS ON AUTOMATED ERROR ANALYSIS The error candidate set E = (g i,j , e i,j ) | Pr(Z i,j = 1) < 1 can have two situations: i) when all the judge trials consistently score 0, i) judge model has disagreement across multiple judge trials. For the first case, we can select any e i,j of the Q trials to get the final x i,j . Usually in our implementation, we select the first explanation e 1 i,j . However, for second case, the f iden will take all e i,j and apply another selective prompt function f selective to decide the low-level error x. We illustrate the entire algorithm in the below pseudo-code: Algorithm 1: Automated Error Analysis Method Input: Judge outputs(Z (q) i,j ,e (q) i,j ,g i,j ) for samples i, subgoals j, trials q = 1...Q. Output: High-level error typesC 1 <---STEP1: Low-level error identification---> 2 InitializeE ←∅,X ←∅ 3 for each (i,j) do 4 if (∀q, Z (q) i,j = 0) or (0∈ Z i,j ∧ 1∈ Z i,j ) then 5 E ←E ∪(g i,j , e i,j ) 6if∀q, Z (q) i,j = 0 then 7// Consistent failure: 8// Select the first judge explanation for error identification 9x i,j ← f iden (g i,j ,e (1) i,j ); 10end 11else 12// Disagreement across judge trials: 13// Iterate on all the judge explanation 14Tmp← [] 15for q = 1 to Q do 16Tmp← Tmp∪f iden (g i,j ,e (q) i,j ) 17end 18x i,j ← f selective (Tmp) 19end 20 X ←X ∪x i,j 21 end 22 end 23 <---STEP2: Semantic clustering of error types---> 24 G ←g i,j | (i,j)∈E 25 C ← f clus (X,G) 26 return high-level error typesC 22 Accepted as a conference paper at ICLR 2026 Prompt Template forf iden in Automated Error Analysis: You are tasked with summarizing the error type in a concise and abstract manner based on the provided explanation. This explanation is generated by a judge model, which evaluates whether the agent’s response satisfies the specified subgoals. In the explanation, a grade of ”C” (Complete) indicates success, while ”I” (Incomplete) indicates failure. Your goal is to produce an error type that: • Clearly captures the core failure or issue at an abstract level. • Avoids restating the explanation verbatim. • Is short, specific, and phrased like a category label rather than a long sentence. • Does not include sensitive details or unnecessary context. • If the error type involves tool usage, explicitly include the tool name in the error type. You will be provided with: • Ground truth subgoals • Judge model’s explanation of the agent’s response [BEGIN DATA] *** [Ground Truth Subgoals] :subgoals *** [Explanation] :explanation *** [END DATA] Please return your output in the following **strict JSON format**: "error_type": "<error_type>", "explanation": "<explanation>" 23 Accepted as a conference paper at ICLR 2026 Prompt Template forf selective in Automated Error Analysis: You are given multiple independent predictions of the same data row. Each prediction includes an error type assigned by a model. Your task is to determine the **most probable true error type** using a majority voting approach. Instructions: 1. Review all provided error types carefully. 2. Group similar or semantically equivalent error types together, even if their wording differs. 3. Count how many times each grouped error type appears. 4. Select the error type with the highest count as the final result. 5. If there is a tie: - Prefer the error type that is more specific and informative. - If still tied, choose the one most consistent with the majority wording. 6. Output ONLY the most probable error type, without extra commentary. [BEGIN DATA] [Error Types] :error typelist [END DATA] Please return your output in the following **strict JSON format**: "most_probable_error_type": "<most_probable_error_type>" 24 Accepted as a conference paper at ICLR 2026 Prompt Template forf clus in Automated Error Analysis: You are tasked with clustering the following error types based on their semantic similarity. The goal is to group related error types under broader, more abstract categories to reduce redundancy and improve generalization. Important: • The cluster label should be primarily grounded in the <subgoals> provided. • Only if you are very certain that an error type is entirely unrelated to the subgoals should you create a new cluster label not derived from them. • Always aim to preserve the subgoal’s intent when naming clusters. Guidelines: • Error types are not mutually exclusive and may overlap in meaning. • Each cluster should reflect the most abstract and inclusive label that unifies all error types within it. • Do not merge error types referring to different tools into a single cluster. • Clusters involving tool usage must be separated by tool name. • The cluster label for each tool-related cluster must explicitly include that tool’s name. • Minimize the number of clusters while maintaining clear and meaningful distinctions. • Avoid overly specific wording—cluster labels should be reusable in other contexts where the same subgoal applies. [BEGIN DATA] *** [Subgoals] :subgoals *** [Error Types] :error types *** [END DATA] Please return your output in the following strict JSON format: "clusters": [ "cluster_label": "<generalized_error_type>", "error_types": ["<error_type_1>", "<error_type_2>", ...], "error_ids": ["<error_id_1>", "<error_id_2>", ...] , ... ] 25 Accepted as a conference paper at ICLR 2026 A.7HUMAN VALIDATION ON THE CORRECTNESS OF USER PROXY To ensure the reliability of our user proxy simulation, we manually validate user proxy utterances through human evaluation on 16 randomly selected expert and 16 non-expert user-agent dialogues on τ 2 -bench airline domain and ToolSandbox dataset. We categorize errors into three types: (1) user role confusion, where the user mistakes their role for the agent’s; (2) failure to follow the specified task instructions i ∈ I , termed as missing or violate instructions; and (3) nonsensical or erroneous user responses. Based on Table 6, we observe that the user proxy in general behaves as expected except for a small number of cases where it does not follow the task instructions. While no AI system can be expected to achieve 100% accuracy, the low number of such errors supports our belief in the user proxy’s inherent potential for agent evaluation. Table 6: Correctness of user proxy. Both agent and user proxy use gpt4.1 model. User Persona Errors Role confusionMissing or violate instructionsErroneous responses Expert0.00.060.0 Non-expert0.00.1250.0 A.8HUMAN VALIDATION ON THE CORRECTNESS OF LLM-AS-A-JUDGE To ensure the reliability of our LLM-as-a-judge evaluation, which uses grading notes (i.e., subgoals) as ground truths, we conducted multiple runs of the judge and used majority vote to determine the final scores. The human validation results as shown in Tables 7 to 10 are conducted on these majority-vote outcomes. Based on Table 7, for the gpt-5 agent model, we randomly select 10 sam- ples containing both expert and non-expert users from the τ 2 -bench airline domain, and another 10 samples from the ToolSandbox dataset, resulting in 42 and 31 subgoals, respectively. Since each subgoal is evaluated independently, we report human validation results using Cohen’s Kappa at the subgoal level. Human annotators are asked to label each subgoal prediction as “success”, “failure”, or “ambiguous”, where “ambiguous” denotes cases that could not be clearly classified as either “success” or “failure”. For “ambiguous” cases, we treat them as being in agreement with the judge prediction. In Table 7, we observe a high Cohen’s Kappa score (0.84 to 0.92) indicating an almost perfect agreement between the LLM-as-a-judge and human rater. Similar high agreement scores for gpt- 4.1, gpt-4o, and gpt-4o-mini agents are also shown in the Tables 8, 9, and 10, respectively. These agreement scores reinforce that our approach using grading notes and LLM-as-a-judge offers a reli- able, scalable, and cost-effective alternative to other more complex evaluation methods. Table 7: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent uses gpt-5 model while the LLM-as-a-judge and the user proxy use gpt-4.1 model. DatasetCohen’s Kappa τ 2 -bench airline (42 subgoals from 10 samples)0.84 ToolSandbox (31 subgoals from 10 samples)0.92 Table 8: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent, LLM-as-judge, and user proxy use gpt-4.1 model. DatasetCohen’s Kappa τ 2 -bench airline (37 subgoals from 9 samples)0.71 ToolSandbox (40 subgoals from 10 samples)0.60 26 Accepted as a conference paper at ICLR 2026 Table 9: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent uses gpt-4o model while the LLM-as-a-judge and the user proxy use gpt-4.1 model. DatasetCohen’s Kappa τ 2 -bench airline (27 subgoals from 10 samples)0.90 ToolSandbox (32 subgoals from 10 samples)0.80 Table 10: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent uses gpt-4o-mini model while the LLM-as-a-judge and the user proxy use gpt-4.1 model. DatasetCohen’s Kappa τ 2 -bench airline (30 subgoals from 10 samples)0.93 ToolSandbox (35 subgoals from 10 samples)0.94 A.9HUMAN VALIDATION ON THE CORRECTNESS OF THE IDENTIFIED TED ERRORS To ensure the relevance and accuracy of the errors identified by our TED framework, we manually validate them against the reference labels. Similarly, since errors are identified based on subgoals, we report human validation results at the subgoal level. Human annotators first identify and catego- rize errors using only the agent’s trajectory and the ground-truth subgoals. These annotated errors serve as reference labels. To measure disagreement, they subsequently evaluate whether the TED- identified errors are semantically consistent with their own reference error annotations. For different agent models, we observe only a small disagreement (6–23%) between the errors identified by TED and the reference errors. The detailed results are shown in the Tables 11, 12, and 13. This level of disagreement is reasonable, as error identification is challenging even for human annotators. Table 11: Correctness of the identified TED errors. The agent, LLM-as-a-judge, and user proxy use the gpt-4.1 model. Dataset Per-subgoal error Disagreement with human τ 2 -bench airline (37 subgoals from 9 samples)0.22 ToolSandbox (40 subgoals from 10 samples)0.23 Table 12: Correctness of the identified TED errors. The agent uses gpt-4o model while the LLM- as-a-judge and the user proxy use gpt-4.1 model. Dataset Per-subgoal error Disagreement with human τ 2 -bench airline (27 subgoals from 10 samples) 0.19 ToolSandbox (32 subgoals from 10 samples)0.09 Table 13: Correctness of the identified TED errors. The agent uses gpt-4o-mini model while the LLM-as-a-judge and the user proxy use gpt-4.1 model. Dataset Per-subgoal error Disagreement with human τ 2 -bench airline (30 subgoals from 10 samples)0.23 ToolSandbox (35 subgoals from 10 samples)0.06 27 Accepted as a conference paper at ICLR 2026 A.10ADDITIONAL EXPERIMENTS ON THE τ 2 -BENCH DATASET Besides analyzing the agent performance on the easy airline domain samples, we extend our analysis to include hard samples as well. Table 14 presents the performance of various agent models under easy and more challenging setting on the airline domain. We gain some insights into agent behavior by including hard samples. First, we observe a general decline across all metrics, reflecting the increased difficulty of the sam- ples. MeanProg@k and pass@k scores drop substantially for all models, indicating that agents are less likely to achieve full task completion on harder samples. While MaxProgressRate@k remains relatively high for most models (e.g., gpt-4.1, gpt-5, mistral-nemo, for expert persona), this metric suggests that most agents can achieve near to completion progress at least one out of the n = k trials. When we shift our focus to MaxAUC@k, the model ranking changes noticeably (eg: gpt-5, mistral- nemo, mistral-large for expert persona). This shift highlights that while many models can achieve near to completion on the difficult tasks, only a few do so efficiently. This effect is more pro- nounced for the non-expert persona: LLM agents using gpt-4.1 and gpt-4o models maintain high MaxProgressRate@k (0.96 and 0.88), but their MaxAUC@k scores are much lower (0.68 and 0.65). Anotherinterestingobservationisthatnon-expertpersonasometimeshashigher MaxProgressrate@k (e.g., gpt-5, gpt-4o-mini).Upon closer examination, we found in- stances where the expert persona provided all relevant information in the very first turn, which overwhelmed the agent and led to hallucinations. In contrast, the non-expert persona distributes information gradually over multiple turns, allowing the agent to respond more effectively. This finding highlights the impact of user interaction style on agent performance. Table 14: Overall performance of different agent models on τ 2 -bench airline domain, using gpt-4.1 as user proxy and LLM-as-a-judge. Dataset contains easy and hard samples. Results are displayed with scores for Expert Persona| Non-expert Persona. For metrics with @k, the number of trials is n = k = 20. Agent Model MeanProg@k MaxProg@k MaxAUC@k MaxPPT@k Pass@k τ 2 -bench Airline Domain (Easy + Hard) gpt-4.10.75| 0.670.94| 0.960.85| 0.680.44| 0.250.81| 0.86 gpt-4o0.63| 0.550.91| 0.880.84| 0.650.44| 0.230.81| 0.71 gpt-4o-mini0.53| 0.530.86| 0.880.79| 0.660.43| 0.260.62| 0.76 gpt-50.80| 0.770.96| 0.970.89| 0.770.47| 0.30 0.91| 0.91 mistral-nemo0.67| 0.360.96| 0.710.87| 0.560.44| 0.250.86| 0.38 mistral-large0.54| 0.510.94| 0.930.87| 0.680.43| 0.240.86| 0.76 Table 15: Overall performance of different agent models on τ 2 -bench retail domain, using gpt-4.1 as user proxy and LLM-as-a-judge. Results are displayed with scores for Expert Persona| Non-expert Persona. For metrics with @k, the number of trials is n = k = 20. Agent Model MeanProg@k MaxProg@k MaxAUC@k MaxPPT@k Pass@k τ 2 -bench Retail Domain gpt-4.10.92| 0.880.99| 1.000.94| 0.830.51| 0.240.92| 0.96 gpt-4o0.87| 0.751.00| 0.990.95| 0.830.54| 0.25 1.00| 0.96 gpt-4o-mini0.81| 0.761.00| 0.980.95| 0.820.54| 0.25 1.00| 0.88 mistral-nemo0.57| 0.580.99| 0.970.86| 0.790.29| 0.230.88| 0.84 mistral-large0.80| 0.811.00| 1.000.94| 0.830.49| 0.230.96| 1.00 In Table 15, we further evaluate our framework on the retail domain.In contrast to the airline domain, retail tasks appear more solvable, with most models achieving near-perfect 28 Accepted as a conference paper at ICLR 2026 MaxProgressRate@k. However, the trend between the personas remains consistent, with expert personas consistently having higher MaxAUC@k scores than the non-expert personas. An interesting observation appears for agents with gpt-4o and gpt-4o-mini models in the non-expert user setting. While both models exhibit nearly identical MeanProgressRate@k, MaxProgressRate@k, MaxAUC@k and MaxPPT@k scores, the agent with gpt-4o-mini shows a lower pass@k score compared to gpt-4o. This suggests that gpt-4o-mini frequently reaches a state of near-completion (i.e., MaxProgressRate@k close to 1), but often fails to complete the full task as compared to gpt-4o that has a higher pass@k score. This further strengthened our justification for the need of fine-grained metrics to quantify the agent’s progression. 0.00.20.40.60.81.0 MaxAUC@k 0.0 0.2 0.4 0.6 0.8 1.0 MaxProgressRate@ k MaxAUC@k vs MaxProgressRate@k 0.00.20.40.60.81.0 MaxPPT@k 0.0 0.2 0.4 0.6 0.8 1.0 MaxPPT@k vs MaxProgressRate@k Figure 6: Scatter plot results on the τ 2 -bench airline domain (easy + hard) illustrating the relation- ship between MaxAUC@k and MaxProgressRate@k (left subplot), and between MaxPPT@k and MaxProgressRate@k (right subplot) using the setting n = k = 20 trials. 29 Accepted as a conference paper at ICLR 2026 gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.900.900.80 MaxProgressRate@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.990.810.960.860.850.730.970.880.960.790.970.67 MaxAUC@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo Model 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.800.500.700.53 0.60 0.37 0.670.54 0.60 0.42 0.670.48 MaxPPT@k Expert Non-Expert Figure 7: Dataset level performance of different agent models on the τ 2 -bench airline do- main (easy), with error bars representing 95% confidence intervals. The top graph shows the MaxProgressRate@k, middle graph shows MaxAUC@k, bottom graph shows MaxPPT@k. 30 Accepted as a conference paper at ICLR 2026 gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.980.970.990.990.950.930.970.910.940.950.920.96 MaxProgressRate@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.960.920.980.960.940.900.95 0.84 0.930.900.88 0.87 MaxAUC@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo Model 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.84 0.73 0.94 0.81 0.89 0.77 0.83 0.66 0.87 0.75 0.76 0.65 MaxPPT@k Expert Non-Expert Figure 8: Dataset level performance of different agent models on the ToolSandbox dataset, with error bars representing 95% confidence intervals. The top graph shows the MaxProgressRate@k, middle graph shows MaxAUC@k, bottom graph shows MaxPPT@k. 31 Accepted as a conference paper at ICLR 2026 gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.940.960.910.880.860.880.960.970.940.930.96 0.71 MaxProgressRate@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.85 0.68 0.83 0.65 0.79 0.66 0.89 0.77 0.87 0.68 0.87 0.56 MaxAUC@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minigpt-5mistral-largemistral-nemo Model 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.44 0.25 0.43 0.23 0.43 0.26 0.47 0.29 0.43 0.24 0.44 0.25 MaxPPT@k Expert Non-Expert Figure 9: Dataset level performance of different agent models on the τ 2 -bench airline domain (easy+hard), with error bars representing 95% confidence intervals. The top graph shows the MaxProgressRate@k, middle graph shows MaxAUC@k, bottom graph shows MaxPPT@k. 32 Accepted as a conference paper at ICLR 2026 gpt-4.1gpt-4ogpt-4o-minimistral-large-2411mistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.991.000.991.000.990.980.97 MaxProgressRate@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minimistral-large-2411mistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.940.950.950.94 0.86 0.83 0.83 0.82 0.83 0.79 MaxAUC@k Expert Non-Expert gpt-4.1gpt-4ogpt-4o-minimistral-large-2411mistral-nemo 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Score 0.51 0.54 0.54 0.49 0.29 0.24 0.25 0.25 0.23 0.23 MaxPPT@k Expert Non-Expert Figure 10: Dataset level performance of different agent models on the τ 2 -bench retail domain, with error bars representing 95% confidence intervals. The top graph shows the MaxProgressRate@k, middle graph shows MaxAUC@k, bottom graph shows MaxPPT@k. 33 Accepted as a conference paper at ICLR 2026 A.11ABLATION STUDY ON THE DIFFERENT USER MODELS Besides varying the model for agents, we also conducted an ablation study by varying the user models across the gpt family as shown in Table 16. We observe a similar trend emerges as in earlier analyses: the gap across traditional metrics becomes narrower, and the agent performance difference between expert and non-expert user personas is relatively small. This further highlights the impor- tance of our proposed metrics in capturing agent behavior with respect to turns which is beyond what conventional metrics do. As expected, the agent interacting with expert user achieves consis- tently higher performance on MaxAUC@k and MaxPPT@k metrics, confirming our hypothesis that agent can resolve tasks more efficiently as expert users tend to understand the system well and provide complete information for the agent. Interestingly, if we use stronger models such as gpt-5 for the user proxy, we see a smaller gap between expert and non-expert personas. This suggests that as the model capability of the user proxy improves, the model proxy with a non-expert persona behaves more like an expert, and achieves performance closer to expert-level outcomes. Based on the current observation, we believe that varying the user model also changes the user expertise level, which potentially simulates different user expertise levels. Table 16: Overall performance of a gpt-4.1 agent with different user proxy models on the τ 2 -bench airline domain, using gpt-4.1 model as LLM-as-a-judge. Dataset contains easy and hard samples. Results are displayed with scores for Expert Persona | Non-expert Persona. For metrics with @k, the number of trials is n = k = 20. User Model MeanProg@k MaxProg@k MaxAUC@k MaxPPT@k pass@k τ 2 -bench Airline Domain (Easy + Hard) gpt-4.10.75| 0.670.94| 0.960.85| 0.680.44| 0.250.81| 0.86 gpt-4o0.72| 0.610.95| 0.950.82| 0.660.36| 0.220.81| 0.81 gpt-4o-mini0.73| 0.640.95| 0.950.80| 0.650.35| 0.22 0.86| 0.86 gpt-50.71| 0.730.92| 0.950.85| 0.830.46| 0.370.76| 0.86 34 Accepted as a conference paper at ICLR 2026 A.12ADDITIONAL INFORMATION ON DATASET AND EXPERIMENTAL SETUP τ 2 -bench. For τ 2 -bench, we split the samples in the airline domain into “easy” and “hard” subsets using the passˆk metric, with n = k = 4. We consider samples that are always completed in all 4 independent runs as “easy”. The original samples are annotated with tool signatures and natural language assertions. Since these assertions closely align with our grading notes, we use them for evaluation. ToolSandbox. For ToolSandbox, we do not split the samples as we consider them to be easy samples. We use gpt-4.1 to convert the milestones into grading notes. Each data sample or scenario consists of a set of milestone M = m 1 ,m 2 ,...,m n , and may include a directed acyclic graph (DAG) of dependencies E = (i,j), where each edge (i,j) indicates that milestone m j depends on m i . The conversion process extracts key information from each milestone, such as required tool calls, expected agent-to-user communications, and ground truth state changes, and assembles this with a DAG structure into a structured prompt. The prompt template is used to produce actionable grading notes, and expresses dependencies using connectors like “before” or “after”. ToolSandbox dataset contains multiple variations of the same scenario - for example, the base sce- nario find daystillholiday has variants like finddaystillholidayalt (which starts with an alternate input message) and find daystillholidaymultipleuserturn (which intentionally provides less in- formation to force a multi-turn conversation). These variations serve as a crude simulation of user expertise, and their grading notes do not differ significantly from the base scenario. Since we have our own generic user persona templates and these variants share similar milestones, we only use the base scenario (eg., finddaystillholiday) and ignore the variants. After this process, we manually reviewed and refined the dataset to ensure that the generated grading notes were correct and mean- ingful. Our setup offers greater variability than the original versions with fixed initial messages. The prompt template used for each scenario is as follows: 35 Accepted as a conference paper at ICLR 2026 Prompt Template for Creating Grading Notes section 4: You are creating grading notes for agent evaluation that include dependency relationships. Convert these milestones into concise statements about what the agent should accomplish, including any sequence requirements. SCENARIO:scenario name DESCRIPTION:scenariodescription TOTAL MILESTONES:total milestones MILESTONE DEPENDENCIES (DAG edges):milestoneedgelist DEPENDENCY ANALYSIS:human readable description of which milestone dependency MILESTONES: Milestone 0:details including constraint type Milestone 1:details including constraint type ... RULES: 1. Create one or MORE natural language subgoals per milestones as needed to capture all required actions 2. Mention specific tool names when relevant: ”Agent should call tool name” 3. Use natural language to describe the purpose: ”Agent should call searchcontacts to find Homer’s information” 4. Include sequence requirements when dependencies exist: ”before”, ”after”, ”then”, ”first” 5. Break down complex milestones into multiple subgoals if needed 6. Use format: ”Agent should [natural action description]” 7. Focus on what needs to be accomplished, be specific and actionable EXAMPLES OF GOOD NATURAL LANGUAGE GRADING NOTES: • ”Agent should call set wifistatus to turn off wifi” • ”Agent should inform the user that wifi is turned off” • ”Agent should enable cellular service before sending message” • ”Agent should enable cellular service before sending message” • ”Agent should update contact phone number after finding the contact” SPECIAL HANDLING FOR COMMUNICATION MILESTONES: • If target data has sender=AGENT and recipient=USER with content, the grading note should be: ”Agent should inform/tell the user [content]” • If targetdata has sender=EXECUTIONENVIRONMENT and recipient=AGENT with tooltrace, focus on the tool call requirement • Focus on what the agent needs to DO or COMMUNICATE, not technical database states CONSTRAINT TYPES: • snapshotsimilarity: Agent should achieve the target state • additionsimilarity: Agent should add/create the target data • removal similarity: Agent should remove/delete the target data • updatesimilarity: Agent should modify/update the target data RESPONSE FORMAT: Return a JSON array where each element can be either a single string or an array of strings for that milestone:jsonschema Each milestone can have one or multiple grading notes as subgoals. Include dependency relationships when they exist. 36 Accepted as a conference paper at ICLR 2026 Additionally, we provide examples of the generated grading notes for ToolSandbox Lu et al. (2024) below. The full dataset is released together with our code. Sample: modifycontactwithmessagerecency • Agent should call getcurrenttimestamp to retrieve the current time • Agent should call search contacts to find the contact information • Agent should call searchmessages after getting the current timestamp to find the last person the user sent a message to • Agent should update the contact’s phone number to +10293847563 after iden- tifying the person is Homer S. • Agent should inform the user: ’The phone number of the person you last talked to has been updated to +10293847563’ after updating the contact Figure 11:Example of generated grading notes for the ToolSandbox sample mod- ifycontactwithmessagerecency. Sample: updatecontactrelationshipwithrelationshiptwicemultipleuserturn • Agent should call search contacts to find contacts with the relationship ’friend’. • Agent should call modify contact to update Fredrik Thordendal’s rela- tionship to ’enemy’ after finding the contact. • Agent should call modifycontact to update John Petrucci’s relationship to ’enemy’ after finding the contact. • Agent should inform the user: ’Fredrik Thordendal and John Petrucci are now your enemies. • Agent should again call modify contact to update Fredrik Thordendal’s and John Petrucci relationship from ’enemy’ to ’friend’ again. Figure 12:Example of generated grading notes for the ToolSandbox sample up- datecontactrelationshipwithrelationshiptwicemultipleuserturn. Sample: findcurrentcitylowbatterymode • Agent should ensure low battery mode is disabled • Agent should enable WiFi • Agent should enable WiFi after ensuring low battery mode is disabled • Agent should enable location services • Agent should enable location services after ensuring low battery mode is dis- abled • Agent should call get currentlocation to retrieve the user’s location • Agent should inform the user: You are currently in Cupertino Figure13:ExampleofgeneratedgradingnotesfortheToolSandboxsample findcurrentcitylowbatterymode. 37 Accepted as a conference paper at ICLR 2026 Sample: convertcurrency • Agent should call convertcurrency with the arguments: amount=2048, fromcurrencycode=’USD’, tocurrencycode=’CNY’ Figure 14: Example of generated grading notes for the ToolSandbox sample convertcurrency. Sample: searchmessagewithrecencyoldest • Agent should call getcurrenttimestamp to retrieve the current times- tamp • Agent should call searchmessages to find the oldest message • Agent should call searchmessages after getting the current timestamp • Agent should inform the user: Your oldest message says ’Hey kid, you want some GPU?’. Figure15:ExampleofgeneratedgradingnotesfortheToolSandboxsample searchmessagewithrecencyoldest. Sample: searchreminderwithrecencyyesterday • Agent should call getcurrenttimestamp to retrieve the current date and time • Agent should call searchreminder to find reminders for the day before the retrieved timestamp • Agent should call searchreminder after getting the current timestamp • Agent should inform the user: Your reminder from yesterday says ’Look for Company SF tickets’. Figure16:ExampleofgeneratedgradingnotesfortheToolSandboxsample searchreminderwithrecencyyesterday. 38 Accepted as a conference paper at ICLR 2026 Sample: searchreminderwithrecencyyesterday Answer AI assistant’s questions given the following task you (LLM-simulated user) want AI assistant to complete: Ask AI assistant to find the content of your (LLM- simulated user’s) reminder from yesterday. It should say Look for Company SF tickets. Do not leak this information. You do not have any more information. When AI assistant completed the task, even if you (LLM-simulated user) don’t have enough information to validate the correctness, break out of the simulation and end the conversation using the termination message. Start your first message as: ”What’s on my reminder yesterday?” Sample: send messagewithphonenumberandcontent Answer AI assistant’s questions given the following task you (LLM-simulated user) want AI assistant to complete: Send a message to +12453344098 saying: How’s the new album coming along. You do not have more information. When AI assistant completed the task, even if you (LLM-simulated user) don’t have enough information to validate the correctness, break out of the simulation and end the conversation using the termination message. Figure 17: Example of user task instructions for the ToolSandbox samples Sample: 1 Domain: airline Reason for Call: You had a mixup with your assistant and booked multiple flights for the same day. Known Information: You are Sophia Martin. Your user id is sophiamartin4574. Task Instructions: You want to first check if there are cases like this in your profile. You want the agent to fix the situation for you. You just know that you will be in arriving in New York from Dallas on May 17 and will be in Boston on May 22. You want to let the agent figure out which flights should be cancelled. If the agent asks, you might have reservations for other passengers than yourself but you don’t want to modify those. The task is considered complete if the instruction goal is satisfied or you are transferred to another agent or you find yourself in a situation in which the scenario does not provide enough information for you to continue the conversation. Sample: 2 Domain: airline Reason for Call: You just faced some money issue and want to downgrade all business flights to economy, without changing the flights or passengers. Known Information: Your name is Omar Davis. Your user id is omar davis3817. Task Instructions: You are fine with refunding to original payment for each reservation. You want to know how much money you have saved in total. You are emotional and a bit angry, but you are willing to cooperate with the agent. The task is considered complete if the instruction goal is satisfied or you are transferred to another agent or you find yourself in a situation in which the scenario does not provide enough information for you to continue the conversation. Figure 18: Example of user task instructions for the τ 2 -bench airline samples 39 Accepted as a conference paper at ICLR 2026 A.13ADDITIONAL DETAILS ON AGENT TRAJECTORIES GPT-5 Dialogue Excerpt: UserProxy: Hey, um, can you check what my reminder is for today? I’m not sure how to see it. Tau2Agent: What time zone are you in? Do you want all reminders scheduled for today, or just the next upcoming one? UserProxy: Uh, I’m in Eastern time, I think. Just, like, whatever’s next for today is fine. Tau2Agent: Got it—I'l look for the next reminder scheduled for today in Eastern time (currently UTC-4). To pull your reminders, I need the identifier for your reminders profile. ︙ UserProxy: Uh, I don’t really remember what email I used, sorry. I think the reminder is about buying something? Not sure what time though. Does that help? Tau2Agent : Thanks—that helps. I searched for reminders in Eastern time, and found this next one: “Buy a nice rich navy bathing dress”... Task Instruction ...Ask AI assistant to find the content of your (LLM-simulated user's) upcoming reminder today. It should say Buy a nice rich navy bathing dress. IMPORTANT: Do NOT leak this information... Mistral-NemoDialogue Excerpt: UserProxy: Hey, um, can you check what my reminder is for today? I think I set something but I’m not totally sure. Tau2Agent: I couldn’t find any reminders that you have set for today. If you still believe you have a reminder, it is possible that it was set for yesterday or tomorrow. UserProxy: Uh, yeah, maybe check for tomorrow then? I’m not really sure what day it was set for. Tau2Agent: I couldn’t find any reminders that you have set for tomorrow. If you still believe you have a reminder, it is possible that it was set for yesterday or in the future. ︙ UserProxy: Hmm, I guess just check for any reminders in the future then? Maybe I set it for a random day. Tau2Agent : I found a reminder that is set to go off in the future. Here are the details: Reminder ID: 54f20be9-eac3-5e76-8e57-724e48d3d477 Content: Buy a nice rich navy bathing dress... No Tool Calls Made 3ToolCallsMade Grading Notes Agent should call get_current_timestamp to retrieve the current time Agent should call search_reminder to find upcoming reminders Agent should call search_reminder after retrieving the current timestamp Agent should inform the user: 'Buy a nice rich navy bathing dress Figure 19: Agent dialogue excerpt for gpt-5 and Mistral-Nemo, illustrating how Mistral-Nemo makes tool calls early and quickly satisfies the grading notes, while gpt-5 asks clarifying questions and progresses more gradually. Despite these differing strategies, both agents complete the task in the same number of turns. 40

Instant research discovery

Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis

Intelligence

Summary

Entities (5)

Relation Signals (4)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations