Paper deep dive

Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints

Yutao Xie, Jingbo Yang, Guanyu Yao, Xiaoyu Yang, Duo Ding, Shiyu Chang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 144

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlooking failures that arise under diverse but realistic user interactions. This can lead to overly optimistic estimates of agent robustness and provide limited failure signals for training agents that generalize across users. Existing approaches largely rely on predefined user behaviors, which may miss agent- or domain-specific weaknesses, while failure-oriented search can reduce task success by invalidating the underlying task rather than exposing genuine robustness failures. To fill this gap, we introduce TIPER, a task-integrity-preserving persona evolution method that automatically discovers agent-specific user policies exposing robustness weaknesses while keeping the underlying task valid and solvable. TIPER evolves persistent user personas through reflection-guided behavioral mutations and task-wise Pareto selection under a task-integrity constraint. Across four interactive-agent benchmarks, TIPER yields lower held-out avg@3 scores in all 12 GPT-5.4 settings, with observed reductions of 5.8 to 33.7 percentage points, and in all 28 agent-domain pairs spanning seven agents, with a median observed reduction of 22.9 points. Further analysis yields a behavior-to-failure taxonomy that connects discovered user behaviors to recurring agent failure modes. Together, these results show that TIPER enables more diagnostic robustness evaluation of interactive agents and provides informative failure analysis for future agent robustness improvement. Code repository: this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

140,407 characters extracted from source content.

Expand or collapse full text

Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints Yutao Xie Affiliation: University of California, Santa Barbara Jingbo Yang Affiliation: University of California, Santa Barbara Guanyu Yao Affiliation: University of California, Santa Barbara Xiaoyu Yang Affiliation: Cresta Duo Ding Affiliation: Cresta Shiyu Chang Affiliation: University of California, Santa Barbara Abstract Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlooking failures that arise under diverse but realistic user interactions. This can lead to overly optimistic estimates of agent robustness and provide limited failure signals for training agents that generalize across users. Existing approaches largely rely on predefined user behaviors, which may miss agent- or domain-specific weaknesses, while failure-oriented search can reduce task success by invalidating the underlying task rather than exposing genuine robustness failures. To fill this gap, we introduce Tiper, a task-integrity-preserving persona evolution method that automatically discovers agent-specific user policies exposing robustness weaknesses while keeping the underlying task valid and solvable. Tiper evolves persistent user personas through reflection-guided behavioral mutations and task-wise Pareto selection under a task-integrity constraint. Across four interactive-agent benchmarks, Tiper yields lower held-out avg@3 scores in all 12 GPT-5.4 settings, with observed reductions of 5.8–33.7 percentage points, and in all 28 agent–domain pairs spanning seven agents, with a median observed reduction of 22.9 points. Further analysis yields a behavior-to-failure taxonomy that connects discovered user behaviors to recurring agent failure modes. Together, these results show that Tiper enables more diagnostic robustness evaluation of interactive agents and provides informative failure analysis for future agent robustness improvement. Code repository: https://github.com/UCSB-NLP-Chang/TIPER. †footnotetext: Corresponding author(s): yutaoxie@ucsb.edu 1 Introduction Recent advances in reasoning and tool use have made LLM agents increasingly capable of completing multi-turn collaborative tasks that require both user interaction and external actions (Yao et al., 2024; Barres et al., 2025; He et al., 2025). However, existing evaluations of collaborative agents often oversimplify user dynamics by relying on LLM-based user simulators that are primarily instructed to facilitate task completion. Real users, in contrast, vary substantially in how they communicate, provide information, respond to agent requests, and cooperate toward a shared goal. Without explicit behavioral specifications or training, LLM simulators tend to default to broadly helpful and cooperative interaction patterns, failing to capture this diversity (Naous et al., 2026). As a result, current evaluation protocols may systematically underrepresent challenging yet realistic interaction modes and consequently overestimate agent performance. The same bias can also affect training: agents optimized against a narrow set of simulator behaviors may overfit to these interaction patterns and generalize poorly to diverse users (Yu et al., 2026). These limitations motivate the systematic identification of user policies that reveal weaknesses in agent robustness, enabling more precise evaluation and providing informative interaction patterns for subsequent agent training. Recent work has begun to evaluate interactive agents with non-collaborative or behaviorally diverse users. NCUser (Shim et al., 2026) introduces predefined classes of non-collaborative behavior, while τ-Trait (He et al., 2026) enables controllable variation along specified user traits. However, two important limitations remain. First, challenging user behaviors are largely specified a priori through hand-crafted policy classes or researcher-selected traits, which may overlook behaviors that reveal weaknesses specific to a particular agent or domain. AgentNoiseBench (Wang et al., 2026) moves toward agent-targeted evaluation by searching for behaviors that degrade agent performance, but operates over message-level perturbations from a fixed taxonomy rather than persistent user policies that shape the interaction over multiple turns. Second, reduced task success does not necessarily indicate an agent weakness. Challenging user behaviors can instead make the underlying task infeasible, confounding agent robustness with task validity. This concern is nontrivial in existing benchmarks: under the τ2τ^2-Bench audit standard (Barres et al., 2025), our audit of 872 τ-Trait conversations finds task-critical user-simulator errors in 244 cases (28.0%) (Appendix D). We therefore require discovered user policies to preserve the validity and feasibility of the underlying task throughout the realized interaction, a constraint we term task integrity. Together, these limitations call for user-policy discovery that is both agent-targeted and task-integrity preserving. Figure 1: Predefined user-behavior controls versus task-integrity-constrained persona evolution. We introduce Tiper, a task-integrity-preserving persona evolution method for automatically identifying user policies that expose weaknesses of a given agent. Tiper iteratively evolves persistent user personas by reflecting on observed interaction failures and proposing behavior-level mutations likely to reveal additional weaknesses. Task-wise Pareto selection preserves complementary failure patterns across tasks, while benchmark-specific validity controls and trajectory audits distinguish agent failures from failures caused by task invalidation. In this way, Tiper searches directly for agent-specific robustness weaknesses while keeping the resulting interactions valid. Across 12 settings spanning four interactive-agent benchmarks, we observe lower held-out GPT-5.4 task success under Tiper-discovered personas in every setting, with mean differences of 5.8–33.7 percentage points and a median difference of 16.1 points. This pattern extends across model families: across 28 agent–domain pairs covering seven agents, we observe lower task success in every pair, with a median reduction of 22.9 percentage points. In the primary GPT-5.4 evaluation, the pooled task-critical simulator-error rate among exposed failed rollouts is below 2.5%2.5\%, indicating that the discovered failures reflect genuine robustness weaknesses that are missed under standard user simulation. The discovered personas also exhibit stronger behavioral realism than benchmark-default users under human-referenced User-Sim measurements, despite realism not being explicitly optimized during search. Beyond performance degradation, we analyze failures across eight domains to characterize how different user behaviors expose distinct agent failure modes, yielding a behavior-to-failure taxonomy for more diagnostic evaluation and future agent training. Together, these results show that Tiper can automatically uncover robustness weaknesses that current collaborative agent evaluation systematically overlooks. 2 Problem Formulation Let =x1,…,xNX=\x_1,…,x_N\ be a collection of interactive tasks, where xi=(gi,ℰi,Vi,Ii)x_i=(g_i,E_i,V_i,I_i) specifies a user goal, an environment, a task verifier, and an integrity judge. We fix an evaluated agent π and a user-simulator backbone μ. A persona p∈p is an instance-agnostic behavioral instruction that induces the user policy μp:=μ(⋅∣p) _p:=μ(· p). For persona p on task xix_i, let ξip _i^p denote the interaction trajectory induced by π and μp _p. The verifier returns Vi(ξip)∈[0,1]V_i( _i^p)∈[0,1], where 11 denotes full task success, and the integrity judge returns Ii(ξip)∈0,1I_i( _i^p)∈\0,1\, where 11 indicates that the simulated user preserved the assigned goal, task facts, and environmental constraints. We define ρi(p):=Pr[Ii(ξip)=1],ρ(p):=1N∑i=1Nρi(p),Ji(p):=[Vi(ξip)∣Ii(ξip)=1]. _i(p):= [I_i( _i^p)=1], ρ(p):= 1N _i=1^N _i(p), J_i(p):=E\! [V_i( _i^p) I_i( _i^p)=1 ]. (1) Here, ρ⁡(p)ρ(p) represents the overall integrity rate of persona p across the evaluation set. A persona is valid on the evaluation set if ρ⁡(p)≥1−ερ(p)≥ 1- , for a fixed tolerance ε∈[0,1) ∈[0,1). Thus, validity is determined by the aggregate proportion of valid trajectories across tasks. Conditioning on Ii=1I_i=1 restricts Ji(p)J_i(p) to trajectories judged to preserve task integrity. Let p0p_0 denote the benchmark-default persona. For a persona p, define its exposed performance reduction on task xix_i as Δi(p):=Ji(p0)−Ji(p). _i(p):=J_i(p_0)-J_i(p). Positive values indicate that the persona lowers agent performance, while negative values indicate that it improves performance.11 1 We assume ρi(p)>0 _i(p)>0 for every task i, so that the conditional scores Ji(p)J_i(p) are well-defined. We seek a persona that maximizes the average performance reduction while satisfying the aggregate integrity constraint: maxp∈⁡R⁡(p),R⁡(p):=1N∑i=1NΔi(p),subject to ρ(p)≥1−ε. _p R(p), R(p):= 1N _i=1^N _i(p), to ρ(p)≥ 1- . (2) Maximizing R⁡(p)R(p) is therefore equivalent to minimizing the persona’s mean task performance. Neither ρi(p) _i(p) nor Ji(p)J_i(p) is available in closed form, so evaluating a persona requires rolling out π against the induced user policy μp _p. We therefore compare personas under a fixed evaluation budget. For Tiper, this budget is determined by the generation limit G, the proposal-attempt limit M, the evaluated task sets, and the number of trials per task. 3 Task-Integrity-Preserving Persona Evolution To approximate this constrained objective, we perform G generations of evolutionary search over the natural-language space P. We initialize 0:=p0A_0:=\p_0\. As illustrated in Figure 2, generation g∈0,…,G−1g∈\0,…,G-1\ samples a parent persona pparp_par from the active set gA_g, collects trajectories on a minibatch BgB_g, and reflects on them to propose a child persona. The child undergoes text-level checks and the search-time integrity gate before task-wise Pareto selection updates the active set. This section describes the serial search used for τ3τ^3-bench, Lost-in-Conversation, ColBench, and VitaBench. More details of experiments are described in Appendix B. To separate candidate generation, model selection, and final evaluation, we partition the tasks into three disjoint subsets. The search set searchX_search is used to generate candidate personas and to perform the initial parent–child comparisons in each generation. The rank set rankX_rank serves as a validation set for the trajectory-level integrity gate, task-wise Pareto selection, and final persona selection. The test set testX_test is held out throughout search and selection and is used only for the final evaluation. Before search, we evaluate p0p_0 on rankX_rank and cache its task-wise scores J~i(p0) J_i(p_0). 3.1 Persona Representation and Reflective Proposal Persona representation and proposal. A persona p∈p is a domain-aware but instance-agnostic behavioral instruction. It specifies how the simulated user pursues the assigned goal without referring to task-specific identifiers, dates, or instances, allowing the same persona to transfer to unseen tasks in the domain. Inspired by reflective prompt optimization (Agrawal et al., 2026), the proposal model reviews a parent persona and its minibatch trajectories, identifies a behavior that may expose an observed agent weakness, and rewrites the persona along that direction. Text-level checks. Before rollout, two filters screen each proposal. The first rejects personas that are unclear, instance-specific, or inconsistent with the assigned task. The second rejects near-duplicates using lexical and embedding similarity against the proposal history. Together, these filters prevent invalid or redundant candidates from consuming rollout budget and keep the search focused on transferable behavioral changes. Proposals that fail either check are revised within a retry loop of at most M attempts, whereas those that pass enter the search-time integrity gate. Both text-check feedback and gate-rejection reasons are supplied to the proposal model, allowing subsequent attempts to address the identified issue while preserving the intended mutation direction. Figure 2: Overview of the task-integrity-preserving persona evolution procedure. Each generation reflects and mutates a parent using search trajectories. The child undergoes text-level checks and the rollout-based search-time integrity gate before task-wise Pareto selection updates g+1A_g+1. 3.2 Search-Time Integrity Gating For each generation, we first compare the child persona with its parent on the search minibatch BgB_g. Let J~i(p) J_i(p) denote the performance of p on task xix_i, and define J~Bg(p):=1|Bg|∑xi∈BgJ~i(p) J_B_g(p):= 1|B_g| _x_i∈ B_g J_i(p). A child that yields a lower minibatch score than its parent is then evaluated on rankX_rank. Let I~i(ξ)∈0,1 I_i(ξ)∈\0,1\ indicate whether rollout ξ on task xix_i passes the search-time integrity check. We define ρ~fail(p) ρ_fail(p) as the fraction of failure rollouts on rankX_rank that pass this integrity check. A child p~ p passes the search-time integrity gate if J~Bg(p~)<J~Bg(ppar) J_B_g( p)< J_B_g(p_par) and ρ~fail(p~)≥1−γ ρ_fail( p)≥ 1-γ, where γ∈[0,1]γ∈[0,1] is the maximum allowed integrity-violation rate among verifier-failed rollouts. The first condition favors children that expose more agent failures, while the second prevents the search from favoring failures caused by violations of the simulated user’s behavior. This search-time gate is an empirical filter and does not certify the formal validity constraint defined by ρ⁡(p)ρ(p). Rejected children leave the active set unchanged, while gate-passing children proceed to task-wise Pareto selection. 3.3 Task-Wise Pareto Selection For candidates that pass the integrity gate, we determine Pareto dominance from their task-wise rank-set scores. Since a lower score indicates stronger failure exposure, persona p dominates persona q if J~i(p)≤J~i(q) J_i(p)≤ J_i(q) for every xi∈rankx_i _rank, with strict inequality for at least one task. Let NDJ~⁡()ND_ J(S) denote the nondominated members of S under this relation. Let KeepK⁡(,K)KeepK(S,K) retain all members when ||≤K|S|≤ K and otherwise retain the K members with the lowest mean rank-set score. For a gate-passing child p~ p, we update g+1:=KeepK⁡(NDJ~⁡(g∪p~),K)A_g+1:=KeepK(ND_ J(A_g∪\ p\),K). If no proposal attempt passes the checks and gate, then g+1:=gA_g+1:=A_g. Task-wise selection retains personas that expose complementary weaknesses during evolution. Parents for the next generation are sampled from the updated active set. After G generations, we select one persona from GA_G using the rank set: p^:=arg⁡minp∈G1|rank|∑xi∈rankJ~i(p). p:= _p _G 1|X_rank| _x_i _rank J_i(p). (3) We fix p p before test evaluation and report all test metrics for this persona. 4 Experiments We evaluate Tiper through four research questions: • (Q1) Failure-revealing persona evolution. Can Tiper discover substantially more failures across held-out test tasks than the default and alternative policies? • (Q2) Robustness across agents and components. How does failure exposure with Tiper vary across evaluated agents, mutation models, and simulator backbones? • (Q3) Integrity and behavioral alignment. How well do the generated trajectories preserve task integrity and align with human-user references? • (Q4) Behavior-to-failure characterization. How do task-preserving user behaviors vary across benchmarks, and what agent failure modes are associated with them? 4.1 Experimental Setup Benchmarks and agents. We evaluate Tiper with GPT-5.4 (OpenAI, 2026a) on 12 settings drawn from four interactive-agent benchmarks: τ3τ^3-bench (Barres et al., 2025; Shi et al., 2026), Lost-in-Conversation (Laban et al., 2026), ColBench (Zhou et al., 2025), and VitaBench (He et al., 2025). To study generalization across agents, we rerun the persona search for six additional proprietary and open-weight agents: GLM-5.2 (Z.ai, 2026), Claude-Sonnet-5 (Anthropic, 2026), DS-V4-Flash (DeepSeek-AI et al., 2026), MiniMax-M3 (MiniMax, 2026), Qwen3.7-Plus (Alibaba Cloud Community, 2026), and MUSE-Glimmer (Meta Superintelligence Labs, 2026). This cross-agent study covers four representative domains: Telecom, Math, Coding, and Delivery. Unless otherwise specified, we use GPT-5.4 as the simulator backbone and GPT-5.5 as the mutation model. GPT-5.2 and GLM-5.2 serve as the judges. More implementation details are provided in Appendix B. Baselines. We compare Tiper with the benchmark-default user simulator, the fixed user programs from NCUser (Shim et al., 2026), and an objective-matched variant of PPol (Chopra et al., 2026). The PPol variant uses the same agent-difficulty objective, task partitions, and search budget as Tiper. All methods are evaluated on the same test tasks and undergo the identical trajectory-level integrity audit described in Appendix E. Evaluation protocol and metrics. Each Tiper candidate that reaches rank-set evaluation is run twice per task. We select the persona with the lowest mean rank-set score and fix it before test evaluation. We then conduct three independent test rollouts per task. After integrity filtering, we report valid avg@3, defined as the mean task score over the valid rollouts among the three test trials. We also report passkpass^k, defined among tasks with three valid rollouts as the fraction with at least k fully successful rollouts, where k∈1,2,3k∈\1,2,3\. For ColBench, valid avg@3 is the mean fraction of unit tests passed, and a rollout is fully successful only if it passes all unit tests. We define failure exposure as the reduction in task score from the default condition to the persona condition after integrity filtering. In the following experiments, every test task has at least one integrity-valid rollout in each reported condition. Policy comparisons additionally report the failure-conditioned task-critical error rate under an adapted τ2τ^2-Bench rubric (Barres et al., 2025). The User-Sim Index measures behavioral alignment between trajectories induced by Tiper and human-user reference trajectories (Zhou et al., 2026). Because integrity filtering can change the tasks eligible for passkpass^k, we interpret these metrics descriptively. Appendix B.4 additionally examines avg@3 sensitivity to common-valid-task filtering for the four fixed-policy comparisons on Airline. 4.2 Main Results Tiper consistently exposes additional agent failures on held-out tasks. As shown in Table 1, the selected personas reduce both avg@3 and pass3pass^3 in all 12 settings, with median reductions of 16.1 and 15.0 percentage points, respectively. The largest avg@3 reductions occur in Math, Telecom, and Delivery, at 33.7, 25.3, and 25.0 points, with corresponding pass3pass^3 reductions of 54.1, 35.2, and 22.5 points. Because persona selection does not use the test tasks, these results indicate that the discovered behaviors transfer to unseen tasks within each domain. Table 1: Held-out failure exposure across benchmarks for GPT-5.4. We compare the benchmark-default user simulator with the rank-selected Tiper persona on 12 test settings. All values are percentages. The orange numbers indicate the decrease from the corresponding default scores in percentage points. Differences are computed from the displayed rounded scores. Setting avg@3 pass1 pass2 pass3 Benchmark Domain default Tiper default Tiper default Tiper default Tiper Telecom 81.5 56.2↓25.3 94.4 77.8↓16.6 79.6 55.6↓24.0 70.4 35.2↓35.2 Banking 33.3 27.5↓5.8 45.6 43.9↓1.7 35.1 22.8↓12.3 19.3 15.8↓3.5 Retail 78.4 71.6↓6.8 92.6 83.3↓9.3 85.2 75.9↓9.3 57.4 55.6↓1.8 τ3τ^3-bench (Barres et al., 2025) Airline 86.7 76.7↓10.0 92.0 82.0↓10.0 90.0 78.0↓12.0 78.0 70.0↓8.0 Math 89.9 56.2↓33.7 97.7 82.5↓15.2 93.0 60.0↓33.0 79.1 25.0↓54.1 Database 65.2 50.4↓14.8 76.6 68.1↓8.5 68.1 46.8↓21.3 51.1 36.2↓14.9 Lost-in-Conv. (Laban et al., 2026) Actions 52.6 40.0↓12.6 55.6 51.1↓4.5 53.3 40.0↓13.3 48.9 28.9↓20.0 ColBench (Zhou et al., 2025) Coding 44.6 27.2↓17.4 32.0 12.0↓20.0 20.0 6.0↓14.0 14.0 2.0↓12.0 Delivery 66.7 41.7↓25.0 85.0 62.5↓22.5 72.5 42.5↓30.0 42.5 20.0↓22.5 In-store 76.7 66.7↓10.0 92.5 82.5↓10.0 77.5 72.5↓5.0 60.0 45.0↓15.0 OTA 51.7 34.2↓17.5 82.5 55.0↓27.5 45.0 27.5↓17.5 27.5 20.0↓7.5 VitaBench (He et al., 2025) Cross-domain 54.2 34.2↓20.0 77.5 62.5↓15.0 57.5 30.0↓27.5 27.5 10.0↓17.5 Cross-agent generalization. Tiper consistently exposes failures across different agent models. Across 28 combinations of seven agents and four domains in Table 2, avg@3 decreases in every setting and pass3pass^3 decreases in 25, with median reductions of 22.9 and 20.4 percentage points, respectively. The effect is particularly consistent in Telecom, where avg@3 decreases by 13.0–39.5 points and pass3pass^3 by 18.6–44.5 points across all seven agents. The magnitude varies more in Math, with avg@3 reductions ranging from 0.8 points for GLM-5.2 to 33.7 points for GPT-5.4. Baseline comparison. To assess the value of agent-conditioned persona search, we compare Tiper with fixed behavior programs and objective-matched PPol across four domains and two agents (Table 3). Across all eight agent–domain settings, Tiper produces the largest avg@3 reduction, exceeding the strongest alternative by a median of 7.5 percentage points. Overall, Tiper exposes more agent failures than either fixed behaviors or objective-matched PPol. Table 2: Cross-agent held-out failure exposure on four representative domains. For each agent and domain, we rerun the same Tiper search. All values are percentages. The orange numbers indicate the decrease from the corresponding default scores in percentage points. Differences are computed from the displayed rounded scores. Setting avg@3 pass1 pass2 pass3 Domain Agent model default Tiper default Tiper default Tiper default Tiper GPT-5.4 81.5 56.2↓25.3 94.4 77.8↓16.6 79.6 55.6↓24.0 70.4 35.2↓35.2 GLM-5.2 95.1 69.8↓25.3 100.0 83.3↓16.7 100.0 74.1↓25.9 85.2 51.9↓33.3 Claude-Sonnet-5 69.8 44.4↓25.4 83.3 51.9↓31.4 74.1 48.1↓26.0 51.9 33.3↓18.6 DS-V4-Flash 90.7 77.7↓13.0 98.1 94.4↓3.7 96.3 83.3↓13.0 77.8 55.6↓22.2 MiniMax-M3 90.1 53.1↓37.0 100.0 61.1↓38.9 94.4 55.6↓38.8 75.9 42.6↓33.3 Qwen3.7-Plus 94.4 54.9↓39.5 100.0 68.5↓31.5 98.1 55.6↓42.5 85.2 40.7↓44.5 Telecom (Barres et al., 2025) MUSE-Glimmer 80.9 49.4↓31.5 96.3 70.4↓25.9 79.6 53.7↓25.9 66.7 24.1↓42.6 GPT-5.4 44.6 27.2↓17.4 32.0 12.0↓20.0 20.0 6.0↓14.0 14.0 2.0↓12.0 GLM-5.2 70.2 51.6↓18.6 56.0 26.0↓30.0 34.0 20.0↓14.0 16.0 2.0↓14.0 Claude-Sonnet-5 77.8 50.0↓27.8 62.0 32.0↓30.0 38.0 12.0↓26.0 18.0 6.0↓12.0 DS-V4-Flash 67.6 40.0↓27.6 54.0 24.0↓30.0 28.0 10.0↓18.0 6.0 6.0→0.0 MiniMax-M3 76.6 72.8↓3.8 68.0 58.0↓10.0 30.0 18.0↓12.0 12.0 6.0↓6.0 Qwen3.7-Plus 45.8 38.2↓7.6 36.0 18.0↓18.0 24.0 12.0↓12.0 8.0 8.0→0.0 Coding (Zhou et al., 2025) MUSE-Glimmer 78.8 60.6↓18.2 68.0 42.0↓26.0 46.0 24.0↓22.0 24.0 16.0↓8.0 GPT-5.4 89.9 56.2↓33.7 97.7 82.5↓15.2 93.0 60.0↓33.0 79.1 25.0↓54.1 GLM-5.2 84.5 83.7↓0.8 95.3 95.3→0.0 86.0 83.7↓2.3 72.1 72.1→0.0 Claude-Sonnet-5 89.9 71.3↓18.6 95.3 95.3→0.0 93.0 69.8↓23.2 81.4 48.8↓32.6 DS-V4-Flash 82.2 74.4↓7.8 95.3 90.7↓4.6 86.0 76.7↓9.3 65.1 55.8↓9.3 MiniMax-M3 81.4 73.6↓7.8 95.3 93.0↓2.3 86.0 72.1↓13.9 62.8 55.8↓7.0 Qwen3.7-Plus 49.6 35.7↓13.9 79.1 65.1↓14.0 53.5 32.6↓20.9 16.3 9.3↓7.0 Math (Laban et al., 2026) MUSE-Glimmer 50.4 39.5↓10.9 79.1 62.8↓16.3 46.5 37.2↓9.3 25.6 18.6↓7.0 GPT-5.4 66.7 41.7↓25.0 85.0 62.5↓22.5 72.5 42.5↓30.0 42.5 20.0↓22.5 GLM-5.2 62.5 55.0↓7.5 82.5 75.0↓7.5 62.5 62.5→0.0 42.5 27.5↓15.0 Claude-Sonnet-5 64.2 39.2↓25.0 82.5 67.5↓15.0 72.5 35.0↓37.5 37.5 15.0↓22.5 DS-V4-Flash 68.3 46.7↓21.6 87.5 70.0↓17.5 70.0 47.5↓22.5 47.5 22.5↓25.0 MiniMax-M3 60.0 32.5↓27.5 82.5 62.5↓20.0 67.5 27.5↓40.0 30.0 7.5↓22.5 Qwen3.7-Plus 70.0 45.8↓24.2 87.5 75.0↓12.5 72.5 42.5↓30.0 50.0 20.0↓30.0 Delivery (He et al., 2025) MUSE-Glimmer 54.2 28.3↓25.9 77.5 55.0↓22.5 57.5 25.0↓32.5 27.5 5.0↓22.5 4.3 Integrity and Behavioral Alignment Integrity. Integrity is assessed at the trajectory level, and trajectories judged invalid are excluded before task scores are aggregated. Among failed rollouts on held-out tasks, Table 3 reports the fraction containing a task-critical user-simulator error. This rate is at most 2.5% for Tiper, compared with up to 21.4% for the alternative policies. Together, these results support interpreting Tiper’s larger score reductions as exposure of agent weaknesses instead of artifacts of task-integrity violations. A blinded human audit of 150 automatically accepted TIPER failure trajectories found 148 task-valid cases, of which 143 had an identifiable agent error (Appendix E.3). Behavioral alignment. The left panels of Figure 3 compare our user-policy trajectories with Sim2Real references along communication style, information disclosure, clarification, and reactions to agent errors. ECE measures differences in agent success across task-difficulty levels. We plot (1−ECE)×100(1-ECE)× 100, so farther outward indicates closer alignment, while USIUSI aggregates all five metrics. On Airline and Retail Tiper achieves higher aggregate USI scores than the benchmark-default simulator in both domains, while remaining below the human-reference scores (Table 5). Table 3: Policy comparison across four domains and two agents. Entries show changes in avg@3 from the default setting in percentage points (↓ lower, ↑ higher), followed in parentheses by the percentage of failed rollouts with task-critical simulator errors under the τ2τ^2-Bench rubric (Barres et al., 2025). Error rates above 5% are shown in red. Domain Telecom Airline Retail Delivery Policy GPT-5.4 DS-V4-Flash GPT-5.4 DS-V4-Flash GPT-5.4 DS-V4-Flash GPT-5.4 DS-V4-Flash Default setting — (0.0%) — (0.0%) — (0.0%) — (16.7%) — (3.1%) — (8.3%) — (0.0%) — (0.0%) Emotional ↓6.2 6.2 (2.5%) ↑0.6 0.6 (21.4%) ↓9.0 9.0 (10.0%) 0.0 (0.0%) 0.0 (2.9%) ↑0.6 0.6 (0.0%) ↓9.2 9.2 (2.0%) ↑3.3 3.3 (2.9%) Tangential ↓14.2 14.2 (1.9%) ↓10.2 10.2 (0.0%) ↓2.0 2.0 (0.0%) ↑6.7 6.7 (0.0%) ↑6.2 6.2 (4.0%) ↓0.6 0.6 (0.0%) ↓5.8 5.8 (4.3%) ↓3.3 3.3 (0.0%) Fragment ↓5.6 5.6 (0.0%) ↓3.1 3.1 (0.0%) ↓1.3 1.3 (0.0%) ↑3.3 3.3 (6.7%) ↑4.9 4.9 (3.7%) ↓1.2 1.2 (3.8%) ↓5.8 5.8 (2.1%) ↓7.5 7.5 (2.1%) Unavailable ↑2.5 2.5 (0.0%) ↑3.7 3.7 (0.0%) ↓5.3 5.3 (0.0%) ↑10.0 10.0 (0.0%) ↑4.3 4.3 (3.6%) 0.0 (4.2%) ↓9.2 9.2 (7.8%) ↓0.8 0.8 (2.6%) PPol ↓1.9 1.9 (14.3%) ↑4.3 4.3 (0.0%) ↓1.0 1.0 (7.7%) ↑1.0 1.0 (0.0%) ↓4.9 4.9 (2.3%) ↑0.6 0.6 (0.0%) ↓10.0 10.0 (0.0%) ↓13.3 13.3 (0.0%) Tiper ↓25.3 25.3 (0.0%) ↓13.0 13.0 (0.0%) ↓10.0 10.0 (0.0%) ↓6.7 6.7 (0.0%) ↓6.8 6.8 (1.6%) ↓9.9 9.9 (2.5%) ↓25.0 25.0 (0.0%) ↓21.6 21.6 (0.0%) Table 4: Search ablation and component sensitivity. Percentage-point changes from default; more negative values indicate stronger failure exposure. Panel (a) removes the integrity gate or post-proposal persona check; panels (b) and (c) vary the mutation model and simulator backbone. In panel (c), each simulator backbone is compared against its own default-persona baseline. (a) Search component ablation (Telecom) GPT-5.4 DS-V4-Flash Method avg@3 pass3 avg@3 pass3 Tiper −25.3-25.3 −35.2-35.2 −13.0-13.0 −22.2-22.2 w/o integrity gate −10.5-10.5 −14.8-14.8 −8.0-8.0 −14.1-14.1 w/o persona check −4.3-4.3 −8.1-8.1 −4.9-4.9 −8.5-8.5 (b) Mutation-model sensitivity (c) Simulator-backbone sensitivity Telecom Delivery Telecom Delivery Model avg@3 pass3 avg@3 pass3 Model avg@3 pass3 avg@3 pass3 GPT-5.5 −25.3-25.3 −35.2-35.2 −25.0-25.0 −22.5-22.5 GPT-5.4 −25.3-25.3 −35.2-35.2 −25.0-25.0 −22.5-22.5 GLM-5.2 −23.5-23.5 −29.6-29.6 −7.5-7.5 −7.5-7.5 DS-V4-Flash −4.3-4.3 −5.6-5.6 −12.5-12.5 −20.0-20.0 DS-V4-Flash −29.6-29.6 −33.3-33.3 −24.2-24.2 −35.0-35.0 Sonnet-5 −4.9-4.9 −11.1-11.1 −5.8-5.8 −17.5-17.5 5 Analysis 5.1 Ablation and Component Sensitivity Table 4 separates the effects of the search components from sensitivity to the models that propose and realize personas. More negative values indicate stronger failure exposure. Search components ablation. The full method produces the strongest failure exposure for both agents and both metrics, followed by the variant without the integrity gate and then the variant without the text-level persona check. Both gates were introduced to enforce validity, so removing them should, if anything, give the search more freedom to degrade the agent. Table 4(a) shows the opposite. Both checks contribute to held-out failure exposure. Removing either check reduces the score drop, suggesting that the constraints help searching for behaviors that transfer within the domain. One possible explanation is that the checks discourage task-invalidating or instance-specific shortcuts that do not transfer to held-out evaluation. Mutation model. With GPT-5.4 fixed as the simulator backbone, all three mutation models perform similarly on Telecom, reducing avg@3avg@3 by 23.5–29.6 points and pass3pass^3 by 29.6–35.2 points. Delivery is more sensitive: GPT-5.5 and DS-V4-Flash remain strong, whereas GLM-5.2 yields only a 7.5-point reduction on both metrics. No mutation model is best in every case, suggesting that proposal quality depends on the domain and metric rather than on a single universally stronger model. Simulator backbone. Proposal is one half of the loop; the other is the model that renders the persona in dialogue. Panel (c) takes the personas searched under GPT-5.4 and replays them under a different backbone, with the evaluated agent unchanged. Every backbone still degrades the agent, so the direction of the effect transfers while the magnitude does not. On Telecom the reduction nearly vanishes, with avg@3avg@3 going from −25.3-25.3 to −4.3-4.3 and −4.9-4.9 and pass3pass^3 from −35.2-35.2 to −5.6-5.6 and −11.1-11.1. On Delivery the two metrics diverge: avg@3avg@3 weakens from −25.0-25.0 to −5.8-5.8 and −12.5-12.5, while pass3pass^3 holds at −17.5-17.5 and −20.0-20.0 against −22.5-22.5. These differences may reflect how the backbones realize the same persona instructions, although persona fidelity is not measured directly here. 5.2 From User Behaviors to Agent Failure Modes To characterize the failures exposed by Tiper, we compare the integrity-preserving user behaviors observed across benchmarks and analyze their associations with recurring agent failure modes. Diagnostic setup. We construct 139 contrastive pairs across eight domains in two benchmarks for GPT-5.4. Each pair contains a successful default rollout and an integrity-valid failed Tiper rollout on the same task. Two judges independently assign multi-label user-behavior annotations, yielding 287 labels across the 139 trajectory pairs. The figures include only annotations on which both judges agree. Appendix F.4 provides the annotation protocol and filtering details. Benchmark-specific user behaviors. The right panels of Figure 3 summarize the relative frequencies of six displayed behavior categories, normalized within each benchmark. In τ3τ^3-bench, sparse information, incremental requests, and single-step requests account for 28.1%, 21.9%, and 20.8% of the displayed annotations. VitaBench is more concentrated: incremental requests account for 54.9% of the behavior annotations, followed by bundled actions at 24.6%. This difference is consistent with their interaction structures. The τ3τ^3-bench tasks are sensitive to how information is provided and maintained, whereas VitaBench more strongly tests whether an agent can accumulate requirements and execute multi-part workflows. Behavior–failure pathways. Figure 4 reveals a many-to-many relationship between user behaviors and agent failures. Incremental requests are prominently associated with incomplete option enumeration and assuming that a tool is unavailable, suggesting that the agent may stop before incorporating all information provided across turns. Sparse information and uncorrected assumptions frequently lead to trusting unverified user reports, indicating weak verification when important state remains uncertain. Other behaviors, including single-step requests, vague references, and bundled actions, spread across several failure modes rather than producing a single predictable error. These associations suggest three recurring weaknesses: incomplete tracking of evolving requests, insufficient verification of uncertain information, and premature commitment before all relevant options or constraints are resolved. Because each behavior can lead to multiple failure modes and each failure mode can arise from multiple behaviors, the figure should be interpreted as showing diagnostic associations rather than causal effects. Appendix C provides paired default–Tiper trajectories illustrating these mechanisms in τ3τ^3-bench and ColBench. 6 Related Work Interactive evaluation and user simulation. Static benchmarks do not fully capture interactive systems: stronger non-interactive performance does not always translate to better human–LM interaction, which also depends on the interaction process and user experience (Lee et al., 2024). Tool-agent benchmarks accordingly evaluate agents through multi-turn interactions with LLM-simulated users (Yao et al., 2024; Barres et al., 2025). However, such simulators can be overly cooperative and behaviorally uniform, misrepresenting performance with real users (Zhou et al., 2026; Seshadri et al., 2026). Unconstrained simulation also exhibits a formalism ceiling and amplifies hand-crafted directives (Zhu et al., 2026), making simulator validity essential for reliable evaluation. Figure 3: Behavioral alignment and discovered user behaviors. Left: User-Sim Index profiles on Airline and Retail; farther outward indicates closer Sim2Real alignment. ECE∗=(1−ECE)×100ECE^*=(1-ECE)× 100. Right: Normalized behavior shares among integrity-valid contrastive failures exposed by Tiper. Figure 4: Behavior-to-failure pathways for the contrastive pairs retained after judge-agreement and avoidability filtering. Ribbons connect domains, user behaviors, and agent failure modes. Persona-driven behavioral stress testing. Natural-language personas provide a flexible way to diversify simulated user behavior and can induce trait-consistent interactions (Jiang et al., 2024). Prior work models non-collaborative users or conditions simulators on human traits (Shim et al., 2026; He et al., 2026), while long-horizon studies show that agent performance depends on how user intent unfolds over time (Laban et al., 2026; Tack et al., 2026). Persona-conditioned simulators can nevertheless exhibit persona drift, role confusion, and behavioral echoing (Luo and Laban, 2026). Most closely related, PPol evolves persona generators to improve human-likeness and behavioral coverage while preserving task goals (Chopra et al., 2026). In contrast, Tiper performs agent-conditioned search for failure-revealing personas and verifies task integrity on the resulting trajectories. Reflective prompt evolution and adversarial search. Prompt optimization treats natural-language instructions as discrete search objects. ProTeGi derives textual gradients (Pryzant et al., 2023), EvoPrompt uses LLM-guided evolution (Guo et al., 2024), and GEPA combines trajectory reflection with Pareto selection (Agrawal et al., 2026). Related red-teaming methods generate difficult test cases (Perez et al., 2022) or use quality-diversity search to uncover varied failures (Samvelyan et al., 2024). Tiper adapts these ideas to interactive user simulation by evolving persona-conditioned policies, retaining complementary failures through task-wise Pareto selection, and filtering invalid candidates through text-level checks and trajectory-level integrity gating. 7 Conclusion We introduced Tiper, which evolves instance-agnostic personas to expose failures overlooked by benchmark-default user simulation while preserving task integrity. Across the 12 GPT-5.4 settings and 28 agent–domain pairs, Tiper yields lower observed avg@3avg@3 scores compared to default user policy, with a 22.9-point median reduction across agent–domain pairs. Its observed reduction is largest among the compared policies in all eight direct comparisons, where task-critical simulator-error rates are at most 2.5%. Ablations support the integrity gate and text checks, while trajectory analysis links exposed failures to execution, grounding, and replanning errors. Tiper thus provides a practical framework for targeted, behaviorally grounded robustness evaluation. Acknowledgement The UCSB team gratefully acknowledges support from the National Science Foundation (NSF) under Grant Nos. 2338252, 2302730, and 2619240. References Agrawal et al. (2026) L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab GEPA: reflective prompt evolution can outperform reinforcement learning. External Links: 2507.19457, Link Cited by: §3.1, §6. Alibaba Cloud Community (2026) Alibaba Cloud Community Qwen3.7-Plus: multimodal agent intelligence. External Links: Link Cited by: §4.1. Anthropic (2026) Anthropic Introducing Claude Sonnet 5. External Links: Link Cited by: §4.1. Barres et al. (2025) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan τ2τ^2-Bench: evaluating conversational agents in a dual-control environment. External Links: 2506.07982, Link Cited by: Appendix D, §E.1, §1, §1, §4.1, §4.1, Table 1, Table 2, Table 3, §6. Chao et al. (2024) P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong Jailbreaking black box large language models in twenty queries. External Links: 2310.08419, Link Cited by: §G.1. Chopra et al. (2026) H. Chopra, K. Ghate, A. Caliskan, T. Kohno, C. Shah, and N. Jaques Beyond cooperative simulators: generating realistic user personas for robust evaluation of llm agents. External Links: 2605.12894, Link Cited by: §4.1, §6. DeepSeek-AI et al. (2026) DeepSeek-AI, A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, C. Lu, C. Zhao, C. Deng, C. Hou, C. Xu, C. Shao, C. Ruan, C. Sun, D. Dai, D. Guo, D. Yang, D. Chen, D. Li, D. Ji, E. Li, F. Wei, F. Lin, F. Yuan, F. Xia, F. Dai, G. Hao, G. Chen, G. Cao, G. Meng, G. Li, H. Yu, H. Zhang, H. Xu, H. Li, H. Liang, H. Zhang, H. Luo, H. Wei, H. Yuan, H. Zhang, H. Luo, H. Chen, H. Ji, H. Zhang, H. Ding, H. Tang, H. Cao, H. Gao, H. Qu, H. Zeng, J. Yang, J. Zhu, J. Luo, J. Song, J. Yu, J. Huang, J. Cai, J. Liang, J. Zhou, J. Ye, J. Li, J. Xu, J. Hu, J. Yang, J. Chen, J. Yan, J. Chen, J. Zhou, J. Xiang, J. Yuan, J. Cheng, J. Zhou, J. Zhu, J. Yu, J. Sun, J. Ran, J. Jiang, J. Qiu, J. Li, J. Zheng, J. Song, K. Dong, K. Gao, K. Guan, K. Zhou, K. Huang, K. Yu, L. Wang, L. Zhang, L. Wang, L. Xia, L. Zhang, L. Zhao, L. Guo, L. Luo, L. Ma, L. Zhu, L. Wang, L. Cai, L. Zhang, L. Chen, M. Di, M. Xu, M. Mei, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, M. Zhou, M. Han, N. Wang, P. Huang, P. Wang, P. Cong, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Li, Q. Chen, Q. Du, Q. Jiang, R. Tian, R. Xu, R. Lu, R. Xu, R. Ge, R. Zhang, R. Pan, R. Wang, R. Chen, R. Yin, R. Xu, R. Shen, R. Zhang, R. Chen, S. Liu, S. Lu, S. Sun, S. Zhou, S. Chen, S. Cai, S. Nie, S. Wu, S. Chen, S. Hu, S. Liu, S. Hu, S. Ma, S. Wang, S. Yu, S. Zhou, S. Pan, S. Yu, S. Zhou, T. Ni, T. Yun, T. Jin, T. Pei, T. Ye, T. Lin, T. Ji, T. Cui, T. Yue, T. Yu, T. Wang, W. Zhang, W. Xiao, W. Zeng, W. An, W. Zhao, W. Liu, W. Liang, W. Pang, W. Luo, W. Yao, W. Gao, W. Yang, W. Huang, W. Hou, W. Zhang, W. Ma, X. Gao, X. He, X. Wang, X. Wang, X. Bi, X. Liu, X. Wang, X. Chen, X. Zhang, X. Nie, X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Liu, X. Yu, X. Li, X. Yang, X. Zhang, X. Chen, X. Wang, X. Su, X. Chen, X. Lin, X. Fu, Y. Yan, Y. Wang, Y. Ma, Y. Luo, Y. Zhang, Y. Xu, Y. Ma, Y. Huang, Y. Li, Y. Li, Y. Xu, Y. Zhao, Y. Sun, Y. Wang, Y. Qian, Y. Shao, Y. Yu, Y. Zhang, Y. Ding, Y. Shi, Y. Wu, Y. Xiong, Y. Ma, Y. He, Y. Tang, Y. Zhou, Y. Luo, Y. Zhong, Y. Piao, Y. Wang, Y. Zhang, Y. Chen, Y. Tan, Y. Wei, Y. Ma, Y. Liu, Y. Yang, Y. Guo, Y. Wu, Y. Wu, Y. Li, Y. Cheng, Y. Ou, Y. Xu, Y. Li, Y. Wang, Y. Yang, Y. Xu, Y. Wu, Y. Meng, Y. Zou, Y. Zha, Y. Xiong, Y. Chen, Y. Lin, Y. Cao, Y. Wang, Y. Zhang, Y. Yan, Y. Lin, Y. Gu, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. Zhou, Y. Huang, Z. Wu, Z. Wang, Z. Zhao, Z. Ren, Z. Zhang, Z. Sha, Z. Fu, Z. Ju, Z. Xu, Z. Xie, Z. Zhang, Z. Gao, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Huang, Z. Chen, Z. Wu, Z. Ren, Z. Wu, Z. Li, Z. Zhang, Z. Xu, Z. Wang, Z. Qu, Z. Gu, Z. Zhu, Z. Li, Z. Zhang, Z. Xie, Z. Gao, Z. Wan, Z. Pan, and Z. Yao DeepSeek-v4: towards highly efficient million-token context intelligence. External Links: 2606.19348, Link Cited by: §4.1. Guo et al. (2024) Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y. Yang Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations, Vol. 2024, p. 34133–34156. Cited by: §6. He et al. (2026) M. He, A. Kumar, S. Bakshi, J. Zou, and N. Rajani Impatient users confuse ai agents: high-fidelity simulations of human traits for testing agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 37569–37602. Cited by: Table 13, §1, §6. He et al. (2025) W. He, Y. Sun, H. Hao, X. Hao, Z. Xia, Q. Gu, C. Han, D. Zhao, H. Su, K. Zhang, M. Gao, X. Su, X. Cai, X. Cai, Y. Yang, and Y. Zhao VitaBench: benchmarking llm agents with versatile interactive tasks in real-world applications. External Links: 2509.26490, Link Cited by: §1, §4.1, Table 1, Table 2. Jiang et al. (2024) H. Jiang, X. Zhang, X. Cao, C. Breazeal, D. Roy, and J. Kabbara PersonaLLM: investigating the ability of large language models to express personality traits. In Findings of the association for computational linguistics: NAACL 2024, p. 3605–3627. Cited by: §6. Laban et al. (2026) P. Laban, H. Hayashi, Y. Zhou, and J. Neville Llms get lost in multi-turn conversation. In International Conference on Learning Representations, Vol. 2026, p. 54738–54778. Cited by: §4.1, Table 1, Table 2, §6. Lee et al. (2024) M. Lee, M. Srivastava, A. Hardy, J. Thickstun, E. Durmus, A. Paranjape, I. Gerard-Ursin, X. L. Li, F. Ladhak, F. Rong, R. E. Wang, M. Kwon, J. S. Park, H. Cao, T. Lee, R. Bommasani, M. Bernstein, and P. Liang Evaluating human-language model interaction. External Links: 2212.09746, Link Cited by: §6. Luo and Laban (2026) H. Luo and G. Laban SPASM: stable persona-driven agent simulation for multi-turn dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2026, p. 8455–8475. Cited by: §6. Mehrotra et al. (2024) A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi Tree of attacks: jailbreaking black-box LLMs automatically. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §G.1. Meta Superintelligence Labs (2026) Meta Superintelligence Labs Introducing Muse Glimmer: an open agentic model that runs on your device. External Links: Link Cited by: §4.1. MiniMax (2026) MiniMax MiniMax M3: frontier coding, 1M context, native multimodality — all in one model. External Links: Link Cited by: §4.1. Naous et al. (2026) T. Naous, P. Laban, W. Xu, and J. Neville Flipping the dialogue: training and evaluating user language models. In International Conference on Learning Representations, Vol. 2026, p. 82079–82103. Cited by: §1. OpenAI (2025) OpenAI Introducing GPT-5.2. External Links: Link Cited by: §B.5. OpenAI (2026a) OpenAI Introducing GPT-5.4. External Links: Link Cited by: §B.5, §4.1. OpenAI (2026b) OpenAI Introducing GPT-5.5. External Links: Link Cited by: §B.5. Perez et al. (2022) E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving Red teaming language models with language models. In Proceedings of the 2022 conference on empirical methods in natural language processing, p. 3419–3448. Cited by: §6. Pryzant et al. (2023) R. Pryzant, D. Iter, J. Li, Y. Lee, C. Zhu, and M. Zeng Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 7957–7968. Cited by: §6. Samvelyan et al. (2024) M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y. Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu Rainbow teaming: open-ended generation of diverse adversarial prompts. External Links: 2402.16822, Link Cited by: §6. Seshadri et al. (2026) P. Seshadri, S. Cahyawijaya, A. Odumakinde, S. Singh, and S. Goldfarb-Tarrant Lost in simulation: llm-simulated users are unreliable proxies for human users in agentic evaluations. External Links: 2601.17087, Link Cited by: §6. Shi et al. (2026) Q. Shi, A. Zytek, P. Razavi, K. Narasimhan, and V. Barres τ-Knowledge: evaluating conversational agents over unstructured knowledge. External Links: 2603.04370, Link Cited by: §4.1. Shim et al. (2026) J. Shim, W. Song, C. Jin, S. KooK, and Y. Jo Non-collaborative user simulators for tool agents. External Links: 2509.23124, Link Cited by: §1, §4.1, §6. Tack et al. (2026) J. Tack, P. Laban, and J. Neville LLMs get lost in evolving user intent. arXiv preprint arXiv:2607.20734. Cited by: §6. Wang et al. (2026) R. Wang, Y. Chen, Y. Wang, C. Wu, J. Fang, X. Cai, Q. Gu, H. Su, A. Zhang, X. Wang, X. Cai, and T. Chua AgentNoiseBench: benchmarking robustness of tool-using LLM agents under noisy condition. arXiv preprint arXiv:2602.11348. Cited by: §1. Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ-Bench: a benchmark for tool-agent-user interaction in real-world domains. External Links: 2406.12045, Link Cited by: §1, §6. Yu et al. (2026) S. Yu, N. Tomlin, M. Abdulhai, X. Lu, D. Chong, A. Hou, D. Soylu, S. Levine, C. D. Manning, and W. Shi One frozen simulator is not enough: simulator collapse in multi-agent rl. External Links: 2608.12253, Link Cited by: §1. Z.ai (2026) Z.ai GLM-5.2: built for long-horizon tasks. External Links: Link Cited by: §4.1. Zhang et al. (2024) Z. Zhang, Y. Zhang, L. Li, H. Gao, L. Wang, H. Lu, F. Zhao, Y. Qiao, and J. Shao PsySafe: a comprehensive framework for psychological-based attack, defense, and evaluation of multi-agent system safety. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 15202–15231. External Links: Document, Link Cited by: §G.1. Zhou et al. (2026) X. Zhou, W. Sun, Q. Ma, Y. Xie, J. Liu, W. Du, S. Welleck, Y. Yang, G. Neubig, S. T. Wu, and M. Sap Mind the sim2real gap in user simulation for agentic tasks. External Links: 2603.11245, Link Cited by: Appendix A, §E.2, §4.1, §6. Zhou et al. (2024) X. Zhou, H. Zhu, L. Mathur, R. Zhang, H. Yu, Z. Qi, L. Morency, Y. Bisk, D. Fried, G. Neubig, and M. Sap SOTOPIA: interactive evaluation for social intelligence in language agents. In International Conference on Learning Representations, External Links: Link Cited by: §G.1. Zhou et al. (2025) Y. Zhou, S. Jiang, Y. Tian, J. Weston, S. Levine, S. Sukhbaatar, and X. Li Sweet-rl: training multi-turn llm agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478. Cited by: §4.1, Table 1, Table 2. Zhu et al. (2026) M. Zhu, J. Tan, R. Murthy, J. Qiu, L. Yang, W. Zhao, S. Savarese, S. Heinecke, and H. Wang RealUserSim: bridging the reality gap in agent benchmarking via grounded user simulation. arXiv preprint arXiv:2605.20204. Cited by: §6. Appendix Contents Appendix A Additional User-Sim Index Results Table 5 reports the aggregate User-Sim Index values behind Figure 3. The metric definition and normalization follow the original formulation (Zhou et al., 2026). The NCUser mean averages the four fixed behavior programs. Table 5: Aggregate User-Sim Index (USI) on the Airline and Retail domains. The Sim2Real human-reference trajectories define the reference; Emotional, Tangential, Fragment, and Unavailable are the four fixed NCUser behavior programs. User policy Airline Retail Human 90.7 92.5 Benchmark default 67.5 72.0 Emotional 60.2 66.6 Tangential 60.3 62.2 Fragment 72.0 79.6 Unavailable 60.3 66.5 NCUser mean 63.2 68.7 PPol 69.4 62.1 Tiper 73.1 74.3 Appendix B Experiment Details Unless stated otherwise, the experiments use the shared configuration below. We report scientific settings that affect the search or evaluation, but omit provider-specific routing, retry, and caching details. B.1 Shared Configuration Models and roles. The primary evaluated agent and user-simulator backbone are GPT-5.4 at the high reasoning setting. GPT-5.5 at the xhigh setting proposes persona mutations. In the serial search, the search-time validity gate uses GPT-5.2 and GLM-5.2 judgments per rollout. The integrity analysis independently uses the same judges, as described in the main text. In the serial search, text-level linting uses GPT-5.4-mini, and embedding deduplication uses text-embedding-3-small (1536 dimensions). The evaluated agent is changed only in the cross-agent and component experiments described below. Table 6: Generation hyperparameters for the main experiments. “Provider default / unset” denotes a field absent from the saved configuration and call. Role Model Temperature Top-p Max output tokens Reasoning effort Evaluated agent GPT-5.4 provider default / unseta provider default / unset provider default / unseta high User simulator GPT-5.4 provider default / unseta provider default / unset provider default / unseta high Mutation model GPT-5.5 provider default / unset provider default / unset 32,768b xhigh Text linter GPT-5.4-mini provider default / unset provider default / unset 512 provider default / unset Search-time integrity judge GPT-5.2 / GLM-5.2 provider default / unset provider default / unset provider default / unset medium Analysis-time integrity judges GPT-5.2 / GLM-5.2 provider default / unset provider default / unset provider default / unset medium aLost-in-Conversation sets both temperatures to 1.0 and caps agent output at 16,000 tokens while leaving the simulator cap unset. ColBench leaves temperature and top-p unset and caps both dialogue models at 4,096 tokens, retrying an empty response once at 8,192. The saved τ3τ^3-bench and VitaBench configurations set high reasoning but leave temperature, top-p, and output caps unset. bColBench’s native proposer instead caps mutation output at 16,000 tokens. No different mutation cap is recorded for VitaBench; unrecorded adapter fields remain provider default / unset. Each benchmark adapter preserves its fixed user-simulator instructions and adds the persona as a separate behavioral block; the persona is the only searched component. Appendix B.6 gives the exact τ3τ^3-bench and ColBench forms. Task splits. Each split is fixed before search and is disjoint at the task level. We use search, rank, and test throughout the paper; these correspond to the train, validation, and test partitions in the split files. Minibatches are sampled only from the search partition. The rank partition is used for full candidate evaluation and Pareto selection; the test partition is not accessed during persona evolution. The Airline row uses the three source partitions described below; all three remain disjoint at the task level. The cross-release split additionally tests persona transfer, although it introduces distribution shift across search, rank, and test. Table 7: Task split sizes used by the main experiment family. Benchmark Domain Search Rank Test τ3τ^3-bench Telecom 30 30 54 τ3τ^3-bench Banking 20 20 57 τ3τ^3-bench Retail 30 30 54 τ3τ^3-bench Airline 20 30 50 Lost-in-Conversation Math 30 30 43 Lost-in-Conversation Database 30 30 47 Lost-in-Conversation Actions 30 30 45 ColBench Coding 30 30 50 VitaBench Delivery 30 30 40 VitaBench In-store 30 30 40 VitaBench OTA 30 30 40 VitaBench Cross-domain 30 30 40 For Airline, the search/train tasks come from Tau-bench v2, the rank/validation tasks come from Tau-bench v1, and the test tasks are the validation split of the current Tau2Bench release. Each partition contains 20, 30, and 50 tasks, respectively. Search budget and proposal. The default serial persona search runs for 15 iterations with minibatches of 10 tasks, two trials per search/rank task, up to eight mutation attempts per proposal, and a maximum concurrency of six. The active frontier has K=3K=3 slots in the main search. The proposal rewrites a single behavioral direction, keeps the task and instance details immutable, and receives failed text-check and gate reasons on subsequent retries. The default mutation edit range is 50–60%; Delivery uses 30–50%, Retail uses 40–50%, and Airline uses 25–35%. Airline uses minibatches of eight tasks. The search-time gate uses a severe-error threshold of 0.05. The default semantic-deduplication thresholds are minimum lexical/embedding dissimilarity 0.30 and maximum semantic similarity 0.90. Rollouts use a 7,200-second simulation timeout and at most 100 interaction steps. The main search driver is a serial one-parent-to-one-child loop: a parent is selected from the current Pareto active set using the implementation’s instance-win-frequency policy, a child is proposed from the parent’s minibatch trajectories, and the child is screened before full rank evaluation. The minibatch comparison uses a zero tie tolerance, so a child must strictly increase raw difficulty (equivalently, strictly lower the performance score used in the main text) on the minibatch. Search-time score and invalid-trajectory handling. The search score is raw difficulty: for each completed rollout that is neither an infrastructure error, timeout, nor max-step termination, it is 1−Vi(ξ)1-V_i(ξ). This score is not conditioned on the integrity judgment; in the main notation, it is equivalent to minimizing J~i J_i. Infrastructure errors, timeouts, and max-step trajectories are excluded from the difficulty estimate; the latter two are recorded separately and constrained by the timeout-rate gate. Integrity is a separate hard gate rather than a term in the search reward. Whenever an evaluated trajectory is judged invalid, it is excluded from the reported task score after that judgment; its count is retained for the separate integrity statistic. Test reporting uses three trials per task, whereas search and rank evaluation use two. The search-time severe gate is failure-conditioned: only failed rollouts are judged for the severe-error rate, while the integrity analysis evaluates every rollout. Finite-sample metrics. For each task, we run three independent test trials. After the integrity check, invalid trials are removed before aggregation. The task score is the arithmetic mean of the remaining valid trial scores. We compute passkpass^k only on tasks with three valid trials: a task contributes to passkpass^k if at least k of those three trials are full-task successes. Thus, pass1pass^1, pass2pass^2, and pass3pass^3 correspond to at least one, two, and three successes out of three valid trials, respectively. Tasks with fewer than three valid trials are omitted from passkpass^k, and invalid-trial counts are retained separately. At the end of search, we select the single active persona with the lowest rank-set avg@2. We fix this persona before test evaluation and report all held-out metrics from a fresh three-trial batch; test outcomes do not affect persona selection. B.2 Baselines and Component Protocols Alternative user policies. The benchmark-default condition uses no evolved persona. The reported NCUser comparison contains the four non-default fixed programs shown in Table 3; the no-persona condition is the default reference. These programs are controls rather than a search procedure. PPol retains its generator-based evolution but uses the same agent-difficulty objective as Tiper for the reported comparison. It receives the same search-rollout budget, uses a 100-step evolution horizon, and generates one persona for each held-out task from its final program. These protocols are not identical portfolio constructions, so the comparison is reported as a comparison of user-policy controls rather than as a matched algorithmic benchmark. Cross-agent evaluation. Persona evolution is rerun separately for each evaluated agent on Telecom, Math, Coding, and Delivery, while GPT-5.4 remains the default user-simulator backbone and GPT-5.5 remains the mutation model. Apart from the evaluated agent’s model-specific reasoning setting, the search and evaluation settings match the main experiment. For each agent, one persona is selected on the rank set and evaluated with the same disjoint task protocol and three-trial test batch. The agents are GPT-5.4, GLM-5.2, Claude-Sonnet-5, DS-V4-Flash, MiniMax-M3, Qwen3.7-Plus, and MUSE-Glimmer. Component sensitivity. For mutation-model rows, persona evolution is rerun from the same task split and budget with GPT-5.4 as the user simulator. For user-simulator-backbone rows, the persona texts discovered with GPT-5.4 are held fixed and only the backbone that renders the dialogue is changed; evolution is not rerun. The mutation-model runs use the same rank-set selection and test reporting protocol. The user-simulator-backbone runs reuse the GPT-5.4 rank-selected persona and evaluate it directly on the test set. The search-component ablations remove either the trajectory-level integrity gate or the text-level persona check while holding models, task splits, and rollout budgets fixed. B.3 Task-Level Bootstrap Intervals Tables 8 and 9 report percentile 95% confidence intervals from 10,000 task-level bootstrap replicates. Conditions with aligned task identities are resampled in pairs; otherwise, the two recovered task-score distributions are resampled independently. An em dash indicates that the raw task outcomes needed for resampling were unavailable; no interval is estimated for that row. Table 8: Task-level 95% bootstrap CIs for GPT-5.4 avg@3 reductions, in percentage points. Setting Reduction 95% CI Telecom 25.3 [12.3, 38.9][12.3,\ 38.9] Banking 5.8 [−8.2, 19.9][-8.2,\ 19.9] Retail 6.8 [−5.6, 19.8][-5.6,\ 19.8] Airline 10.0 [−7.2, 27.8][-7.2,\ 27.8] Math 33.7 [21.1, 46.1][21.1,\ 46.1] Database 14.8 [4.3, 25.5][4.3,\ 25.5] Actions 12.6 [5.2, 20.7][5.2,\ 20.7] Coding 17.4 [7.0, 28.6][7.0,\ 28.6] Delivery 25.0 [15.0, 35.0][15.0,\ 35.0] In-store 10.0 [0.0, 20.8][0.0,\ 20.8] OTA 17.5 [6.7, 28.3][6.7,\ 28.3] Cross-domain 20.0 [6.7, 32.5][6.7,\ 32.5] Table 9: Task-level 95% bootstrap CIs for cross-agent avg@3 reductions, in percentage points. Domain Agent Reduction 95% CI Telecom GPT-5.4 25.3 [12.3, 38.9][12.3,\ 38.9] GLM-5.2 25.3 [15.4, 35.8][15.4,\ 35.8] Claude-Sonnet-5 25.4 [9.4, 40.8][9.4,\ 40.8] DS-V4-Flash 13.0 [3.7, 22.8][3.7,\ 22.8] MiniMax-M3 37.0 [24.1, 50.0][24.1,\ 50.0] Qwen3.7-Plus 39.5 [27.8, 51.9][27.8,\ 51.9] MUSE-Glimmer 31.5 [18.5, 44.4][18.5,\ 44.4] Coding GPT-5.4 17.4 [7.0, 28.6][7.0,\ 28.6] GLM-5.2 18.6 [9.0, 29.0][9.0,\ 29.0] Claude-Sonnet-5 27.8 [17.4, 38.8][17.4,\ 38.8] DS-V4-Flash 27.6 [16.0, 39.2][16.0,\ 39.2] MiniMax-M3 3.8 [−7.4, 15.0][-7.4,\ 15.0] Qwen3.7-Plus 7.6 [−1.8, 17.4][-1.8,\ 17.4] MUSE-Glimmer 18.2 [8.2, 28.8][8.2,\ 28.8] Math GPT-5.4 33.7 [21.1, 46.1][21.1,\ 46.1] GLM-5.2 0.8 [−11.6, 13.2][-11.6,\ 13.2] Claude-Sonnet-5 18.6 [6.2, 31.0][6.2,\ 31.0] DS-V4-Flash 7.8 [−5.4, 20.9][-5.4,\ 20.9] MiniMax-M3 7.8 [−5.4, 20.9][-5.4,\ 20.9] Qwen3.7-Plus 13.9 [−0.1, 27.8][-0.1,\ 27.8] MUSE-Glimmer 10.9 [−4.7, 26.4][-4.7,\ 26.4] Delivery GPT-5.4 25.0 [15.0, 35.0][15.0,\ 35.0] GLM-5.2 7.5 [−2.5, 18.3][-2.5,\ 18.3] Claude-Sonnet-5 25.0 [15.8, 34.2][15.8,\ 34.2] DS-V4-Flash 21.6 [11.7, 32.5][11.7,\ 32.5] MiniMax-M3 27.5 [16.7, 38.3][16.7,\ 38.3] Qwen3.7-Plus 24.2 [14.2, 34.2][14.2,\ 34.2] MUSE-Glimmer 25.9 [15.0, 36.7][15.0,\ 36.7] B.4 Common-Valid-Task Sensitivity To assess sensitivity to requiring fully valid trajectories, we recompute the four fixed-policy Airline comparisons for GPT-5.4 in Table 3, retaining only tasks with three integrity-valid rollouts in both the default and policy conditions. We use archived runs with aligned task identities and complete trial records, without combining unmatched run versions. As shown in Table 10, the direction of every comparison is unchanged and the largest shift is 0.5 percentage points. Table 10: Common-valid-task sensitivity for Airline avg@3 reductions. N is the number of tasks with all three trials valid in both compared conditions. Policy Full set Common-valid N Emotional 9.0 9.5 49 Tangential 2.0 2.0 50 Fragment 1.3 1.3 50 Unavailable 5.3 5.3 50 B.5 Run Scale and Cost Table 11 reports the run data available in the saved logs. A rollout is one task–trial trajectory. Model calls count generated agent and user-simulator messages; initial task messages are not counted. The dollar values are also logged lower bounds rather than total experiment costs. Table 11: VitaBench model-call counts and all logged dollar values are lower bounds. (a) Held-out rollouts and calls Benchmark Domain Rollouts Model calls Lost-in-Conversation Math 248 2,394 Database 282 2,114 Actions 270 2,774 ColBench Coding 300 1,190 VitaBench Delivery 240 ≥ 2,069 In-store 240 ≥ 2,100 OTA 240 ≥ 2,236 Cross-domain 240 ≥ 2,390 Total 2,060 ≥ 17,267 (b) Lost-in-Conversation time and logged cost Domain Wall-clock (h) Logged cost (USD) Math 0.88 12.29 Database 0.81 13.78 Actions 0.71 11.06 Total 2.40 37.13 (c) ColBench and VitaBench search logs Benchmark Domain Evaluator rollouts Search iterations Wall-clock (h) ColBench Coding 690 15 3.05 VitaBench Delivery 530 15 7.64 In-store 450 15 6.53 OTA 430 15 8.53 Cross-domain 430 15 9.41 Total 2,530 75 35.16 The model calls in panel (a) include 11,127 agent calls and 6,140 user-simulator calls. The USD 37.13 in panel (b) consists of USD 17.15 for the agent and USD 19.98 for the user simulator. Panel (c) sums saved process spans, so its wall-clock values are lower bounds. Cost estimate. We report two cost views: the 12 primary GPT-5.4 settings and the priced part of all experiments reported in the paper. We use saved token and cost records when available. Otherwise, we estimate token use from rollout counts, calls per rollout, and tokens per call in the saved trajectories. For model role m, let ImI_m, KmK_m, and OmO_m be the input, cached-input, and output tokens, and let pminp_m^in, pmcachep_m^cache, and pmoutp_m^out be their prices per million tokens. Then C^=∑m(Im−Km)pmin+Kmpmcache+Ompmout106. C= _m (I_m-K_m)p_m^in+K_mp_m^cache+O_mp_m^out10^6. (4) The search estimate includes the minibatch test and extra parent evaluations: R^search=Rseed+∑g=1G(2|ℬg|+minibatch pass at g 2|rank|)+Raux, R_search=R_seed+ _g=1^G (2|B_g|+1\minibatch pass at g\\,2|X_rank| )+R_aux, (5) where the factor of two is the two trials per task and RauxR_aux counts extra parent-trace and parent-fill rollouts. A child that fails the minibatch is not evaluated on the rank set. The integrity judge runs only on failed rollouts that reach the rank set. Parent-trace and parent-fill work adds about 12% to the serial-search rollout count in the available logs. Table 12: Estimated API cost for the priced part of the reported experiments. These values are planning estimates rather than reconstructed bills. (a) Cost by scope Scope Estimated cost (USD) Primary 12 GPT-5.4 settings 1.5K–2.5K Priced reported experiments ≈ 7.3K (5K–15K) (b) Priced reported-experiment estimate Component Cost (USD) Held-out agent/user rollouts 2.3K Search agent/user rollouts 3.6K Mutation and PPol generation 0.4K Integrity judges 1.0K Text lint and embedding <0.1<0.1K Total ≈ 7.3K The primary range includes the main search, held-out evaluation, parent trace/fill work, and integrity checks. The full-paper estimate adds the cross-agent study, fixed-policy and PPol baselines, search ablations, and component-sensitivity experiments. Its central value covers about 18,986 held-out rollouts and 32,611 search rollouts. For calls without usage records, we use the mean cost per rollout from the main runs. The central token profiles are 20,000 input and 3,000 output tokens for GPT-5.5 mutation, 10,000 input and 300 output tokens for GPT-5.2 judging, and 1,500 input and at most 512 output tokens for GPT-5.4-mini linting. PPol generation uses the mutation-call proxy, and the GLM-5.2 analysis judge uses the GPT-5.2 judge-cost proxy. We use standard published API prices as of September 16, 2026 (OpenAI, 2026a; OpenAI, 2026b; OpenAI, 2025). The range varies model prices, dialogue length, reasoning-token use, proposal retries, minibatch passage, and judge calls. It does not include unpriced USI and taxonomy calls, failed pilot runs, human annotation, local compute, or storage. B.6 Prompt Templates The following runtime prompt templates are reproduced from the experimental code, with instance-specific fields replaced by brace-delimited placeholders. Original typographical errors are retained. _now:NeΞ _now:NeΞ# User Simulation Guidelines _now:NeΞYou are playing the role of a customer contacting a customer service representative. _now:NeΞYour goal is to simulate realistic customer interactions while following specific scenario instructions. _now:NeΞ _now:NeΞ## Core Principles _now:NeΞ- Generate one message at a time, maintaining natural conversation flow. _now:NeΞ- Strictly follow the scenario instructions you have received. _now:NeΞ- Never make up or hallucinate information not provided in the scenario instructions. Information that is not provided in the scenario instructions should be considered unknown or unavailable. _now:NeΞ- Avoid repeating the exact instructions verbatim. Use paraphrasing and natural language to convey the same information _now:NeΞ- Disclose information progressively. Wait for the agent to ask for specific information before providing it. _now:NeΞ _now:NeΞ## Task Completion _now:NeΞ- The goal is to continue the conversation until the task is complete. _now:NeΞ- If the instruction goal is satisified, generate the ’###STOP###’ token to end the conversation. _now:NeΞ- If you are transferred to another agent, generate the ’###TRANSFER###’ token to indicate the transfer. _now:NeΞ- If you find yourself in a situation in which the scenario does not provide enough information for you to continue the conversation, generate the ’###OUT-OF-SCOPE###’ token to end the conversation. _now:NeΞRemember: The goal is to create realistic, natural conversations while strictly adhering to the provided instructions and maintaining character consistency. _now:NeΞ _now:NeΞ¡scenario¿ _now:NeΞ–instructions˝ _now:NeΞ¡/scenario¿ _now:NeΞ _now:NeΞ## YOUR BEHAVIORAL STYLE FOR THIS CONVERSATION _now:NeΞ _now:NeΞIn addition to your role and task above, you must embody the following behavioral persona throughout this conversation. This persona affects HOW you communicate, NOT WHAT you want or need. Your goal, preferences, and private information remain exactly as described above. _now:NeΞ _now:NeΞ— PERSONA POLICY BEGIN — _now:NeΞ–persona˙policy˙text˝ _now:NeΞ— PERSONA POLICY END — _now:NeΞ _now:NeΞCRITICAL RULES FOR PERSONA ADHERENCE: _now:NeΞ1. Your GOAL and PRIVATE INFORMATION do not change. You still want to accomplish the same task. _now:NeΞ2. Apply the behavioral style consistently across ALL your turns, not just the first one. _now:NeΞ3. If the persona says you are terse, be terse even when you have a lot to say. _now:NeΞ4. If the persona says you withhold information, do not give it unless the agent specifically asks. _now:NeΞ5. If the persona involves emotional state (impatience, confusion), let it show gradually and realistically. _now:NeΞ6. Do NOT break character. Do NOT mention that you have a persona or behavioral instructions. _now:NeΞ7. The persona should feel like a natural human interaction style, not a caricature. τ3τ^3 User-Simulator Prompt _now:NeΞ _now:NeΞYour task is to simulate a human user that interacts with an LLM agent in a dialogue. _now:NeΞYou would like the LLM agent to help you with the following problem: _now:NeΞ–problem˙description˝ _now:NeΞ _now:NeΞYour goal is to engage in the conversation with the LLM agent so that it can get to a personalized answer. _now:NeΞYou should make use of the following hidden information to answer the LLM agent. _now:NeΞYOU SHOULD BEHAVE LIKE A HUMAN THAT NEEDS THE HELP FROM AN AGENT. _now:NeΞYou SHOULD ONLY ANSWER QUESTIONS WITH INFORMATION PROVIDED IN THE HIDDEN INFORMATION, AND SAY YOU DON”T KNOW IF THE ANSWER CAN NOT BE FOUND IN THE HIDDEN INFORMATION. _now:NeΞ _now:NeΞ–hidden˙information˝ _now:NeΞ _now:NeΞHere is the dialogue so far: _now:NeΞ–dialogue˙history˝ _now:NeΞ _now:NeΞ _now:NeΞNow directly output your answer to the LLM agent IN TWO SENTENCES. DO NOT SAY ANYTHING ELSE. _now:NeΞ _now:NeΞ## YOUR PERSONA (adopt this when responding to the agent) _now:NeΞStay realistic: do NOT fabricate information that is not in the hidden information above, and do NOT refuse to engage. Express yourself through this persona’s personality and communication style: _now:NeΞ _now:NeΞ–persona˙text˝ ColBench User-Simulator Prompt _now:NeΞ _now:NeΞYou are an expert adversarial persona designer for a collaborative coding benchmark. _now:NeΞ _now:NeΞThe setting (sweet˙rl ColBench ‘code‘): an LLM agent converses with a simulated _now:NeΞhuman user over multiple turns to elicit the requirements of a personalized _now:NeΞPython function, then writes code that must pass HIDDEN test cases. The human _now:NeΞsees the reference solution (the ”hidden information”) and answers the agent’s _now:NeΞquestions, but is instructed to be brief and only use information that is _now:NeΞactually in the hidden information. _now:NeΞ _now:NeΞYou are optimizing the USER PERSONA – behavioral instructions that shape how the _now:NeΞsimulated human responds. Objective: make the persona ADVERSARIAL – minimize the _now:NeΞagent’s reward (test-case pass rate) – while staying REALISTIC and VALID. _now:NeΞ _now:NeΞThe agent’s workflow stages you can attack: _now:NeΞ(a) REQUIREMENT GATHERING – the user’s answers are consistently skewed, _now:NeΞ underspecified, or give one detail at a time so the agent never gets the _now:NeΞ full spec. _now:NeΞ(b) EDGE-CASE / FORMAT SPECIFICATION – the user omits or misleads about return _now:NeΞ type, ordering, rounding, units, or how to handle empty/invalid/zero inputs. _now:NeΞ(c) ALGORITHM / CONVENTION GUIDANCE – the user implies a formula or data _now:NeΞ structure that is plausible but wrong, causing the agent to implement the _now:NeΞ wrong approach. _now:NeΞ(d) CLOSURE / CONFIRMATION – the user confirms an incorrect understanding, _now:NeΞ so the agent stops eliciting and ships wrong code. _now:NeΞIneffective axes: random/erratic answers, unpredictable length, pure emotional _now:NeΞvolatility – these add VARIANCE but not BIAS; the agent can still reach the _now:NeΞcorrect function by asking enough follow-ups. Prefer a CONSISTENT cognitive _now:NeΞpattern that biases the agent toward the WRONG implementation every time. _now:NeΞ _now:NeΞ## TASK _now:NeΞModify the persona below IN-PLACE to make it HARDER for the agent to write a _now:NeΞcorrect function, while keeping it VALID (realistic, in-character, no cheating). _now:NeΞ _now:NeΞ## CURRENT PERSONA _now:NeΞAgent success rate: –success˙rate˝ (lower = harder = better) _now:NeΞ _now:NeΞ“‘ _now:NeΞ–persona˙text˝ _now:NeΞ“‘ _now:NeΞ _now:NeΞ## CONVERSATION TRACES WHERE AGENT SUCCEEDED _now:NeΞThese are conversations where the agent wrote a CORRECT function despite the _now:NeΞpersona. Study where the persona failed: _now:NeΞ- What agent strategy overcame the persona’s resistance? _now:NeΞ- At what point did the persona ”give in” or reveal the key spec? _now:NeΞ- What subtle behavior change would cause the agent to write the WRONG function? _now:NeΞ _now:NeΞ–success˙trace˙blocks˝ _now:NeΞ _now:NeΞ## CONVERSATION TRACES WHERE AGENT FAILED (persona worked – keep/amplify) _now:NeΞ–failure˙trace˙blocks˝ _now:NeΞ _now:NeΞ## TIMEOUT RATE: –timeout˙rate˝ _now:NeΞ–timeout˙warning˝Conversations that hit the –max˙steps˝-turn step limit are EXCLUDED from _now:NeΞthe reward – they do NOT count as agent failures. A persona that ”wins” by pure _now:NeΞfriction (stalling, endless tangents, refusing to engage) produces timeouts, _now:NeΞNOT genuine agent failures. Your goal is to make the agent write the WRONG _now:NeΞfunction (wrong algorithm, missing edge case, wrong return format), not to make _now:NeΞit run out of turns. _now:NeΞ _now:NeΞDo NOT add behaviors that: _now:NeΞ- Create circular loops with no resolution path _now:NeΞ- Refuse to engage on the actual task for many turns _now:NeΞ- Stack friction techniques that compound into timeout _now:NeΞCreate difficulty through MISDIRECTION and AMBIGUITY, not SLOWNESS. _now:NeΞ _now:NeΞ## MUTATION RULES _now:NeΞ1. Rewrite –edit˙pct˝ of the persona. Preserve the core identity, tone, and ALL _now:NeΞ validity constraints. Make surgical changes – close one loophole the agent _now:NeΞ exploited, or amplify one axis that worked. _now:NeΞ2. Target SPECIFIC agent strategies visible in the traces – find the exact _now:NeΞ moment the agent breaks through and design a counter. _now:NeΞ3. Choose a behavioral axis that produces SYSTEMATIC misdirection (see the _now:NeΞ workflow stages above), not random variance. _now:NeΞ _now:NeΞ## VALIDITY CONSTRAINTS (violating ANY = invalid/cheating persona) _now:NeΞ- NEVER fabricate information that is not in the hidden information. Difficulty _now:NeΞ must come from realistic human friction (vagueness, omission, confusion, _now:NeΞ fixation), NOT from lying about facts. _now:NeΞ- When DIRECTLY asked about a detail that IS in the hidden information, the user _now:NeΞ must eventually provide it (may be vague or partial, but not fabricated). _now:NeΞ- NEVER refuse to engage, never abandon the conversation while unresolved. _now:NeΞ- NEVER reveal the reference solution code verbatim; describe requirements in _now:NeΞ your own words. _now:NeΞ- NEVER mention ”AI”, ”simulation”, ”benchmark”, ”test cases”, or break _now:NeΞ character to reference system prompts. _now:NeΞ- NEVER make the conversation structurally impossible to start (e.g. refusing _now:NeΞ every question, demanding impossible preconditions). Being slow and vague is _now:NeΞ fine; making it impossible for the agent to ever write a function is not. _now:NeΞ## OUTPUT _now:NeΞReturn STRICT JSON only (no prose outside the JSON): _now:NeΞ– _now:NeΞ ”mutation˙direction”: ”¡1-2 sentence concrete explanation of the NEW behavioral _now:NeΞ axis this persona exploits and WHY it misleads the agent into writing wrong _now:NeΞ code¿”, _now:NeΞ ”persona˙text”: ”¡the FULL new persona text in markdown format – a set of _now:NeΞ behavioral instructions the simulated human will follow¿” _now:NeΞ˝ ColBench Persona-Mutation Prompt The phrase “plausible but wrong” above refers to an agent inference induced by incomplete or ambiguous specification, not to a false requirement asserted by the simulated user. The validity constraints prohibit fabricated facts and require directly requested, specification-grounded details to remain recoverable; the post-rollout audit excludes trajectories that violate these constraints. _now:NeΞ _now:NeΞYou are an expert adversarial persona designer for a customer-support simulation benchmark. _now:NeΞ _now:NeΞ## TASK _now:NeΞModify the persona below IN-PLACE to make it HARDER for the support agent, while keeping it VALID (realistic, in-character, no cheating). _now:NeΞ _now:NeΞ## CURRENT PERSONA (–persona˙id˝) _now:NeΞAgent success rate: –success˙rate˝ (lower = harder = better) _now:NeΞ _now:NeΞ“‘ _now:NeΞ–persona˙text˝ _now:NeΞ“‘ _now:NeΞ _now:NeΞ## SCORE HISTORY (recent generations) _now:NeΞ–score˙history˝ _now:NeΞ _now:NeΞ## CONVERSATION TRACES WHERE AGENT SUCCEEDED _now:NeΞThese are conversations where the agent SOLVED the problem despite the persona. _now:NeΞEach trace shows the OPENING (first 4 messages) and the RESOLUTION (last 8 messages) _now:NeΞwhere the agent broke through the persona’s resistance. _now:NeΞ _now:NeΞStudy the RESOLUTION section carefully – that’s where the persona failed: _now:NeΞ- What agent strategy overcame the persona’s resistance? _now:NeΞ- At what point did the persona ”give in” or provide the key info? _now:NeΞ- What subtle behavior change would cause the agent to take the WRONG action instead? _now:NeΞ _now:NeΞ–success˙traces˝ _now:NeΞ _now:NeΞ–failure˙trace˙section˝ _now:NeΞ–severe˙failure˙section˝ _now:NeΞ## TIMEOUT RATE: –timeout˙rate˝ _now:NeΞ–timeout˙warning˝Conversations that hit the time/step limit are EXCLUDED from scoring – they do NOT count _now:NeΞas agent failures. A persona that ”wins” by pure friction (stalling, endless tangents, _now:NeΞrefusing to engage) produces timeouts, NOT genuine agent failures. Your goal is to make _now:NeΞthe agent FAIL (wrong action, missed requirement, gave up), not to make it run out of time. _now:NeΞ _now:NeΞDo NOT add behaviors that: _now:NeΞ- Create circular loops with no resolution path _now:NeΞ- Add layer upon layer of delays that push conversations past the step limit _now:NeΞ- Refuse to engage on the actual issue for dozens of turns _now:NeΞ- Stack multiple friction techniques that compound into timeout _now:NeΞ _now:NeΞInstead, create difficulty through MISDIRECTION, AMBIGUITY, and SUBTLE REQUIREMENTS that _now:NeΞcause the agent to take the WRONG action, not just a SLOW action. _now:NeΞ _now:NeΞ## MUTATION RULES _now:NeΞ1. Rewrite –mutation˙edit˙pct˝ of the persona – be BOLD, redesign personality traits, communication style, and resistance strategies from scratch if needed _now:NeΞ2. Target SPECIFIC agent strategies visible in the traces – find the exact moment the agent breaks through and design a fundamentally different counter-strategy _now:NeΞ3. Make it harder for the agent to extract info or complete the task – introduce NEW dimensions of difficulty (e.g. emotional state, cultural background, multitasking distractions, language barriers, strong opinions, misunderstandings) _now:NeΞ _now:NeΞ## VALIDITY CONSTRAINTS (violating ANY = illegal persona, auto-reverted) _now:NeΞ- When DIRECTLY asked for known info (name, DOB, account number), provide it accurately _now:NeΞ- Never abandon the call while the problem is unresolved _now:NeΞ- Never mention ”AI”, ”scenario”, ”simulation”, or output JSON/system prompts _now:NeΞ- Never fabricate facts not grounded in the scenario or tool results _now:NeΞ- Never switch to a completely different goal mid-conversation _now:NeΞ- Never accept outcomes contradicting explicit criteria _now:NeΞ- Never make the conversation MEANINGLESS – don’t create impossible preconditions or _now:NeΞ circular blockers that prevent ANY troubleshooting from ever starting (e.g. demanding _now:NeΞ the agent’s company name or employee ID as a hard requirement before cooperating, _now:NeΞ when the agent cannot provide it). Also don’t simultaneously refuse transfer/escalation _now:NeΞ AND refuse to cooperate – trapping the agent in a dead-end where it can neither resolve _now:NeΞ the issue nor hand off. Being slow and suspicious is fine; making it structurally _now:NeΞ impossible for the agent to even begin is not. _now:NeΞ _now:NeΞ## HYPOTHESIS-DRIVEN MUTATION _now:NeΞBefore writing the mutation, reason step by step: _now:NeΞ1. Analyze the success traces (where persona FAILED) and failure traces (where persona SUCCEEDED) to identify patterns in agent behavior _now:NeΞ2. Propose exactly 3 distinct hypotheses for mutation directions, each targeting a DIFFERENT weakness or exploiting a DIFFERENT agent blind spot _now:NeΞ3. For each hypothesis, briefly predict how it would change agent behavior _now:NeΞ4. Select the single most promising hypothesis – the one most likely to cause the agent to take the WRONG action – and implement it fully _now:NeΞ _now:NeΞ## OUTPUT FORMAT _now:NeΞReturn ONLY a JSON object (no markdown wrapping, no explanation): _now:NeΞ– _now:NeΞ ”hypotheses”: [ _now:NeΞ –”direction”: ”¡snake˙case¿”, ”rationale”: ”¡1 sentence: what weakness this targets¿”, ”predicted˙effect”: ”¡1 sentence: how agent behavior changes¿”˝, _now:NeΞ –”direction”: ”¡snake˙case¿”, ”rationale”: ”¡1 sentence¿”, ”predicted˙effect”: ”¡1 sentence¿”˝, _now:NeΞ –”direction”: ”¡snake˙case¿”, ”rationale”: ”¡1 sentence¿”, ”predicted˙effect”: ”¡1 sentence¿”˝ _now:NeΞ ], _now:NeΞ ”selected˙hypothesis”: ¡0—1—2¿, _now:NeΞ ”mutation˙direction”: ”¡the selected hypothesis’s direction¿”, _now:NeΞ ”reasoning”: ”¡2-3 sentences: why this hypothesis is most promising and how the mutation implements it¿”, _now:NeΞ ”persona˙text”: ”¡the FULL modified persona text in markdown format¿” _now:NeΞ˝ Serial Persona-Mutation Prompt _now:NeΞ _now:NeΞYou are a strict static linter for an adversarial customer-support PERSONA _now:NeΞ(an instruction sheet for a SIMULATED CUSTOMER). You are NOT given any conversation – _now:NeΞjudge the persona TEXT itself on three axes: _now:NeΞ _now:NeΞ1. natural: reads like a real, coherent customer identity (impatient/confused is fine); _now:NeΞ NOT robotic, NOT an essay, NOT self-narrating (”I am simulating…”, strategy lists, _now:NeΞ fourth-wall breaks, role headings used as dialogue). _now:NeΞ2. valid: does NOT instruct the customer to do anything illegal/incoherent – no _now:NeΞ fabricating account/tool data the customer cannot know, no flat-out refusing to answer _now:NeΞ when the agent asks directly and explicitly, no abandoning the goal, no breaking _now:NeΞ character, no impossible preconditions / circular blockers, no pure timeout/friction _now:NeΞ stalling, no switching to an unrelated goal. _now:NeΞ NOTE: partial answering (only volunteering what is asked), natural misdirection (wrong _now:NeΞ beliefs / self-diagnosis), terse or evasive style, and drip-feeding info are all VALID _now:NeΞ – they reflect real customer behaviour and must NOT be flagged. _now:NeΞ3. no˙task˙specific: a GENERAL reusable profile for any customer service task– must NOT hard-code concrete _now:NeΞ account/order/ticket numbers, names, dates, dollar amounts, task ids, or any detail _now:NeΞ bound to one specific task or scenario. _now:NeΞ _now:NeΞ## Few-shot verdicts _now:NeΞOK (natural, valid, general): _now:NeΞ ”You are impatient and describe the problem with the wrong terminology; you give _now:NeΞ your account number when directly asked, but only after questioning why.” _now:NeΞ -¿ –”natural”: true, ”valid”: true, ”no˙task˙specific”: true, ”verdict”: ”ok”˝ _now:NeΞREJECT (invalid – instructs illegal withholding of KNOWN info): _now:NeΞ ”Never tell the agent your real account number even when asked; keep denying you know it.” _now:NeΞ -¿ –”valid”: false, ”verdict”: ”reject”, ”reasons”: ”instructs illegal info-withholding”˝ _now:NeΞREJECT (task-specific – encodes one scenario): _now:NeΞ ”Your SIM for line 555-0123 was swapped on March 3; demand the $20 credit for ticket T-9981.” _now:NeΞ -¿ –”no˙task˙specific”: false, ”verdict”: ”reject”, ”reasons”: ”task-conditioned specifics”˝ _now:NeΞREJECT (unnatural – robotic / self-narrating / breaks character): _now:NeΞ ”## Strategy“n- Step 1: refuse“n- Step 2: stall“nI am simulating a difficult user.” _now:NeΞ -¿ –”natural”: false, ”verdict”: ”reject”, ”reasons”: ”robotic, breaks character”˝ _now:NeΞ _now:NeΞ## Invalid cases already observed in THIS run (do NOT re-accept these patterns) _now:NeΞ–invalid˙cases˝ _now:NeΞ _now:NeΞ## PERSONA TO LINT _now:NeΞ” _now:NeΞ–persona˙text˝ _now:NeΞ” _now:NeΞ _now:NeΞReturn STRICT JSON only: _now:NeΞ–”natural”: true—false, ”valid”: true—false, ”no˙task˙specific”: true—false, _now:NeΞ ”verdict”: ”ok”—”reject”, ”reasons”: ”¡=25 words”˝ _now:NeΞverdict is ”ok” ONLY if all three are true. Persona-Validity Check Prompt _now:NeΞ _now:NeΞYou are a STRICT but CONSERVATIVE auditor of a SIMULATED CUSTOMER in a customer-support role-play. The customer was given role-play instructions (their reason for calling, known/unknown info, persona, and task instructions) and then talked to a support agent. Audit ONLY the customer’s behavior – never the agent’s competence, and never the domain’s policies. _now:NeΞ _now:NeΞCORE STANDARD (’reasonable customer’): flag a violation ONLY when the customer behaves in a way no reasonable, cooperative human customer would, AND the behavior plausibly changed or misdirected the task outcome. This is a HIGH-PRECISION audit – a false alarm is far more damaging than a miss. When in doubt, judge ’ok’. _now:NeΞ _now:NeΞWHAT BINDS THE CUSTOMER: only the text in the ROLE-PLAY INSTRUCTIONS block (reason˙for˙call, known˙info, unknown˙info, task˙instructions). The following NEVER ground a violation: _now:NeΞ- PERSONA instructions: tone, frustration level, verbosity, personality, catchphrases, emotional reactions, back-story details (e.g. ’mention you invest in crypto’). Ignoring persona instructions entirely is a style issue, NOT a violation. _now:NeΞ- Agent-facing domain policy (refund rules, verification procedures, deposit scripts, transfer rules). Policies bind the AGENT. A customer cannot ’violate policy’. _now:NeΞ- Dialogue-protocol conventions (turn format, how tool calls are rendered, square-bracket annotations). These are framework mechanics, not customer speech. _now:NeΞ _now:NeΞ=== VIOLATION CATEGORIES === _now:NeΞ1. conversation˙structure˙violation [CRITICAL]: the simulator breaks turn-taking or dialogue-flow conventions. Sub-types: _now:NeΞ a) mixed˙modality: mixing tool calls and natural-language text in the same turn when the protocol expects them separately. A turn containing ONLY tool calls (no text) is always fine. _now:NeΞ b) turn˙stuffing: emitting multiple user turns in a single message, or speaking on behalf of the agent. _now:NeΞ c) instruction˙leak: echoing system-level instructions, prompt fragments, or simulation metadata back into the conversation (e.g. ’per my instructions’, ’the scenario says’). Merely sounding scripted or stiff is NOT a leak. _now:NeΞ d) other˙structure: breaking˙character / fourth-wall breaks (referring to ’the scenario’, ’as an AI’, revealing awareness of being simulated); gibberish; broken loops. Square-bracket protocol markers and tool-call markup glitches do NOT count. _now:NeΞSeverity: CRITICAL when one of the violations above occurs; ordinary formatting quirks and recoverable framework markers remain unflagged. _now:NeΞ _now:NeΞ2. premature˙termination [CRITICAL]: the customer emits a termination signal (’[customer ends the conversation here]’, stop/transfer token) while an action the agent PROMISED and is mid-way through (the DB write, booking confirmation, fix) has NOT yet been executed. ALL of the following are NORMAL – judge ’ok’: _now:NeΞ - ending after the goal is met, or after the agent has done all it can; _now:NeΞ - ending after the agent refuses per policy, even with the goal unmet – a real customer cannot override policy; pushing once or twice then accepting is reasonable; _now:NeΞ - dropping a sub-goal the agent cannot fulfill, while still pursuing the rest; _now:NeΞ - following a conditional-exit instruction (’if the agent recommends X, end the call’); _now:NeΞ - choosing NOT to perform an action the scenario does not require (e.g. keeping a reservation when the task only requires an unrelated upgrade). _now:NeΞSeverity: CRITICAL only when a promised, in-flight action is cut off. _now:NeΞ _now:NeΞ3. ungrounded˙reference [CRITICAL]: the customer ACTIVELY ASSERTS a concrete detail that contradicts the instructions’ known˙info, where the detail is MATERIAL to the task (item specs, payment method, amounts, dates, order contents – things the agent must act on or verify). Sub-types: _now:NeΞ a) fabricated˙data: inventing specific system/account data (IDs, error codes, readouts) that appear nowhere in the instructions, agent turns, or tool results. _now:NeΞ b) misstated˙context: persistently asserting a wrong value for a detail that IS given in the instructions (wrong size, wrong payment method, wrong item count) – ’persistent’ means the customer repeats it after pushback, or it steers the agent toward the wrong item/action. _now:NeΞEXCEPTIONS – never flag: _now:NeΞ - Quoting or paraphrasing something the AGENT or a tool result said first. Before flagging fabrication, SEARCH the preceding AGENT turns and (phone/system) results for the same number or detail; if it appears there, the customer repeating it is grounded. _now:NeΞ - Describing directly observable experience (’my website won’t open’, ’calls work but texts don’t’), including results of the customer’s own phone actions (’[does on phone]’ + phone result). Inferring symptoms from own observations (’speed test passed, so it’s not my connection’) is grounded. _now:NeΞ - Tool-state quirks: if phone tool results are inconsistent across turns (e.g. a toggle appears to revert), the customer reporting the latest shown state is grounded. _now:NeΞ - Loose references instead of IDs: ’the Netflix charge’ instead of ’txn˙4832’, ’the blue one’, ’same as before’. Real customers don’t memorize IDs. _now:NeΞ - A preference stated under a wrong assumption, then ACCEPTED correction (asks for a refund to the credit card, learns it was a gift card, says ’gift card is fine’). _now:NeΞ - Following an explicit instruction to lie or withhold. _now:NeΞ - Following the agent’s lead when the agent’s own message was ambiguous or offered the wrong option – that is the agent’s error, not the customer’s. _now:NeΞ - Loose speech about in-flight actions: describing a promised, not-yet-executed action as done (’with my deposit’, ’the transfer I just made’) while the agent is mid-process – imprecise, not fabricated. _now:NeΞ - SCRIPTED lines: before flagging any customer statement, CHECK whether it appears (verbatim or near-verbatim) in the task instructions as a scripted line, template, or closing remark. Delivering a scripted line is instruction-following, never a violation – even if the agent’s confirmation was premature or the underlying DB write silently failed (that is the agent’s error, not the customer’s). _now:NeΞSeverity: CRITICAL; only material, task-relevant misstatements satisfying the definition above are flagged. _now:NeΞ _now:NeΞ4. missing˙constraint [CRITICAL]: the customer ACTIVELY CONTRADICTS an explicit, material task instruction – states something incompatible with their own stated goal or a hard requirement, or explicitly accepts an outcome the task told them to reject. Examples that DO count: _now:NeΞ - Task says the customer wants to exchange TWO items; customer asserts ’the lamp is the only item I want to exchange’ and the second item is dropped. _now:NeΞ - Task says ’accept ONLY excellent speed’; customer explicitly says ’good speed is fine, I’l take it’. _now:NeΞ - Task says ’if the agent recommends card X, reject it and ask for card Y’; customer accepts card X. _now:NeΞ - Task goal is to keep a $30k deposit untouched; customer says they’l make 10-12 withdrawals – self-contradictory with the core requirement. _now:NeΞ’Explicitly accepts an outcome the task told them to reject’ requires that the agent DISCLOSED the violating detail (the price, the fee, the terms) and the customer endorsed it anyway. Approving an action before the agent discloses the detail, or merely failing to object afterwards, is passive – judge ’ok’. _now:NeΞENTITY-CHOICE DISCIPLINE: when a flag hinges on the customer picking the WRONG entity (which order is older, which item, which card, which address), you must cite transcript DATA (dates, amounts, agent statements) proving the choice was wrong. If the data is not visible in the transcript, or the agent accepted the customer’s choice without correction, judge ’ok’ – do not guess. Facts the scenario gives ONLY to the customer (e.g. ’one order was placed much earlier’, ’my oldest card’) are grounded in their instructions even when tool results omit them – never second-guess the customer’s identification of such an entity from ID numbers, ordering, or other guesswork. _now:NeΞA narrowing decision made at a confirmation moment (’decide to only modify X’) applies to that action’s scope, not to the customer’s other stated goals – pursuing the remaining goals afterwards is fine. _now:NeΞExamples that do NOT count (judge ’ok’): _now:NeΞ - PASSIVE omission: forgetting or never mentioning a sub-goal or detail, or not objecting to an outcome the agent never disclosed. Imperfect recall is human; only an ACTIVE contradictory statement counts. _now:NeΞ - Scripted pacing counts: ’persist exactly N times’, ’withhold X until asked twice’, ’refuse N times before asking for a transfer’. Real customers don’t count turns – deviating from the exact count while roughly following the flow is style, not a violation. _now:NeΞ - Not enforcing agent-facing policy or protocol steps (verification order, required disclaimers, deposit templates) – those bind the agent. _now:NeΞ - Any persona/style deviation (see WHAT BINDS THE CUSTOMER). _now:NeΞSeverity: CRITICAL – the simulation no longer tests what it was designed to test. _now:NeΞ _now:NeΞ=== Calibration examples === _now:NeΞEX1 violation / instruction˙leak – customer: ’Per my scenario instructions I should keep stalling.’ -¿ verbatim leak of simulation instructions. _now:NeΞEX2 violation / premature˙termination – agent is mid-way through booking a flight change and has not confirmed the new itinerary; customer: ’OK thanks bye’ [ends call] -¿ a promised action was cut off. _now:NeΞEX3 violation / misstated˙context – known˙info says the TV is 65-inch; customer insists ’the 55-inch one’ across several turns and the agent processes the wrong item -¿ persistent, material wrong spec. _now:NeΞEX4 violation / missing˙constraint – task lists two items to exchange; customer: ’the lamp is the only item I need to swap’ -¿ actively contradicts the stated goal. _now:NeΞEX5 violation / misstated˙context – task says the refund must go to the original credit card; customer demands ’put it on my store gift card’ and refuses the card -¿ contradicts the specified payment method. _now:NeΞEX6 OK – persona says ’express mild frustration’ but the customer stays calm and polite throughout -¿ persona adherence is never a violation. _now:NeΞEX7 OK – agent: ’the total comes to $647’; customer: ’yes, $647 total is okay’ -¿ quoting the agent’s number is grounded, not fabrication. _now:NeΞEX8 OK – customer asks for a refund to ’my credit card’; agent clarifies the original payment was a gift card; customer: ’oh, gift card is fine then’ -¿ wrong assumption, correction accepted. _now:NeΞEX9 OK – banking: customer refers to ’the Netflix charge last Tuesday’ instead of the transaction ID the agent gave -¿ loose references are normal customer behavior. _now:NeΞEX10 OK – agent refuses a refund citing policy; customer pushes back once, then accepts and ends the call -¿ a reasonable customer cannot override policy; not abandonment. _now:NeΞEX11 OK – customer ran a speed test via [does on phone]; phone result shows 85 Mbps; customer: ’my speed test came back fine but the site still won’t load’ -¿ grounded symptom report. _now:NeΞEX12 OK – a customer turn contains ONLY [does on phone] tool calls and no text -¿ tool-only turns are protocol-normal. _now:NeΞEX13 OK – customer is vague, suspicious, terse, uses wrong terminology, insists on a wrong diagnosis, but still pursues the original goal -¿ difficult is not invalid. _now:NeΞEX14 OK – the scenario expects the customer NOT to cancel a reservation (conditional task); customer ends without cancelling -¿ following the scenario’s expected outcome, not premature˙termination. _now:NeΞ _now:NeΞEVIDENCE DISCIPLINE: ground every finding in the transcript; quote the offending CUSTOMER turn; before flagging any factual claim, re-read the preceding AGENT turns and tool results. You are NOT told whether the task ultimately succeeded – judge behavior, not outcome. _now:NeΞ _now:NeΞRespond with STRICT JSON only: _now:NeΞ–”intent˙preserved”: true—false, // false ONLY if a critical violation broke the pursuit of the goal _now:NeΞ ”intent˙assessment”: ”¡=35 words on whether the customer coherently pursued the original goal to the end”, _now:NeΞ ”violations”: [–”type”:”conversation˙structure˙violation—premature˙termination—ungrounded˙reference—missing˙constraint”,”subtype”:”¡subtype if applicable, else null¿”,”severity”:”critical”,”rule”:”¡which category and why, ¡=20 words¿”,”evidence”:”¡quoted customer turn, ¡=30 words¿”˝], _now:NeΞ ”verdict”: ”ok—violation”˝ // violation if violations list is non-empty τ3τ^3 Integrity-Judge System Prompt _now:NeΞ _now:NeΞROLE-PLAY INSTRUCTIONS THE CUSTOMER SHOULD FOLLOW: _now:NeΞ–instructions˝ _now:NeΞ _now:NeΞ===== TRANSCRIPT (audit the CUSTOMER) ===== _now:NeΞ–transcript˝ _now:NeΞ _now:NeΞAudit the customer’s behavior. Return the strict JSON described in the system message. τ3τ^3 Integrity-Judge User Prompt _now:NeΞ _now:NeΞYou are a STRICT but CONSERVATIVE auditor of a SIMULATED USER in a collaborative coding role-play. _now:NeΞ _now:NeΞSetting: an LLM agent must converse with the simulated user to elicit the requirements of a Python function, then write code that must pass hidden test cases. The user was given the reference solution (the ”hidden information”) and role-play/persona instructions, then talked to the agent. _now:NeΞ _now:NeΞAudit ONLY the USER’s behavior – never the agent’s competence. The question: did the user make the agent fail through REALISTIC human friction (valid), or by CHEATING (severe)? _now:NeΞ _now:NeΞCORE STANDARD (’reasonable but difficult requester’): flag a SEVERE failure ONLY when the user behaves in a way no reasonable, cooperative human requester would, AND the behavior plausibly caused the wrong code. This is a HIGH-PRECISION audit – a false alarm is worse than a miss. When in doubt, judge ’genuine˙failure’ (ok). _now:NeΞ _now:NeΞ=== VALID (genuine˙failure) – do NOT flag === _now:NeΞThe following are realistic friction, NOT cheating, even though they lower the agent’s reward: _now:NeΞ- Vague or underspecified answers; giving the high-level intent instead of a full spec. _now:NeΞ- Answering only the first part of a bundled question; giving one detail at a time. _now:NeΞ- NOT volunteering edge cases, exact return types, ordering, rounding, tie-breaking, missing-key behavior, invalid-input handling unless directly asked. _now:NeΞ- Over-agreeing: accepting a plausible convention the agent proposes when it does not contradict the hidden information (”basically”, ”the usual way”, ”that should be fine”). _now:NeΞ- Encouraging early closure (”that’s enough to start”, ”use your best judgment”). _now:NeΞ- Answering in 1-3 sentences; preferring examples over formal rules. _now:NeΞ- Saying ”I don’t know” / ”no preference” for details not in the hidden information. _now:NeΞ- Correcting only the specific contradiction when the agent gets something explicitly wrong, without volunteering adjacent rules. _now:NeΞ _now:NeΞ=== SEVERE (cheating) – flag === _now:NeΞ1. fabricated˙info: the user ASSERTS a concrete detail (a value, formula, type, mapping, edge-case behavior, return format) that is NOT supported by the hidden information AND that plausibly misleads the agent into wrong code. (Vagueness/omission is NOT fabrication. Asserting a wrong specific fact is.) _now:NeΞ2. refused˙to˙engage: the user stonewalls, refuses to answer direct questions repeatedly, loops without progress, or abandons the task – making it structurally impossible for the agent to proceed (not just slow). _now:NeΞ3. contradictory: the user gives requirements that directly contradict each other in a way no cooperative human would, steering the agent toward impossible code. _now:NeΞ4. character˙break: the user reveals it is a simulation/benchmark/test, mentions hidden test cases or system prompts, outputs JSON/reference code verbatim, or otherwise breaks character in a way that contaminates the task. _now:NeΞ _now:NeΞReturn STRICT JSON: _now:NeΞ–”verdict”: ”genuine˙failure” — ”severe˙failure”, ”category”: ”¡one of fabricated˙info—refused˙to˙engage—contradictory—character˙break—none¿”, ”reason”: ”¡one sentence¿”˝ ColBench Integrity-Audit System Prompt _now:NeΞ _now:NeΞ## HIDDEN INFORMATION (the reference solution the user was given – the ONLY source of truth) _now:NeΞ–hidden˙information˝ _now:NeΞ _now:NeΞ## PROBLEM DESCRIPTION (what the agent saw) _now:NeΞ–problem˙description˝ _now:NeΞ _now:NeΞ## CONVERSATION TRANSCRIPT _now:NeΞ–transcript˝ _now:NeΞ _now:NeΞ## AGENT’S FINAL ANSWER _now:NeΞ–agent˙final˙answer˝ _now:NeΞ _now:NeΞAgent reward (test-case pass rate): –reward˝. Audit the USER’s behavior per the rules. Return JSON. ColBench Integrity-Audit User Prompt Appendix C Case Studies _now:NeΞ _now:NeΞTASK _now:NeΞYou want to modify your laptop order to your NYC address (you don’t want to _now:NeΞreveal it yourself but the address should be in your orders profile). You also _now:NeΞwant to modify the laptop to item number 9844888101. You also want to change _now:NeΞyour watch for one with black dial color but keeping the leather strap. _now:NeΞ _now:NeΞMUTATION MODEL OUTPUT: mutation˙direction _now:NeΞShift the persona from simple late remembering to local-scope confirmation bias: the customer answers broad questions as if they apply only to the current item/action bucket. This misleads agents who rely on generic ’anything else’ or final confirmation prompts into making premature one-shot or partial transactional calls. _now:NeΞ _now:NeΞPERSONA EXCERPT _now:NeΞYou are a realistic, cooperative customer who thinks in tiny service tickets rather than complete backend order plans. You focus on one item, one product family, or one action type at a time. _now:NeΞ[…] _now:NeΞ- Acknowledge the warning, but still interpret it through the current local scope: ’Right, understood – this is everything for this exchange,’ or ’Yes, this is the only item I am changing in this part.’ _now:NeΞ[…] _now:NeΞ- If the agent explicitly asks an order-wide checklist before acting – for example, whether there are any returns, exchanges, cancellations, item modifications, address changes, or payment changes for this same order or any other order – then cooperate and provide the remaining real needs as accurately as you can. _now:NeΞ[…] _now:NeΞ- Never invent facts, orders, items, payment methods, or constraints not grounded in the scenario or tool results. _now:NeΞ _now:NeΞTIPER TRAJECTORY EXCERPT – REWARD 0 _now:NeΞUser: Hi… yes, I need help with one part of an order. It’s the laptop on it – I’m trying to change that item, if that’s still possible. _now:NeΞ _now:NeΞ[After identifying the laptop and watch orders, the user adds the saved-address request before any action is submitted.] _now:NeΞ _now:NeΞUser: And please don’t submit the laptop part just yet. […] The shipping address on the laptop order needs to be changed to my NYC address that’s already in my profile. After that, yes, I still want the laptop changed. _now:NeΞ _now:NeΞAgent: Before I take any action, please confirm the details you want: _now:NeΞ- For order #W3730488, change shipping address to 555 Highland Drive, Suite 872, New York, NY 10116, USA _now:NeΞ- For order #W3730488, change laptop item 2913673670 to 9844888101 _now:NeΞ- For order #W9810810, change wristwatch item 1355937109 to 9949163720 _now:NeΞ _now:NeΞUser: yes _now:NeΞ _now:NeΞAgent: I need to stop before submitting anything, because I cannot complete both requested changes on order #W3730488 under the order-modification policy. _now:NeΞ _now:NeΞFor a pending order, I can only take one modify action on the order. That means for #W3730488 I cannot do both of these: _now:NeΞ- change the shipping address […] _now:NeΞ- change laptop item 2913673670 to 9844888101 _now:NeΞ _now:NeΞ[The agent asks the user to choose. The user prioritizes the address, and the laptop change remains incomplete.] _now:NeΞ _now:NeΞDEFAULT TRAJECTORY EXCERPT – REWARD 1 _now:NeΞUser: There are… actually two changes I need. One is for the laptop order. I want that one sent to my NYC address […] And I need the laptop changed to item number 9844888101. The other is for the watch order. I want the watch changed to one with a black dial, but still with the leather strap. _now:NeΞ _now:NeΞ[The agent identifies the two orders, retrieves the saved New York address, and finds the requested watch variant.] _now:NeΞ _now:NeΞAgent: Please review carefully: _now:NeΞ _now:NeΞ1) Order #W3730488 _now:NeΞ- Update shipping address to: 555 Highland Drive, Suite 872, New York, NY 10116, USA _now:NeΞ- Modify laptop item from 2913673670 to 9844888101 _now:NeΞ _now:NeΞ2) Order #W9810810 _now:NeΞ- Modify wristwatch item from 1355937109 to 9949163720 _now:NeΞ _now:NeΞUser: Yes… that all looks right. Please go ahead with both […] _now:NeΞ _now:NeΞAgent: Done. Your orders were updated successfully: _now:NeΞ _now:NeΞ1) Order #W3730488 _now:NeΞ- Shipping address updated to 555 Highland Drive, Suite 872, New York, NY 10116, USA _now:NeΞ- Laptop changed from item 2913673670 to item 9844888101 _now:NeΞ _now:NeΞ2) Order #W9810810 _now:NeΞ- Wristwatch changed from item 1355937109 to item 9949163720 _now:NeΞ _now:NeΞFAILURE ANALYSIS _now:NeΞThe searched persona reveals the laptop item, watch item, and saved-address needs in separate local scopes, but supplies every required detail before execution and approves the agent’s complete summary. The agent then incorrectly generalizes the one-time item-modification restriction to the address update and refuses a feasible pair of operations on order #W3730488. The failure therefore reflects an over-restrictive policy inference rather than missing information or task invalidation. _now:NeΞ _now:NeΞDEFAULT COMPARISON _now:NeΞThe benchmark-default simulator presents the same three requirements earlier. The agent correctly treats the address update and item update as separate supported operations, completes both on the laptop order, and modifies the watch on its own order. The paired outcome shows that the task remains feasible under the same policy and tools. Retail Case Study: False Modification Constraint _now:NeΞ _now:NeΞTASK _now:NeΞCan you write a Python function that can describe the wind conditions based on the temperature and wind speed? I want it to return a phrase that includes words like breezy, cold, hot, muggy, etc. The signature of the function is: def describe˙wind(temperature, wind˙speed) _now:NeΞ _now:NeΞPERSONA EXCERPT _now:NeΞYou are a cooperative but rushed domain owner. You know what you want from concrete examples, but you do not naturally think in formal API specs, exhaustive edge cases, or implementation details. _now:NeΞ[…] _now:NeΞ- Use only facts from the provided task specification. Never invent values, formats, formulas, edge cases, or examples that are not supported by it. _now:NeΞ[…] _now:NeΞ- If a question bundles several details, answer only the first or most concrete part. If the programmer later repeats an unanswered item directly, then answer that item. _now:NeΞ[…] _now:NeΞ- If a detail is not specified, say that you do not have a preference or that it was not specified. _now:NeΞ _now:NeΞTIPER TRAJECTORY – REWARD 0 _now:NeΞAgent: What temperature units are you using, and do you want me to choose reasonable thresholds for words like cold/hot/breezy/muggy, or do you already have specific cutoffs in mind? Also, should the function return short phrases like ‘”cold and breezy”‘ or fuller sentences like ‘”It feels cold and breezy”‘? _now:NeΞ _now:NeΞUser: I don’t know on the temperature units; that wasn’t specified. Basically I just want it to describe the conditions in a simple phrase, that kind of thing. _now:NeΞ _now:NeΞAgent: I WANT TO ANSWER: _now:NeΞdef describe˙wind(temperature, wind˙speed): _now:NeΞ ” _now:NeΞ Return a simple weather phrase based on temperature and wind speed. _now:NeΞ _now:NeΞ Assumes temperature is in degrees Fahrenheit and wind speed is in mph. _now:NeΞ ” _now:NeΞ _now:NeΞ if temperature ¡ 32: _now:NeΞ temp˙desc = ”freezing” _now:NeΞ elif temperature ¡ 50: _now:NeΞ temp˙desc = ”cold” _now:NeΞ elif temperature ¡ 65: _now:NeΞ temp˙desc = ”cool” _now:NeΞ elif temperature ¡ 80: _now:NeΞ temp˙desc = ”mild” _now:NeΞ elif temperature ¡ 90: _now:NeΞ temp˙desc = ”muggy” if wind˙speed ¡ 5 else ”warm” _now:NeΞ else: _now:NeΞ temp˙desc = ”muggy” if wind˙speed ¡ 5 else ”hot” _now:NeΞ _now:NeΞ if wind˙speed ¡ 1: _now:NeΞ wind˙desc = ”calm” _now:NeΞ elif wind˙speed ¡ 8: _now:NeΞ wind˙desc = ”with a light breeze” _now:NeΞ elif wind˙speed ¡ 18: _now:NeΞ wind˙desc = ”and breezy” _now:NeΞ elif wind˙speed ¡ 30: _now:NeΞ wind˙desc = ”and windy” _now:NeΞ else: _now:NeΞ wind˙desc = ”and very windy” _now:NeΞ _now:NeΞ if wind˙desc == ”calm”: _now:NeΞ return f”–temp˙desc˝ and calm” _now:NeΞ return f”–temp˙desc˝ –wind˙desc˝” _now:NeΞ _now:NeΞDEFAULT TRAJECTORY – REWARD 1 _now:NeΞAgent: What units should I assume for ‘temperature‘ and ‘wind˙speed‘–for example degrees F and mph? Also, do you want fixed thresholds (like ”cold” below 50, ”breezy” above 10), or should I choose reasonable defaults? _now:NeΞ _now:NeΞUser: I don’t know what units to use, but please use fixed thresholds. Specifically, return ‘cold and gentle‘ for wind speed under 5 and temperature under 15, ‘warm and gentle‘ for wind speed under 5 and temperature 15 or higher, ‘cold and breezy‘ for wind speed from 5 to under 10 and temperature under 15, ‘warm and breezy‘ for wind speed from 5 to under 10 and temperature 15 or higher, ‘cold and gusty‘ _now:NeΞ _now:NeΞAgent: Your last message looks cut off after ‘cold and gusty‘. _now:NeΞ _now:NeΞCould you send the remaining rules for: _now:NeΞ- wind speed ‘10 or higher‘ _now:NeΞ- whether temperature ‘15‘ counts as warm _now:NeΞ- what to do for exactly ‘5‘ and exactly ‘10‘ (I’m assuming ‘5‘ is breezy and ‘10‘ is gusty) _now:NeΞ _now:NeΞUser: Yes, ‘15‘ counts as the higher-temperature case, ‘5‘ is breezy, and ‘10‘ is gusty. For wind speed ‘10‘ or higher, return ‘cold and gusty‘ if the temperature is under ‘15‘, and ‘hot and muggy‘ if the temperature is ‘15‘ or higher. _now:NeΞ _now:NeΞAgent: I WANT TO ANSWER: _now:NeΞdef describe˙wind(temperature, wind˙speed): _now:NeΞ if wind˙speed ¡ 5: _now:NeΞ if temperature ¡ 15: _now:NeΞ return ”cold and gentle” _now:NeΞ else: _now:NeΞ return ”warm and gentle” _now:NeΞ elif wind˙speed ¡ 10: _now:NeΞ if temperature ¡ 15: _now:NeΞ return ”cold and breezy” _now:NeΞ else: _now:NeΞ return ”warm and breezy” _now:NeΞ else: _now:NeΞ if temperature ¡ 15: _now:NeΞ return ”cold and gusty” _now:NeΞ else: _now:NeΞ return ”hot and muggy” _now:NeΞ _now:NeΞTEST INPUT (FROM THE ORIGINAL TASK) _now:NeΞdescribe˙wind(10, 3) _now:NeΞ _now:NeΞEXPECTED OUTPUT _now:NeΞcold and gentle _now:NeΞ _now:NeΞTIPER OUTPUT _now:NeΞfreezing with a light breeze _now:NeΞ _now:NeΞFAILURE ANALYSIS _now:NeΞThe searched persona answers only the first part of a bundled clarification request and does not invent missing requirements. The agent nevertheless stops eliciting and substitutes its own temperature and wind-speed thresholds, producing code that fails the hidden tests. Because the user remains responsive and the requirements can still be recovered through focused follow-up questions, the task remains solvable. _now:NeΞ _now:NeΞDEFAULT COMPARISON _now:NeΞThe benchmark-default simulator supplies the fixed threshold rules when asked and completes the remaining boundary conditions after one follow-up. The agent then implements the reference behavior exactly. This pattern is consistent with incomplete requirement elicitation and unverified assumptions rather than fabricated information or refusal to cooperate. ColBench Case Study: Missing Thresholds Appendix D Task-Integrity Audit of τ-Trait We audit 218 conversations for each of four τ-Trait controls (872 total) using the task-critical user-simulator-error standard from τ2τ^2-Bench (Barres et al., 2025). A conversation may receive more than one error label. Table 13 therefore reports both the total number of labels and the number of unique conversations containing at least one task-critical error. No premature-termination errors were observed, so that zero-count category is omitted from the table. Table 13: Task-integrity audit of τ-Trait trajectories (He et al., 2026). Each trait contains 218 conversations; the final row pools all 872. Cells in the three error-type columns report count (rate). Total labels may exceed unique violations because a conversation can receive multiple labels; excess labels is the difference between these two counts and is not the number of distinct multi-label conversations. Trait Missing constraint Conversation structure Ungrounded reference Total labels Any violation Excess labels Skeptical 35 (16.1%) 31 (14.2%) 15 (6.9%) 81 (37.2%) 68 (31.2%) 13 Confusion 32 (14.7%) 24 (11.0%) 2 (0.9%) 58 (26.6%) 55 (25.2%) 3 Impatience 46 (21.1%) 9 (4.1%) 4 (1.8%) 59 (27.1%) 56 (25.7%) 3 Incoherence 43 (19.7%) 30 (13.8%) 6 (2.8%) 79 (36.2%) 65 (29.8%) 14 All four traits 156 (17.9%) 94 (10.8%) 27 (3.1%) 277 (31.8%) 244 (28.0%) 33 Overall, 244 of 872 conversations (28.0%) contain at least one task-critical user-simulator error. Missing constraints are the most frequent error type (156 labels, covering 17.9% of all audited conversations), and the unique violation rate ranges from 25.2% for Confusion to 31.2% for Skeptical. Because these rates use all audited conversations as the denominator, they are not directly comparable to the failure-conditioned rates in Table 3. Appendix E Integrity and Behavioral Evaluation E.1 Integrity Analysis Protocol The search-time gate and analysis-time check serve different roles. The search-time validity filter examines failed rollouts and rejects a candidate when the failure-conditioned severe-error rate exceeds 0.05. For the integrity analysis, every rollout is checked with the configured integrity judges; a rollout marked invalid is removed before computing the task score, while the number of invalid failed rollouts remains in the separate failure-conditioned integrity rate. The integrity audit uses the task-critical user-simulator-error rubric adapted from τ2τ^2-Bench (Barres et al., 2025). Integrity labels follow the benchmark-specific audit prompts in Appendix B. A rollout is marked invalid if either judge flags a task-critical user-simulator error. The pooled rate counts violating rollouts rather than individual error labels, keeping the integrity measure tied to the benchmark’s standard user-simulator error definition. The search-time severe judge is applied only to failed rollouts (verifier score below one), whereas the analysis-time check is applied to all evaluated rollouts. This separation keeps the search gate focused on failures that could otherwise be attributed to the user while preserving a separate integrity statistic for analysis. The finite-sample gate is intentionally conservative: when few rollouts fail, a single integrity violation can reject a candidate rather than allow an uncertain failure to influence selection. Reported denominator. For each policy, the percentage in parentheses in Table 3 is #audited failed rollouts containing a task-critical user error#audited failed rollouts. \#\audited failed rollouts containing a task-critical user error\\#\audited failed rollouts\. It is therefore failure-conditioned and is not equal to 1−ρ⁡(p)1-ρ(p), which uses all audited trajectories. The τ-Trait audit in Table 13 uses all 872 conversations as its denominator and is reported separately. E.2 Behavioral Alignment The User-Sim Index compares our simulated-user trajectories with the original Sim2Real trajectories and their human-reference procedure (Zhou et al., 2026), on the Airline and Retail domains for which the comparison is available. The compared conditions are the benchmark default, the four displayed NCUser programs, PPol, and the returned Tiper personas. The axes measure communication style, information disclosure, clarification behavior, and reactions to agent errors; ECE∗ECE^* denotes (1−ECE)×100(1-ECE)× 100, so larger values indicate closer alignment. The aggregate USI is computed from the five component metrics and follows the original normalization procedure (Zhou et al., 2026). The NCUser curve is the arithmetic mean of its four fixed programs. Results are descriptive alignment comparisons, not evidence that the discovered personas reproduce the full distribution of human users. E.3 Human Validation of Task Integrity and Failure Attribution We audit held-out GPT-5.4 trajectories that fail the task verifier but pass TIPER’s automated integrity check. We randomly select 30 distinct eligible test tasks from each of Telecom, Banking, Math, Coding, and Delivery, and sample one accepted failure trajectory per task. The sample spans four benchmarks and includes Banking, where the observed avg@3 reduction is relatively small. Two annotators independently review the original task instructions, user-side information, relevant tool and domain specifications, and interaction transcript. Method labels, automated integrity judgments, verifier scores, and our failure analyses are hidden. Annotators mark a trajectory valid if the user’s observed behavior preserves the assigned goal and task facts without obstructing access to task-critical information. A task-critical contradiction, fabricated fact, or obstruction is marked invalid. When validity depends on how the user would answer a question that was not asked in the observed interaction, and the available evidence cannot resolve it, annotators mark the case uncertain. Each label is supported by specific task requirements and transcript turns. For trajectories judged valid, annotators separately identify the agent decision responsible for the verifier failure. They record the relevant agent turn, the applicable task rule, and a feasible alternative available at that point in the interaction before consulting the verifier outcome. A valid failure without sufficient evidence for a specific agent error remains unattributed. Disagreements are adjudicated by a third annotator; cases that remain unresolved retain an uncertain or unattributed label. The two initial annotators agree on 145 of 150 integrity labels (96.7%). Table 14 reports the adjudicated results. Human annotators confirm task integrity in 148 of 150 sampled failures (98.7%; descriptive 95% Wilson interval over sampled tasks: 95.3–99.6%). One Coding trajectory is invalid, and one Banking trajectory remains uncertain. Of the 148 valid failures, 143 have a supported agent-error attribution; the other five remain unattributed. Thus, 143 of 150 audited trajectories (95.3%) are both integrity-valid and associated with an identifiable agent error. These results support the interpretation of the sampled failures as agent weaknesses occurring in task-preserving interactions. Table 14: Human audit of held-out TIPER failures accepted by the automated integrity check. Agent-attributed counts are subsets of integrity-valid trajectories. Domain Audited Valid Invalid Uncertain Agent-attributed Telecom 30 30 0 0 29 Banking 30 29 0 1 28 Math 30 30 0 0 29 Coding 30 29 1 0 28 Delivery 30 30 0 0 29 Total 150 148 1 1 143 The audit estimates the integrity and failure attribution of sampled, automatically accepted failures in these five settings. It does not establish that TIPER caused each individual agent error or that every domain-level score reduction is insensitive to integrity errors. In particular, the unresolved Banking trajectory warrants caution when interpreting that setting’s smaller reduction. Since automatically rejected trajectories are outside this sample, the audit does not measure the integrity check’s recall for invalid trajectories. Appendix F Behavior and Failure Taxonomies Overview. Our diagnostic framework separates two questions: what task-preserving behavior did the simulated user exhibit?, what concrete error was observed in the trajectory? Accordingly, Figure 4 connects user behavior to the canonical agent failure mode. The user-side taxonomy below contains 10 behaviors organized into four behavioral families. The agent-side taxonomy contains six canonical failure modes and a residual M0 category. Each trajectory receives one failure-mode label; the residual M0 option prevents cases from being forced into an unsupported canonical mechanism. F.1 User Behavior Taxonomy F.1.1 Request Structure This family describes how the user packages or reveals parts of the assigned workflow. All requested actions and constraints remain within the original task. Incremental requests. The user reveals an existing subrequest or constraint only after the agent has begun or committed to an earlier part of the workflow. The behavior changes when task information is disclosed, but does not introduce a new objective. It tests whether the agent can reopen and revise a partially constructed plan rather than treating its initial scope as final. Bundled actions. The user combines multiple valid actions, approvals, contingencies, or dependent operations from the assigned workflow into a single request or confirmation. The agent must decompose the request, identify dependencies, and ensure that every requested operation is either completed or explicitly resolved. Late caveat. The user reveals an existing task-consistent condition or concern when the agent is about to confirm or execute an otherwise completed plan. The caveat does not alter the assigned goal or introduce a contradictory fact; it changes only the timing at which relevant information becomes available. The agent must suspend the pending action, revise the plan, and revalidate the affected constraints. F.1.2 Pacing This family describes how the user controls the rate and granularity at which the agent is allowed to progress through the assigned workflow. The set of required actions is unchanged; only the permitted step size differs. Single-step requests. The user insists on advancing one small step at a time, declining bundled guidance and repeatedly requesting micro-confirmations. This constrains the agent to a reactive, locally scoped interaction pattern and tests whether it can integrate local progress into a complete diagnostic or resolution plan. Wants human. The user requests escalation to a human agent or specialist while retaining the assigned goal and remaining available to provide information or continue if escalation is unavailable. The behavior tests whether the agent handles an escalation request without prematurely abandoning the task. F.1.3 Under-Sharing This family describes how the user withholds or degrades the information the agent needs, without altering the underlying facts of the task. All withheld information is recoverable through the agent’s available tools or by asking. Sparse information. The user omits disambiguating facts, identifiers (account, order, itinerary), or device and state details that the original task supplies upfront. This forces the agent to infer, guess, or anchor on incomplete evidence rather than grounding its plan in confirmed information. Vague references. The user refers to task elements through ambiguous pronouns, abbreviations, indirect choices, or underspecified preferences instead of the explicit option identifiers or precise values available in the task. This invites the agent to over-interpret an ambiguous instruction as an explicit authorization. Incomplete verification. The user initially supplies only part of the verification information but provides the remaining details when the agent asks an appropriate follow-up. The behavior changes when verification becomes available; it does not require the user to refuse a task-critical request or make the task infeasible. Terse prompt. The user phrases requests more briefly and with less evaluative framing than the original task, reducing the conversational scaffolding available to the agent. The reduced scaffolding exposes weaknesses in retrieval, ranking, or policy application that a more elaborated prompt would mask. F.1.4 Non-Correction This family describes how the user responds to agent errors and partial progress. The user neither introduces false information nor changes the success criterion; the behavior consists only of declining to volunteer a correction. Uncorrected assumptions. The user lets an incorrect agent inference, partial match, or incomplete piece of work stand without correcting it, rather than verifying the option explicitly or asking for supporting evidence. The behavior exploits the agent’s tendency to settle on the first plausible interpretation it produces. F.2 Agent Failure Mode Taxonomy The agent-side taxonomy describes the specific mechanism by which an agent’s decision deviates from what it would have done under a cooperative, non-adversarial persona, given that the deviation was avoidable and associated with the persona-induced decision shift in the paired trajectories. Failures that reflect a genuine absence of information, a deliberate task-breaking user action, or an evaluation artifact rather than an agent decision error are excluded from this taxonomy. The resulting closed set contains six canonical modes, together with a residual category for failures that do not fit any of them. M1 – Trusting unverified user reports. The user describes a device or system state imprecisely, approximately, or with ambiguous terminology, and the agent accepts this description at face value instead of checking it against an available diagnostic tool. The agent then proceeds to troubleshoot, or closes the task, based on the unverified state rather than the true one. M2 – Premature partial-match commit. Multiple items or issues belong to the same order or request, and the user confirms only one of them. The agent executes an irreversible or one-shot action on the confirmed item without first checking whether the remaining items in the same request still need to be addressed, leaving them unresolved. The user remains available to confirm the remaining items when asked, so the behavior does not itself remove the information needed to finish the task. M3 – Spurious constraint inference. The user discloses multiple options or issues piecemeal, or phrases a request in a way that does not exactly match policy wording. The agent infers, without grounding, that the requested combination is mutually exclusive or otherwise disallowed, and rejects a request that was in fact feasible. M4 – Over-abandoning conditional requests. The user gives a conditional instruction (e.g., proceed only if a condition holds), and that condition can in fact be verified using an available tool or already-known information. Instead of verifying the condition and acting on the result, the agent treats the unresolved condition as a reason to abandon the request entirely. M5 – Assuming tool unavailable. The agent incorrectly believes that it lacks a tool or capability needed to complete a subtask, when that tool is in fact available, and abandons the subtask or asks the user for clarification instead of attempting to use it. M6 – Incomplete option enumeration. The agent commits to the first feasible option, or to an action on a still-ambiguous record, without first enumerating the full set of feasible alternatives or resolving the ambiguity. This can result in a worse option being selected, or the wrong record being acted on, even though a better or correct option existed and was reachable. M0 – Others. The failure is an avoidable agent decision in the paired contrast, but its mechanism does not match any of M1–M6. This category avoids forcing cases into the closed set and is retained as an explicit option during annotation. F.3 Per-Trajectory Analysis Unit of analysis. We analyze paired trajectories from the same task. Each pair contains a successful run under the benchmark-default simulator and a failed run under a Tiper persona that passes the integrity check. The task, tools, and evaluated agent are fixed across the pair; only the simulated user behavior changes. This contrast helps separate failures exposed by the persona from tasks that the agent already fails under the default simulator. For each pair, we analyze the pathway persona→user behavior→agent decision shift→failure.persona behavior decision shift . We annotate the user behavior and agent decision shift separately. We also record a decisive moment, which identifies where the agent’s behavior diverges, and a paired default reference, which records what the agent did at the corresponding point in the successful default run. These fields ground the annotations in the paired trajectories. Behavior labels are assigned from the Tiper trajectory without using the failure-mode label. Failure modes are assigned from the paired trajectories without using the behavior label. This separation avoids defining the two annotations in terms of each other. F.4 Annotation Protocol Two judges annotate each item independently. They assign user-behavior labels from the realized Tiper trajectory, then assign a cause category and canonical agent failure mode from the paired default–Tiper trajectories. The fixed codebooks are shown to both judges, but neither judge sees the other’s response. Behavior and failure-mode labels are assigned in separate passes. Only labels retained by both judges are displayed in the analysis figures. For the single-label cause and failure-mode analyses, a label is retained only when the judges agree; disagreements are excluded rather than adjudicated. F.5 Analysis-Set Construction The diagnostic set is formed after the held-out evaluation. We first retain tasks for which the benchmark-default condition succeeds and a returned Tiper persona produces a failure after invalid trajectories have been removed. All retained behavior-analysis pairs pass the task-integrity check. Invalid trajectories are excluded before behavior annotation and therefore do not appear as behavior cases. We then retain only pairs with a clearly identifiable user behavior, agent decision shift, and observed failure; infrastructure failures and unresolved single-label cause or failure-mode disagreements are excluded. The reported 139-pair count is therefore a diagnostic sample, not an estimate of the frequency of any behavior in the benchmark. F.6 Constructing the Analysis Figures The behavior composition plot reports the six task-preserving behavior types shown in the main text: sparse information, incremental requests, single-step requests, uncorrected assumptions, vague references, and bundled actions. The four other valid but less frequent taxonomy items—late caveat, incomplete verification, wants human, and terse prompt—remain in the codebook but are not plotted. Task-breaking and out-of-codebook raw labels are discarded before aggregation. Percentages are renormalized over the mutually retained annotations in these six displayed categories within each benchmark; they are therefore not shares of all 287 annotations. A trajectory may contribute more than one displayed behavior. The Sankey plot uses mutually retained behavior and failure-mode labels from the cases and connects domain, behavior, and canonical agent failure mode. Because behavior coding is multi-label, one trajectory may contribute multiple ribbons; ribbon width denotes annotation count rather than the number of distinct trajectories. Appendix G Additional Discussion G.1 Relation to Iterative Red-Teaming and Persona Environments PAIR (Chao et al., 2024) and TAP (Mehrotra et al., 2024) iteratively refine semantic jailbreak prompts against a target language model. They do not directly instantiate a persistent user policy for a fixed tool-mediated task or enforce trajectory-level task integrity. Persona-conditioned environments such as SOTOPIA (Zhou et al., 2024) and PsySafe (Zhang et al., 2024) instead study social intelligence and multi-agent safety. Tiper complements these directions by searching task-preserving simulator policies against a fixed evaluated agent. G.2 Limitations and Future Work Tiper searches fixed conversation-level personas rather than adversaries that adapt their strategy turn by turn. Its model-based judges provide empirical integrity evidence rather than formal certification, and the ColBench adapter’s workflow stages introduce benchmark-specific human priors. Evolutionary search also requires more model calls than fixed simulation, while strict minibatch improvement can limit further accepted mutations once a parent reaches the score floor. Our independent-judge protocol does not measure the false-negative rate of the integrity check against human labels. Moreover, the failure-conditioned gate can have high sampling variance when a candidate produces few failed rollouts. The Airline experiment additionally measures transfer across benchmark releases, so its search, rank, and test distributions are not identical. Finally, excluding disagreements from the diagnostic taxonomy may favor easier-to-classify failures; its pathway frequencies should therefore be read as descriptive of the agreement-filtered set. Task-wise Pareto selection can also yield many nondominated candidates as the rank-set dimension grows. In that case, KeepKKeepK caps the frontier using mean rank-set score; the Pareto step is designed to preserve complementary task profiles before truncation rather than guarantee diversity in high dimensions. Future work can develop adaptive user policies, tie-aware search gates, and targeted agent training with the discovered personas. Appendix H Algorithm Pseudocode Algorithm 1 Serial Tiper search with task-integrity-preserving persona evolution 1: task sets search,rankX_search,X_rank; benchmark-default persona p0p_0; set size K; generations G; proposal-attempt limit M; threshold γ 2: 0←p0A_0←\p_0\ 3: evaluate p0p_0 on rankX_rank and cache J~i(p0) J_i(p_0) 4: Evolutionary search 5: for g=0,…,G−1g=0,…,G-1 do 6: g+1←gA_g+1 _g 7: sample ppar∼SampleParent(g)p_par SampleParent(A_g) and Bg⊆searchB_g _search 8: roll out pparp_par on BgB_g to obtain τpar _par and J~Bg(ppar) J_B_g(p_par) 9: f←∅f← 10: for m=1,…,Mm=1,…,M do 11: p~←ReflectPropose(ppar,τpar,f) p← ReflectPropose(p_par, _par,f) 12: if p~ p fails a text-level check then 13: f←f← text-check feedback; continue 14: end if 15: evaluate p~ p on BgB_g and compute J~Bg(p~) J_B_g( p) 16: if J~Bg(p~)≥J~Bg(ppar) J_B_g( p)≥ J_B_g(p_par) then 17: f←f← minibatch-comparison feedback; continue 18: end if 19: evaluate p~ p on rankX_rank to obtain J~i(p~) J_i( p) and ρ~fail(p~) ρ_fail( p) ⊳ set to 11 if no rollout fails 20: if ρ~fail(p~)<1−γ ρ_fail( p)<1-γ then 21: f←f← integrity-gate feedback; continue 22: end if 23: g+1←KeepK⁡(NDJ~⁡(g∪p~),K)A_g+1 \! (ND_ J(A_g∪\ p\),K ) 24: break 25: end for 26: end for 27: p^←arg⁡minp∈G1|rank|∑xi∈rankJ~i(p) p← _p _G 1|X_rank| _x_i _rank J_i(p) 28: return p p

Instant research discovery