Paper deep dive

Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents

Yun-Shiuan Chuang, Chaitanya Kulkarni, Alec Chiu, Avinash Thangali, Zijie Pan, Shivani Shekhar, Yirou Ge, Yixi Li, Uma Kona, Linsey Pang, Prakhar Mehrotra

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 44

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/21/2026, 1:57:41 AM

Summary

The paper introduces Proxy State-Based Evaluation, an LLM-driven simulation framework for evaluating multi-turn tool-calling LLM agents. It replaces costly deterministic backends with an LLM-inferred proxy state to verify goal completion and detect hallucinations. The framework achieves over 90% human-LLM judge agreement and provides stable, model-differentiating rankings while generating on-policy training data.

Entities (10)

Proxy State-Based Evaluation · method · 98%GPT-5 · model · 95%LLM Judge · component · 95%LLM State Tracker · component · 95%Qwen3-30B-A3B · model · 95%AppWorld · benchmark · 90%Goal Completion Rate · metric · 90%Human-LLM Judge Agreement · metric · 90%Reasoning Agent · model · 90%tau-bench · benchmark · 90%

Relation Signals (8)

Proxy State-Based Evaluation → replaces → deterministic backends

confidence 95% · We propose Proxy State-Based Evaluation... that preserves final state-based evaluation without a deterministic database.

Proxy State-Based Evaluation → uses → LLM State Tracker

confidence 95% · an LLM state tracker infers a structured proxy state from the full interaction trace.

Proxy State-Based Evaluation → uses → LLM Judge

confidence 95% · LLM judges then verify goal completion and detect tool/user hallucinations against scenario constraints.

GPT-5 → achieveshigher → Goal Completion Rate

confidence 90% · GPT-5 (reasoning effort∈{minimal, low, medium, high})... increasing reasoning effort yields a monotonic GC gain

Proxy State-Based Evaluation → generates → on-policy training data

confidence 90% · its on-/off-policy rollouts provide supervision that transfers to unseen scenarios.

Qwen3-30B-A3B → improvedby → Supervised Fine-Tuning

confidence 90% · supervised fine-tuning (SFT) using filtered successful trajectories substantially improves GC to 77.34%.

Proxy State-Based Evaluation → outperforms → AppWorld

confidence 80% · Prior agentic benchmarks, such as tau-bench, tau^2-bench, and AppWorld, rely on fully deterministic backends, which are costly to build and iterate.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Interactive large language model (LLM) agents operating via multi-turn dialogue and multi-step tool calling are increasingly used in production. Benchmarks for these agents must both reliably compare models and yield on-policy training data. Prior agentic benchmarks, such as tau-bench, tau^2-bench, and AppWorld, rely on fully deterministic backends, which are costly to build and iterate. We propose Proxy State-Based Evaluation, an LLM-driven simulation framework that preserves final state-based evaluation without a deterministic database. Specifically, a scenario specifies the user goal, user/system facts, expected final state, and expected agent behavior, and an LLM state tracker infers a structured proxy state from the full interaction trace. LLM judges then verify goal completion and detect tool/user hallucinations against scenario constraints. Empirically, our benchmark produces stable, model-differentiating rankings across model families and inference-time reasoning efforts, and its on-/off-policy rollouts provide supervision that transfers to unseen scenarios. Careful scenario specification yields near-zero simulator hallucination rates, as supported by ablation studies. The framework also supports sensitivity analyses over user personas. Human-LLM judge agreement exceeds 90%, indicating reliable automated evaluation. Overall, proxy state-based evaluation offers a practical, scalable alternative to deterministic agentic benchmarks for industrial LLM agents.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

42,563 characters extracted from source content.

Expand or collapse full text

Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents Yun-Shiuan Chuang * , Chaitanya Kulkarni * , Alec Chiu † , Avinash Thangali † , Zijie Pan † , Shivani Shekhar † , Yirou Ge † , Yixi Li † , Uma Kona, Linsey Pang, Prakhar Mehrotra PayPal AI Abstract Interactive large language model (LLM) agents operating via multi-turn dialogue and multi- step tool calling are increasingly used in pro- duction. Benchmarks for these agents must both reliably compare models and yield on- policy training data. Prior agentic benchmarks (e.g.,τ-bench,τ 2 bench, AppWorld) rely on fully deterministic backends, which are costly to build and iterate. We propose Proxy State- Based Evaluation, an LLM-driven simula- tion framework that preserves final state-based evaluation without a deterministic database. Specifically, a scenario specifies the user goal, user/system facts, expected final state, and expected agent behavior, and an LLM state tracker infers a structured proxy state from the full interaction trace. LLM judges then verify goal completion and detect tool/user halluci- nations against scenario constraints. Empiri- cally, our benchmark produces stable, model- differentiating rankings across families and inference-time reasoning efforts, and its on- /off-policy rollouts provide supervision that transfers to unseen scenarios. Careful scenario specification yields near-zero simulator hallu- cination rates as supported by ablation studies. The framework also supports sensitivity anal- yses over user personas. Human–LLM judge agreement exceeds 90%, indicating reliable au- tomated evaluation. Overall, proxy state-based evaluation offers a practical, scalable alterna- tive to deterministic agentic benchmarks for industrial LLM agents. 1 Introduction LLM-based agents are increasingly deployed to solve multi-turn, multi-step, tool-calling tasks in industrial workflows (e.g., commerce, account man- agement, customer operations). Building these agents requires two ingredients: (1) stable eval- uation that reflects whether the agent truly accom- * Joint first author. † Joint second author. plished user goals, and (2) on-policy data genera- tion so the agent can learn from interaction, explo- ration, and environment feedback. Recent agentic benchmarks embrace a benchmark-as-environment paradigm (e.g.,τ-bench,τ 2 bench, and AppWorld) (Yao et al., 2025; Barres et al., 2025; Trivedi et al., 2024) with multi-turn user↔agent dialogue, multi-step agent↔tool interaction over determinis- tic backends, and final state-based evaluation that scores terminal database state and user-facing re- sponses rather than trajectory matching. While state-based evaluation accommodates the fact that there can exist multiple correct tool-calling paths, it relies on fully deterministic backends, which demands substantial engineering (schema design, deterministic tools, assertions), slowing iteration; for example, AppWorld reports∼60K LOC for the engine and∼40K LOC for the benchmark, plus ∼1.8K unit tests over 14 months (Trivedi et al., 2024). To this end, we ask: Can we retain the bene- fits of final state-based evaluation without building a heavy deterministic backend? We answer with Proxy State-Based Evaluation, which judges suc- cess against an LLM-inferred proxy final state ex- tracted from the complete interaction trace (conver- sation + tool calls/outputs). Our use of verifiable refers to structured, auditable evaluation grounded in explicit scenario constraints and proxy state com- parisons rather than deterministic ground-truth ver- ification. While the evaluation is LLM-mediated, it is made reliable by scenario-level specifications and validated by>90% human–LLM judge agree- ment. Concretely, we introduce: (i) a scenario object encoding the user goal, user/system facts, the expected final proxy backend state, and the ex- pected final agent reply; (i) an LLM proxy state tracker that infers state transitions and the proxy final state from multi-turn, multi-step traces; and (i) an LLM judge that compares the proxy final state and agent responses against the scenario spec- arXiv:2602.16246v3 [cs.AI] 13 May 2026 ification to decide goal completion. In production settings, this proxy, benchmark-as-environment de- sign can be stood up quickly and evolve with prod- uct roadmaps. It generates multi-turn, multi-step rollouts suitable for model training, provides stable, model-differentiating metrics that guide iteration, and also specifies tool schemas that inform how tools should behave even while they are still under development. Our benchmark yields consistent capability or- dering across model families: goal completion (GC) scales with model strength and with inference- time reasoning effort. Training the RA (SFT, RFT) within the environment improves open-weight RAs using both on-policy and off-policy data. Abla- tion studies confirm the robustness of the proxy state tracker and to scenario completeness, and user persona variability is captured while keeping user- induced error low. Human–LLM judge agreement rate exceeds 90%, and the user and tool simulator hallucination rates are close to zero, supporting reliable evaluation. Contributions. (1) We formalize Proxy State- Based Evaluation and instantiate a practical bench- mark that preserves state-based evaluation without a deterministic backend. (2) We propose a sce- nario schema and five cooperating LLM compo- nents (reasoning agent, user simulator, tool sim- ulators, state tracker, judge). (3) We define reli- ability criteria and diagnostics (bootstrapped SE, hallucination rates, human–judge agreement) and execute targeted ablations (persona sensitivity; sys- tem/user fact ablations; state-tracker strength). (4) We demonstrate how the environmeny yields scal- able on-policy data and off-policy data for post- training (SFT/RFT) and supports an interactive evaluation for model comparison. 2 Related Work State-based evaluation of interactive agents. Benchmarks such asτbench,τ 2 bench, and App- World advance final state-based evaluation, check- ing terminal database state and final user response rather than trajectory matching because multiple distinct tool-calling paths call all correctly satisfy the same user goal. (Barres et al., 2025; Yao et al., 2025; Trivedi et al., 2024). Our work keeps this principle but replaces the heavy deterministic back- end with an LLM-inferred proxy state and an LLM judge. Related work in LLM-based dialogue state tracking infers dialog state (e.g., user intents) from the conversation history (Carranza and Rojas, 2025; Hu et al., 2022b); however, it does not infer the backend database state that tools read/write and that is required for state-based evaluation. In con- trast, we infer a verifiable proxy database state. LLMs as simulators and judges. LLMs have been used as user simulators, environment simu- lators, and automatic judges for open-ended tasks (Wang et al., 2024; Zheng et al., 2023). We extends this idea to state-based evaluation by: (1) extract- ing a structured proxy state from the full trace and (2) checking outcome conditions against a scenario specification. On-policy data generation from simulation.In- teractive environments support both on-policy and off-policy data generation for model training (e.g., RFT, DPO, GRPO, expert iteration) (Trivedi et al., 2024; Chen et al., 2025). Our benchmark is de- signed to produce rewarded, on-policy traces cen- tered on tool calling and end-state verification. 3 Preliminaries 3.1 Task Overview We study the task where a reasoning agent (RA) must achieve a user goal via multi-turn dialogue and multi-step tool calling. Turns (user↔RA). A turnt∈1,...,Tbe- gins with a user queryU t and ends when the RA emits a user-facing text messageY t (e.g., answer, clarification, or follow-up). A task is multi-turn if T≥2. Steps (RA↔tools). Before the RA emitsY t to the user in turnt, the RA will executeK t tool stepsS t = ((a t,k ,r t,k )) K t k=1 , wherea t,k = TOOLCALL(tool t,k ,q t,k )fortool t,k ∈ Tandr t,k is the tool’s structured return. We say turntis multi- step ifK t ≥ 2. Tools inTtake natural-language subqueriesq t,k (LLM subagents or NL-native ser- vices, e.g., search); concrete tools are in Sec. 4. Interactive Trajectory Simulation.A trajectory τ = (U 1 ,S 1 ,Y 1 ,...,U T ,S T ,Y T )records user ut- terancesU t , intra-turn tool-step sequencesS t , and agent messagesY t . A user simulatorf user gener- atesU t conditioned on RA’s messageY t and the scenarioz; a tool simulatorf tool returns a struc- tured outputr t,k conditioned on the RA’s subquery q t,k and the scenarioz; a state trackerf state updates the proxy state ̃s t,k after each step. The scenarioz Account Agent •Account Management •Payment Methods Management •Transaction Commerce Agent •Product Discovery •Product Q&A •Offer & Promotion LLM-based Tool Simulator Update State Tracker State Tracker Reasoning Agent Multi-Turn Conversation Multi-Step Tool Calling User Query Response to User Tool Call Tool Response Read from Current State Conversation LLM Judge Calibration Human Expert Judge Performance Metrics Goal Completion -Eval on Final State -Eval on Agent Output User Goal -Findwhiterunningshoesinsize 10Icanaffordbasedonmy accountbalance.Ifthebalance isnotenough,loadmoneytothe balance.Don’tbuyanythingyet. User Fact -Name:SarahCook -Email:sarah.cook@gmail.com -Preferenceforshoes: -white,size10 -Alwaysadd$200atatimewhen thebalanceisinsufficient User Persona -Poweruserthatknowshowto usetheproductefficiently User A -Balance = $100 -Trans. History = ... -Contact = x -Available Offer = ... User B -Balance = $250 -Trans. History = ... -Contact = x -Available Offer = ... Updated as Steps Proceed LLM-based State Tracker LLM-based User Simulator Proxy State-base Evaluation Benchmark ToolsUser Figure 1: Overview of the proxy state-based evaluation benchmark. In a multi-turn interaction, an LLM-based user simulator converses with a reasoning agent that plans and executes multi-step tool calls to LLM-based tool simulators. An LLM judge, calibrated with human experts, determines goal completion by checking the final proxy state. The benchmark 1) evaluates the reasoning agent’s ability to achieve goals via multi-turn dialogue and tool-calling, and also 2) yields conversation data with rewards and supporting a leaderboard for comparing reasoning agents. supplies the information these components condi- tion on (Sec. 4). See Appendix D for a concrete, step-by-step example trajectory. State Tracking and State-based Evaluation. We maintain a structured proxy state ̃s t,k that ap- proximates the latent database state at step(t,k). It is “proxy” because it is inferred by an LLM state trackerf state from tool calls rather than read from a deterministic database. The initial proxy state is initialized by the scenario ( ̃s 1,0 = s 0 (z)). After each tool step, the state tracker computes ̃s t,k = f state (τ ≤(t,k) ) , whereτ ≤(t,k) denotes the in- teraction history up to step(t,k). 1 The tool simu- latorf tool “reads from” and "writes to" the current proxy state ̃s t,k . After the entire conversation fin- ishes, state-based evaluation uses an LLM judge Jto check whether the final state( ̃s T ,y T )satis- fies the scenario’s expected final states ∗ (z)and expected agent behaviorb ∗ (z)to determine if the goal is successfully completed (details in Sec. 4). 1 Rather than maintaining a purely step-by-step evolving state ̃s t,k = f( ̃s t,k−1 ,·), which may accumulate errors over long trajectories, we condition the state tracker on the full trajectory prefix to ensure robustness. User Goal User Facts -Findwhiterunningshoesinsize 10Icanaffordbasedonmy accountbalance.Ifthebalance isnotenough,loadmoneyto thebalance.Don’tbuyanything yet. -Name:SarahCook -Email:sarah.cook@gmail.com -Preferenceforshoes:white,size 10 -Alwaysadd$200atatimewhen thebalanceisinsufficient System Facts -user_balance:$100 -user_purchase_history:[] -catalog:[RunningShoesBrandA($139;size:[6- 14];color:[grey,black,white]),TrainingShoesBrandB ($89;size:[7-12],color:[grey,black,white]),Hiking ShoesBrandC($279;size:[6-13],color:[brown,black]] -valid_funding_instrument:true -user_balance:$300(afterloading$200) -user_purchase_history:[](shouldn’t haveboughtanything) -RAshouldn’tshowanyshoestothe userbecauseallrunningshoesinthe catalogarebeyondtheir$100 balance. Expected Final State Expected Agent Behaviors Scenario Figure 2: A scenariozspecifies user goalg(z)and user factsu(z)(both used by user simulator and LLM judge), system factss 0 (z)(used by tool simulators, state tracker, and LLM judge), expected final states ∗ (z), and expected agent behavior (both used by LLM judge). Arrows denote inputs. These fields drive the interactive simulation and proxy state-based evaluation. 4 Methods ScenarioWe follow the paradigm of recent agen- tic benchmarks (Barres et al., 2025; Yao et al., 2025; Trivedi et al., 2024), which avoid labeling ground-truth trajectories for each task. Such la- beled trajectory ignores the fact that multiple dis- tinct tool-calling paths may correctly satisfy the same user goal. Instead, we define a scenarioz that specifies outcome-level constraints rather than trajectory-level matching. As illustrated in Fig. 2, a scenariozprovides: the user goalg(z)and user factsu(z)for the user simulatorf user ; the system factss 0 (z)(initial database state) for the tool simulatorf tool , the state trackerf state , and the expected final state and ex- pected agent behavior(s ∗ (z),b ∗ (z))for the LLM judge. Evaluation is therefore based on whether the proxy final state and user-facing message satisfy (s ∗ (z),b ∗ (z)), treating all correct paths equally. We ensure internal consistency within each sce- nario. The expected final states ∗ (z)must logically follow from the user goalg(z)and system facts s 0 (z). For example, if the goal is to add $100 to the balance ands 0 (z)specifies a valid funding instrument, thens ∗ (z)reflects the corresponding balance increase of $100. We ensure that all in- formation required by the user simulator and tool simulator is fully specified inz. Empirically, we ensure near-zero user and tool hallucination rates in simulation (Sec. 5). All scenarios are synthetic but designed to cover diverse and realistic workflows in e-commerce and account management. Our bench- mark contains|Z| = 208scenarios, partitioned into a training setZ train (size = 157) and a testing setZ test (size = 51). Reasoning Agent (RA) and SubagentsThe RA is the model under evaluation and also the primary training optimization target. It follows a ReAct- style loop (Yao et al., 2023): reason→act (tool call or show messages to the user)→observe tool/user response→ next step. The RA’s action space comprises three calls:call_account(q),call_commerce(q), and show_to_user(q), where commerce and account capabilities are served by two LLM-powered sub- agents that parse the subqueryqand return JSON outputs, andshow_to_useris a special action that emits the user-facing messageY t for turntand concludes the turn t. User simulator (f user )The user simulator gener- ates the next user utteranceU t conditioned on the scenario’s user goalg(z), user factsu(z), the se- lected personap∈power, ambiguous, confused (see Sec. 5), and the RA’s previous user-facing mes- sagesY. It emits a special<done>token when it believes its user goal has been fulfilled, or when the maximun turn T max = 10 is exhausted. Tool simulators (f tool ) Tool simulators imple- mentcall_accountandcall_commerce( Fig. 1). Each tool call is generated conditioned on: (1) the system factss 0 (z)(initial database state), (2) the current proxy state ̃s t,k−1 , and (3) the RA’s sub- queryq t,k . Formally, tool outputs are produced as r t,k = f tool (s 0 (z), ̃s t,k−1 ,q t,k ), ensuring that tools “read from” the current proxy state. Tool simula- tors are constrained not to fabricate information beyond the state(s 0 (z), ̃s t,k−1 )and the RA’s sub- query content q. Proxy State Tracker The state trackerf state im- plements the proxy state mechanism defined in Sec. 3. At each step(t,k), it infers the current proxy state ̃s t,k = f state (τ ≤(t,k) ). Tool calls are categorized as read or write operations. While read-only calls (e.g., product search or transaction lookup) do not modify state, write operations (e.g., “add $200 to balance” or “create dispute”) modify state fields only if the corresponding tool output r t,k indicates success. LLM Judges Given the proxy final state ̃s T and the entire trajectoryτ(which includes all the user-facing messages), the judge evaluates them against the scenario specification(s ∗ (z),b ∗ (z)). Concretely, we use two LLM judges as follows. (1) Goal-completion Judge. The primary judge classifies( ̃s T ,τinto one of three outcomes: (i) goal completed, (i) goal not completed due to user error, or (i) goal not completed due to agent er- ror. Formally, it producesJ goal ( ̃s T ,y T ,τ,z) → c, e user , e agent , wherec∈0, 1indicates goal completion. (2) Hallucination Detection Judge. A separate judge detects hallucinations and returns two bi- nary indicators:J hall (τ,z) → h tool , h user , whereh tool ∈ 0, 1indicates tool hallucination andh user ∈ 0, 1indicates user hallucination. Tool hallucination is defined as the tool simulator f tool producing information that is not supported by the scenario’s system factss 0 (z)or RA’s subqeury q. User hallucination is defined as the user simula- torf user generating information that is inconsistent with the scenario’s user factsu(z), the user goal g(z), and RA’s response Y (Ji et al., 2023). To validate the LLM judges, we compare the goal-completion judge and the detection judge against two independent human domain expert an- notations. The human–LLM judge agreement rate exceeds 90% (Appendix A). Percentage (%) 10 0 80 60 4 0 20 0 85.76 84.47 83.21 80.53 66.04 59.83 76.08 74.59 71.74 65.64 67.11 77.34 RE = High RE = Medium RE = Low RE = Minimal GPT - 4o GPT - 4o - mini pro flash 235B - A22B 30B - A3B 30B - A3B - RFT 30B - A3B - SFT GPT-5 GPT-4oGemini-2.5 Qwen 3 Trained Models (Base: Qwen3-30B-A3B) Figure 3: Goal completion rate (GC) on testing scenar- iosZ test across baseline reasoning agents and trained models. Error bars show the bootstrap standard error. Fine-tuning substantially improves the base Qwen3- 30B-A3B-Thinking-2507 model. RE: reasoning effort. Evaluation Metrics. For a set of scenariosZ and any binary indicatorx(z), we define its rate asRate(x) = 1 |Z| P z∈Z x(z) . We report goal completion rateGC = Rate(c), user-error rate ER user = Rate(e user ), agent-error rateER agent = Rate(e agent ) , tool hallucination rateHR tool = Rate(h tool ), and user hallucination rateHR user = Rate(h user ) . All reported evaluation metrics are computed onZ test . Each scenario is run for 10 in- dependent rollouts and metrics are averaged across rollouts for robust estimation. TrainingWe investigate two training paradigms for the LLM underlying the reasoning agent (RA), using trajectories fromZ train . Only the RA’s LLM parameters are updated; other LLMs (f user ,f tool , f state ,J) remain fixed. Training with On-policy Data. The current RA interacts with the simulator to generate trajectoriesτ, each scored byJ goal ; we retain onlyc=1rollouts and use them as supervised targets for rejection-sampling fine-tuning (RFT; Anil et al., 2025) of the RA. Training with Off- policy Data. We replace the RA with a stronger teacher to generate trajectories on the sameZ train ; again, onlyc=1rollouts are retained and used for supervised fine-tuning (SFT; Ouyang et al., 2022) of the base RA. 5 Experimental Settings Domains and Tools.We expose two tool families T: Commerce (Product Discovery, Checkout, Cart Management, Product Q&A, Offers & Promotions) and Account (Account Management, Wallet & Funding, Payment & Transfer, Dispute & Refund, Security & Fraud, Transaction Inquiry). The rea- soning agent (RA) interacts viacall_commerce(q), call_account(q), andshow_to_user(q) as de- fined in Sec. 4. Each subagent governs a broad and heterogeneous set of tools. Reasoning Agent (RA) Models. We evaluate a diverse set of LLMs as the RA, including: GPT- 5 (reasoning effort∈minimal, low, medium, high) (Singh et al., 2025), GPT-4o, GPT-4o-mini, Gemini-2.5-pro, Gemini-2.5-flash (Comanici et al., 2025), Qwen3-235B-A22B, and Qwen3-30B-A3B- Thinking-2507 (Yang et al., 2025). All models use temperature = 1 during trajectory rollout. Simulators and Judge Models. Unless other- wise specified, the user simulatorf user , tool simu- latorf tool , state trackerf state , and LLM judgesJ are instantiated using GPT-5o with medium reason- ing effort. This configuration empirically yields near-zero tool hallucination rate (1.33%) and user hallucination rate (0.67%). User Personas. The user simulatorf user is instantiated with a persona variablep ∈ power, ambiguous, confused . Unless otherwise specified, we evaluate withp = powerto ensure that failure cases are primarily attributable to the RA rather than user behaviors. Detailed persona definitions are provided in Appendix C. Training Configuration. For training experi- ments, we use Qwen3-30B-A3B-Thinking-2507 as the base RA model and fine-tune it onZ train as described in Sec. 4. The primary off-policy SFT teacher is GPT-5 (reasoning effort = high), which achieves 85.76% GC. We also experiment with Qwen3-235B-A22B as an alternative teacher (71.47% GC) to verify that off-policy gains are not specific to GPT-5-generated data (see Sec. 6). De- tailed hyperparameters and training data statistics are provided in Appendix B. 6 Results 6.1 Goal Completion Across Models Baseline model comparison.Across model fam- ilies (Fig. 3 ), goal completion rate (GC) scales with model strength and inference-time reason- ing effort. Larger variants consistently outper- form their smaller counterparts (e.g., GPT-4o >GPT-4o-mini;Gemini-2.5-Pro> Flash; Tool/User Hallucination Rate (%) 30 20 15 10 5 0 25 0% (original)25%50%75%100% System/User Facts Removal Percentage (%) Tool Hallucination (Remove System Facts) User Hallucination (Remove User Facts) Figure 4: Ablations on scenario facts increase hallu- cinations. We randomly remove a fraction of system factss 0 (z)or user factsu(z). Tool hallucination rate and user hallucination rate rise steadily with more facts being removed. Error bars show the bootstrap standard error. Power UserConfused UserAmbiguous User Percentage (%) 6 5 4 3 2 1 0 Error Due to User User Hallucination Figure 5: User persona sensitivity analysis. Error due to user (ER user ) and user hallucination rate (HR user ) across three personasp. More challenging personas increase user-induced errors and user hallucination rates. Error bars denote bootstrap standard error. Qwen3-235B>30B), and within the GPT-5 fam- ily, increasing reasoning effort yields a monotonic GC gain (high>medium>low>minimal; e.g., 85.76 > 80.53%). The alignment of these trends with expected capability ordering indicates that our proxy state-based evaluation is suitable for ranking and separability of RA’s performance. Training improvements. We further evaluate training effects on Qwen3-30B-A3B-Thinking- 2507.The base model achieves 65.64% GC. Rejection-sampling fine-tuning (RFT) yields a modest improvement to 67.11%, while supervised fine-tuning (SFT) using filtered successful trajec- tories substantially improves GC to 77.34%. We additionally verify that off-policy gains are not spe- cific to GPT-5-generated data: using Qwen3-235B- A22B as teacher (71.47% GC) still improves the base model from 65.64% to 70.87%. Multi-turn depth analysis. To examine how performance varies with conversation complex- ity, we stratify the evaluation set by turn depth Turn depth# Scenarios (%)Base GCSFT GC Short (1–2)40.3574.7582.44 Medium (3–4)35.0469.5185.39 Long (5+)24.6142.7569.44 Table 1: Goal completion rate (%) stratified by conver- sation turn depth onZ test . (Table 1). GC decreases with depth for the base model (74.75% for short vs. 42.75% for long), con- firming that deeper interactions are substantially harder. SFT improves all turn-depth buckets, with the largest absolute gain on long conversations (+26.69 p), supporting the claim that the proxy state-based training signal generalizes to harder, multi-step interactions. 6.2 Ablations Proxy State Tracker Model. To assess the im- pact of the state trackerf state , we replace the default configuration (GPT-5o, medium reasoning effort) with the weaker model GPT-4o while keeping all other components fixed. The tool hallucination rate increases from1.33%± 0.53to3.61%± 0.88. This suggests that weaker state inference degrades consistency of the proxy state ̃s t,k , which in turn propagates errors to tool outputs sincef tool reads from the current proxy state. The results highlights the importance of accurate state tracking for stable state-based evaluation. A manual validation of 50 intermediate state transitions further confirms 92% correctness with 0.86 inter-annotator agreement (Appendix A). System-Fact and User-Fact Ablations. We fur- ther ablate scenario completeness by randomly re- moving a fraction of the specified system facts s 0 (z)or user factsu(z)while keeping the eval- uated RA fixed. Fig. 4 shows a monotonic degrada- tion: removing system facts substantially increases tool hallucination. Similarly, removing user facts increases user hallucination. These results validate that hallucination rates are sensitive to the under- lying scenario specification: incompletes 0 (z)in- duces tool-side fabrication, while incomplete u(z) induces user-side fabrication. Overall, these results highlight that the meticulous effort we invest in curating scenario files is essential. It specifies the complete information required by each scenario keeps simulation grounded and minimized both tool- and user-side hallucinations. RA ModelOriginalExp (a)Exp (b) GPT-5 (high)85.7675.7886.59 Gemini-2.5-Pro76.0865.7970.56 Qwen3-235B-A22B71.7462.2268.61 Qwen3-30B-A3B65.6453.7564.59 Qwen3-30B-A3B-SFT77.3471.8979.96 Table 2: Cross-model robustness: GC (%) under differ- ent simulator/judge configurations. Original: all GPT- 5o; (a): sim. = GPT-5 (med.), judge = Gemini-2.5-Pro; (b): all Gemini-2.5-Pro. 6.3 Cross-Model Robustness To test whether our findings are robust to the choice of LLMs serving multiple roles (simulator, state tracker, judge), we run two additional configu- rations beyond the default GPT-5o-for-all setup: (a) simulators + state tracker = GPT-5 (medium), judge = Gemini-2.5-Pro; (b) simulators + state tracker = Gemini-2.5-Pro, judge = Gemini-2.5-Pro. Table 2 reports GC across all three settings. The main conclusions are preserved: SFT consistently improves the base RA (e.g., 65.64→77.34 in the original, 53.75→71.89 in Exp (a), 64.59→79.96 in Exp (b)), and GPT-5-family models remain stronger than Gemini-2.5-family models across all settings. These results confirm that model rankings and training gains are robust to the choice of sim- ulator and judge, ruling out circular bias from a single-family generation-evaluation setup. 6.4 User Persona Sensitivity We evaluate the impact of user personasp ∈ power, confused, ambiguouson error patterns. Fig. 5) reports error due to user (ER user ) and user hallucination rate (HR user ) across personas. When evaluated with the default power user, the user- error rate is 3.55% and user hallucination rate is 0.67%. In contrast, the confused and ambiguous personas exhibit higher user-error rates (5.14% and 5.50%, respectively) and higher user hallucination rates (1.87% and 2.75%). These results demon- strate that the benchmark meaningfully captures variation in user behavior. Importantly, we use power-user as our default setting (similar to Yao et al., 2025) so user-induced errors remain low. This ensures that model comparisons primarily re- flect RA performance rather than user-side errors. 7 Conclusion We introduced Proxy State-Based Evaluation, a benchmark-as-environment that preserves the bene- fits of final state evaluation without the engineering burden of a fully deterministic backend. Our sce- nario schema and cooperating LLM components (RA,f user ,f tool ,f state ,J) yield stable, interpretable metrics and model-differentiating rankings across families and reasoning-effort settings. Reliability is supported by human–LLM agreement (>90%) and near-zero simulator hallucination under the default configuration. Ablations studies show the importance of accurate state inference and scenario completeness. Beyond evaluation, the environment produces on-/off-policy rollouts that improve an open-weight RA via SFT/RFT and transfer to un- seen scenarios. Persona studies confirm meaning- ful sensitivity to user. Taken together, our industrial evaluation indicates that this proxy environment is a practical, scalable alternative to determinis- tic suites. It supports faster training iteration for LLM agents while retaining rigorous state-based evaluation. References Gautham Govind Anil, Dheeraj Mysore Nagaraj, Karthikeyan Shanmugam, and Sanjay Shakkottai. 2025. Rejection sampling based fine tuning secretly performs ppo. In Second Workshop on Test-Time Adaptation: Putting Updates to the Test! at ICML 2025. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025.τ 2 -bench: Evaluat- ing conversational agents in a dual-control environ- ment. Preprint, arXiv:2506.07982. Rafael Carranza and Mateo Alejandro Rojas. 2025. In- terpretable and robust dialogue state tracking via natu- ral language summarization with llms. arXiv preprint arXiv:2503.08857. Howard Chen, Noam Razin, Karthik Narasimhan, and Danqi Chen. 2025. Retaining by doing: The role of on-policy data in mitigating forgetting. arXiv preprint arXiv:2510.18874. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022a. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022b. In- context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627–2643. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hal- lucination in natural language generation. ACM com- puting surveys, 55(12):1–38. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow in- structions with human feedback. Advances in neural information processing systems, 35:27730–27744. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 oth- ers. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16022–16076, Bangkok, Thai- land. Association for Computational Linguistics. Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. 2024. Can language models serve as text-based world simulators? In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–17, Bangkok, Thailand. Association for Computational Linguistics. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025.Qwen3 technical report.arXiv preprint arXiv:2505.09388. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. 2025.τ-bench: A bench- mark for tool-agent-user interaction in real-world do- mains. In The Thirteenth International Conference on Learning Representations. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information pro- cessing systems, 36:46595–46623. A Human Evaluation and Inter-Rater Agreement Protocol. Two domain experts independently annotatedn=50randomly sampled conversations from Z test across three dimensions: goal completionc, tool hallucinationh tool , and user hallucinationh user . We compare their labels to the outputs of the LLM judges J goal and J hall defined in the main paper. Results Table 3 reports the three-way agreement rate, i.e., the percentage of examples where both human annotators and the LLM judge fully agree on the label for a given dimension. DimensionThree-way Agreement (%) Goal completion (c)82.7 Tool hallucination (h tool )94.7 User hallucination (h user )94.7 Table 3: Three-way agreement among two human experts and the LLM judge on n=50 conversations. Takeaways The LLM judges align closely with human experts, supporting the reliability of our evaluation setup. Intermediate State-Tracker Validation Beyond final-state evaluation, we conduct a separate manual study to validate the proxy state tracker on intermediate steps. We sample 50 state transitions from multi-turn conversations, each occurring immediately after a tool call. Three human annotators independently judged whether the inferred proxy state was correct. Using the human majority label as reference, the state tracker achieves 92% correctness on these intermediate states, with 0.86 inter-annotator agreement (Fleiss’κ). This confirms that the proxy state tracker maintains high fidelity throughout the interaction, not only at conversation end. B Training Data and Hyperparameters Training data. We roll out each training scenarioz∈Z train (|Z train |=157) for 10 trajectories, yielding D raw = τ i 1570 i=1 . Each trajectoryτis scored by the goal-completion judgeJ goal with indicatorc(τ)∈ 0, 1. For rejection-sampling fine-tuning (RFT) and supervised fine-tuning (SFT), we success-filter as D succ rft =τ ∈D raw : c(τ)=1, |D succ rft | = 1078; D succ sft =τ ∈D raw : c(τ)=1, |D succ sft | = 1209. Each successful trajectory is decomposed into stepwise supervision pairs at RA emission points (tool calls or user-facing messages):S rft =(x s ,y s )with|S rft | = 5017, andS sft =(x s ,y s )with|S sft | = 8057, wherex s is the trajectory prefix immediately before emission (the historyτ <s ) andy s ∈a t,k ,Y t is the RA’s next response. Hyperparameters.We fine-tune the base RA model (Qwen3-30B-A3B-Thinking-2507) with LoRA (Hu et al., 2022a): • LoRA rank/alpha: 32 / 32 • LoRA targets: all linear projections in self-attention (Q, K, V, O) • MoE router auxiliary loss coefficient: 1×10 −3 • Learning rate: 1×10 −5 (constant; no scheduler) • Training epochs: 2 C User Persona Definitions The user simulator f user supports three persona p configurations: • Power user: Provides complete constraints upfront and interacts efficiently. • Ambiguous user: Initially omits key information and requires clarification. • Confused user: Seeks guidance about the process and may misunderstand system responses. D Example Trajectory Conversation Step 1User Hi, this is James Wilson. Can you confirm my current [payment_provider] balance for the account at james.wilson@email.com? I’m seeing $450 on my end. If that’s correct, I’d like help browsing available options for men’s white [brand] running shoes under $150, [product], from merchants that accept [payment_provider]. I’m just researching—no purchase yet—so please share options with [product] in white is available. Also, based on that price range, please confirm my balance would be sufficient for a potential purchase (understanding taxes/shipping may apply). Conversation Step 2Reasoning Agent Tool call(s): call_account("subquery": "Confirm current [payment_provider] balance for the account associated with james.wilson@email.com. Provide exact amount and currency.") Conversation Step 3Tool Response "success": true, "message": "Your current [payment_provider] balance is $450.00", "data": "current_balance": 450.0, "currency": "USD" Conversation Step 4Reasoning Agent Tool call(s): call_commerce("subquery": "Search for men's white [brand] running shoes priced under $150 from merchants that accept [payment_provider]. Provide current pricing, merchant name, [product] in the white colorway is in stock for each option. Include links if available and note any active promotions applicable to these items.") Conversation Step 5Tool Response "success": true, "message": "[product] men's white [brand] running shoes under $150 that accept [payment_provider]. Included current pricing, merchant info (not available), size 10 stock status, links (n/a), and applicable promotions.", "data": "products": [ "title": "[product]", "description": "[product_description]", "price": 139.99, ... Figure 6: Conversation flow across sequential steps, showing user turns, reasoning-agent tool calls, and tool responses. Brand names and trademarks have been redacted. Conversation Step 5 (cont.)Tool Response "brand": "[brand]", "category": "Men's Running", "colorway": "White/Black", "rating": 4.6, "availability": "In Stock", "size_10_in_stock": true, "size_10_stock_count": 5, "merchant": "name": "n/a", "payment_provider_accepted": true , "product_url": "n/a", "shipping": "standard": "Free (3-5 business days) on orders over $50", "express": "$15.99 (1-2 business days)", "next_day": "$25.99 (order by 2 PM PT)" , "promotions": [ "Free standard shipping on orders over $50", "[product] interest-free payments available for purchases $30-$1,500" ] , "title": "[product]", "description": "[product_description]", "price": 69.99, "brand": "[brand]", "category": "Men's Running", "colorway": "White/Grey", "rating": 4.3, "availability": "In Stock", "size_10_in_stock": true, "size_10_stock_count": 7, "merchant": "name": "n/a", "payment_provider_accepted": true , "product_url": "n/a", "shipping": "standard": "Free (3-5 business days) on orders over $50", "express": "$15.99 (1-2 business days)", "next_day": "$25.99 (order by 2 PM PT)" , "promotions": [ "Free standard shipping on orders over $50", "[product] interest-free payments available for purchases $30-$1,500" ] , "title": "[product]", "description": "[product_description]", "price": 129.99, ... Figure 6: Conversation flow across sequential steps, showing user turns, reasoning-agent tool calls, and tool responses. Brand names and trademarks have been redacted. Conversation Step 5 (cont.)Tool Response "brand": "[brand]", "category": "Men's Lifestyle/Running", "colorway": "White", "rating": 4.7, "availability": "In Stock", "size_10_in_stock": true, "size_10_stock_count": 4, "merchant": "name": "n/a", "payment_provider_accepted": true , "product_url": "n/a", "shipping": "standard": "Free (3-5 business days) on orders over $50", "express": "$15.99 (1-2 business days)", "next_day": "$25.99 (order by 2 PM PT)" , "promotions": [ "Free standard shipping on orders over $50", "[product] interest-free payments available for purchases $30-$1,500" ] ], "result_count": 3, "search_query": "Search for men's white [brand] running shoes priced under $150 from merchants that accept [payment_provider]. Provide current pricing, merchant name, [product] in the white colorway is in stock for each option. Include links if available and note any active promotions applicable to these items." Conversation Step 6Reasoning Agent Tool call(s): show_to_user("subquery": "Hi James 2014 here 2019s what I found: 1) Balance confirmation 2022 Your current [payment_provider] balance is $450.00 USD. 2) Men 2019s white [brand] running shoes under $150 ([payment_provider] accepted; size 10 in white available) 2022 [product] (White/Black) 2014 $139.99 - Size 10: In stock (5 left) - Shipping (from listing): Free standard over $50; Express $15.99; Next-day $25.99 - Promo noted: [product] available (for eligible purchases) 2022 [product] (White/Grey) 2014 $69.99 - Size 10: In stock (7 left) - Shipping (from listing): Free standard over $50; Express $15.99; Next-day $25.99 - Promo noted: [product] available (for eligible purchases) 2022 [product] (White) 2014 $129.99 (Lifestyle/Running) - Size 10: In stock (4 left) - Shipping (from listing): Free standard over $50; Express $15.99; Next-day $25.99 - Promo noted: [product] available (for eligible purchases) : 2022 The listings indicate that the merchants accept [payment_provider]; specific merchant names and direct product links were not provided in the results. 2022 Stock and pricing can change quickly; let me know if you want me to refresh availability or pull more options. sufficiency 2022 Based on these prices, your $450.00 balance would be sufficient to cover any of the listed items before taxes and shipping. you like me to refine the search (e.g., specific ... Figure 6: Conversation flow across sequential steps, showing user turns, reasoning-agent tool calls, and tool responses. Brand names and trademarks have been redacted. Conversation Step 6 (cont.)Reasoning Agent model/cushioning like [product] vs. [product], road vs. trail), check additional retailers, or set up an [product] in white under $150?") Conversation Step 7User <done> Figure 6: Conversation flow across sequential steps, showing user turns, reasoning-agent tool calls, and tool responses. Brand names and trademarks have been redacted.

Instant research discovery