Paper deep dive

Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

Yiwen Zhao, Zhihao Wen, Yuchen Mao, Mingxuan Jiang, Yihao Hu, Pan Wang, Xin Zhang, Wei Wu

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 97

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/19/2026, 4:59:29 AM

Summary

The paper introduces Feedback-Aware Credit Assignment (FACA), a method for training multi-turn user-facing agents that incorporates next-turn user reactions as local credit signals. Unlike standard Interactive GRPO which only uses terminal rewards, FACA aligns user reactions (progress or friction) with preceding user-to-user segments to derive a locally normalized reaction advantage. This is added to the terminal outcome advantage to improve agent performance in tool-use scenarios.

Entities (9)

FACA · method · 98%Interactive GRPO · method · 95%τ-Bench · dataset · 92%Co-Gym · dataset · 90%Pare-Bench · dataset · 90%Qwen3-14B · model · 90%Qwen3-8b · model · 90%Telecom · domain · 88%User-to-User Segment · concept · 85%

Relation Signals (8)

FACA → improves → Interactive GRPO

confidence 95% · FACA improves the nine-domain τ-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively.

FACA → uses → User Reaction

confidence 92% · FACA aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage

FACA → evaluatedon → τ-Bench

confidence 90% · Our main evaluation suite contains nine domains: Airline and Retail from τ-bench... FACA improves the nine-domain τ-family average

FACA → testedwith → Qwen3-8b

confidence 90% · We experiment with Qwen3-8B and Qwen3-14B... FACA improves... at 8B

FACA → testedwith → Qwen3-14B

confidence 90% · We experiment with Qwen3-8B and Qwen3-14B... FACA improves... at 14B

FACA → evaluatedon → Co-Gym

confidence 88% · The same ordering holds zero-shot on Pare-Bench and Co-Gym.

FACA → evaluatedon → Pare-Bench

confidence 88% · The same ordering holds zero-shot on Pare-Bench and Co-Gym.

Telecom → showslargestgain → FACA

confidence 85% · Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce \textbf{F}eedback-\textbf{A}ware \textbf{C}redit \textbf{A}ssignment (\textsc{FACA}), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, \textsc{FACA} improves the nine-domain $\tau$-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

94,046 characters extracted from source content.

Expand or collapse full text

BBAAD9C2010037A16BA0000FCD90B6403519B27D1D132B20A7D9FE32B1A92BBA9B41B73861D17B0725792708B84C113CD0E926A3E1D03B116B1AC805767E3102841088021B72C89794E38D776BE341D6A973EC0275E3C39112E119B2BEA1C92D993819B8E3 August 19, 2026 Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context Yiwen Zhao 1,2* Zhihao Wen 2* Yuchen Mao 2* Mingxuan Jiang 1 Yihao Hu 2 Pan Wang 2 Xin Zhang 2 Wei Wu 2 1 Fudan University 2 Ant International, Ant Group Correspondence: z.wen@antgroup.com Abstract User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce Feedback-Aware Credit Assignment (Faca), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, Faca improves the nine-domain휏-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents. Keywords: LLM agents, multi-turn interaction, reinforcement learning, credit assignment, user simulation 1 Introduction Tool-using language agents are moving from single-turn assistants toward user-facing systems that operate reliably and safely over extended conversations. A travel agent may need to inspect a reservation, elicit missing preferences, compare options, obtain authorization, and only then modify the database. Unlike fully specified tasks that provide all relevant information upfront, these interactions require the agent to discover user intent while acting in an external environment under evolving constraints. Goals may be omitted initially or revealed only after the agent elicits them, making the user an active participant rather than a fixed input [Yao et al., 2025, Qian et al., 2025a, Barres et al., 2025]. Figure 1 illustrates this structure. Between adjacent user turns푢 푡 and푢 푡+1 , the agent may produce several messages and tool interactions before returning control. We call this block a user-to-user (U2U) segment. The next user turn may provide information, approve an action, reject a proposal, or correct a misunderstanding. Success therefore requires coordinating dialogue, tool use, and user decisions across segments throughout the evolving interaction process. Interactive benchmarks in the휏family expose these de- mands [Yao et al., 2025, Barres et al., 2025, Shi et al., 2026]. Training frameworks such as MUA-RL and UserRL go fur- ther by placing an LLM-simulated user inside reinforcement- learning rollouts [Zhao et al., 2025b, Qian et al., 2025b]. This shift lets the policy explore complete conversations while the final database or world state provides objective success supervision. Outcome-only credit accommodates many valid trajectories, but collapses their internal structure: effective elicitation, errors, and later repair receive the same advantage. The reward says whether a trajectory succeeded, but not where it changed course. Our central observation is that the next user turn is more than context. Prospectively, it guides the next decision; retrospectively, it provides local evidence about the preced- ing U2U segment. Supplying requested information can indicate effective elicitation; challenges or clarifications can expose friction. These context-dependent reactions are not correctness labels, yet outcome-only training leaves them unused. Can this process evidence improve local credit while verified task completion remains the outcome objective? * Equal Contributions1 arXiv:2608.17499v1 [cs.AI] 18 Aug 2026 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context I need to move my flight to tomorrow morning. I’l inspect the booking and available flights. get_reservation(...)booking found Booking found; checking morning options. search_flights(..., AM)UA123 · 09:10 · available UA123 is available. Shall I make the change? I’l update your reservation. change_flight(..., UA123)reservation updated UserAgentTool callTool result You’re now booked on UA123 at 09:10. ⋯ → → Implicit attitude / strategy: acceptance + progress → local feedback unused by outcome-only credit Yes, choose the morning flight. Figure 1: Multi-turn tool interaction. A user-to-user segment contains all agent responses, tool calls, and tool results between two adjacent user turns; the ellipsis denotes continued interaction. We introduce Feedback-Aware Credit Assignment (Faca). Faca aligns each next-user reaction with the preceding U2U segment, normalizes reactions within a local rollout group, and adds the resulting process advantage to terminal outcome advantage. It changes only credit assignment: Faca and the matched outcome-only Interactive GRPO control share the frozen simulator, agent-visible dialogue, initialization, rollout, and optimization, while private reaction metadata remains hidden from the agent. Across three independently trained runs per scale, Faca improves the nine-domain휏-family average by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Effects are heterogeneous, with the largest gains in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. Faca-trained policies also outperform matched controls zero- shot on Pare-Bench and Co-Gym. These results support a conditional benefit when user reactions carry informative local structure, rather than universal agent improvement. Overall, our main contributions are as follows: •Formulation. We formalize U2U segments as local credit-assignment units, and subsequent user reactions as temporally localized evidence of interaction progress. • Method. We introduce Faca, which adds U2U-level reaction credit without a learned critic, additional rollouts, or agent-visible labels. • Findings. Results show cross-scale gains, reaction sensi- tivity, zero-shot transfer, and domain variation. 2 Related Work Interactive user-agent learning. Modern user-facing agents solve practical tasks through iterative communication, tool use, and clarification of underspecified goals [Qian et al., 2024, Barres et al., 2025, Qian et al., 2025a, Zhao et al., 2025b]. Work spans offline trajectory synthesis and online simulator-in-the-loop RL. Offline methods construct social or tool-use interactions for subsequent training, including SOTOPIA-휋/Ω, APIGen-MT, and Magnet [Wang et al., 2024, Zhang et al., 2025b, Prabhakar et al., 2025, Yin et al., 2025]. LAM SIMULATOR and Simia-SFT instead generate trajectories through interactive exploration or seed-set ex- pansion [Hoang et al., 2025, Li et al., 2025]. Online methods generate interactions against the current policy: MUA-RL and UserRL retain simulated users inside multi-turn RL roll- outs, while Simia-RL uses model-simulated environment feedback during policy optimization [Zhao et al., 2025b, Qian et al., 2025b, Li et al., 2025]. They place simulated user behavior directly inside the learning loop. This makes interaction data responsive to the policy’s evolving behavior. Credit assignment in multi-turn agents. Terminal task rewards verify overall success but cannot identify which turns elicited useful information, introduced errors, or en- abled recovery. Prior work obtains turn-level credit from learned critics or intermediate evaluators [Zhou et al., 2024, 2025, Choudhury, 2025, Wei et al., 2025]. Agent Light- ning instead decomposes trajectories into transitions and converts monitoring signals into intermediate rewards [Luo et al., 2025]. Critic-free alternatives derive local advantages from continuations, repeated states, policy information, out- come potentials, structured reasoning events, or tool-call entropy [Guo et al., 2025, Zong et al., 2026, Feng et al., 2025, Wang et al., 2026, Kong et al., 2026, Hu et al., 2026a, Zhang et al., 2026, Li et al., 2026, Zhu et al., 2026, Xie et al., 2026]. SEAL instead uses verifier-grounded diagnoses to reweight trajectory-level GRPO advantages [Hu et al., 2026b]. User follow-ups provide complementary supervi- sion: FLR scores candidate responses using the likelihood of curated positive and negative follow-ups, while RLUF learns from production reactions and documents reward hacking under over-optimization [Zhang et al., 2025a, Han et al., 2025]. Unlike these response-level objectives, Faca uses the partner’s next reaction as credit for the immediately preceding U2U segment, alongside terminal task advantage. The reaction already occurs inside the same rollout, so the method needs neither additional continuations nor a learned turn-level evaluator. Its private strategy annotation is used only to construct credit and remains hidden from the agent. Accordingly, the signal is neither policy-derived uncertainty nor a reward inferred from latent simulator state. This positions Faca narrowly as reaction-grounded local U2U supervision, not a hierarchical credit-assignment framework. 2 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context (a) Multi-turn user-interacting rollout final task reward Prompt x goal + tools + environment + U2U segment detail in (b) ↓ ++ ⋯ → R 1 0 −+ ⋯ → R 2 − ⋯ → R K User Simulator AgentTool callTool result + −0Reaction polarity (b) Inside one U2U segment agent span 1: inspect the reservation A k,q get_reservation(...) → booking found user invisible agent span 2: search morning options search_flights(...) → UA123 available user invisible agent delivery: UA123 is available. Shall I book it? private ReAct (agent-invisible) goal · progress · withheld → strategy z = confirm → f = +1 + visible utterance u k,q+1 : f k,q = +1 (c) Two-level advantage assignment Trajectory-level outcome advantage Turn-level reaction advantage R k → A o k → A o k A o k A o k ⋯ f k,q → A p k,q → A p k,1 A p k,q A p k,Q ⋯ Total U2U advantage one value for every trainable token in U2U q ⋯ s 1,q,0 t 1,q,1 o 1,q,1 ⋯ t 1,q,m o 1,q,m s 1,q,m ⋯ ⋯ ⋯ → → → s 2,q,0 t 2,q,1 o 2,q,1 ⋯ t 2,q,n o 2,q,n s 2,q,n s k,q,0 t k,q,1 o k,q,1 ⋯ t k,q,p o k,q,p s k,q,p Advantage computing detail in (c) ↓ Traj 1 Traj 2 Traj k A k,q A k,q + USER FEEDBACK SIGNAL USER FEEDBACK CREDIT NEXT-USER FEEDBACK Yes, that works, please book it. Figure 2: FACA overview. (a) For each prompt, we sample퐾multi-turn user-interacting rollouts and obtain a verified final task reward. (b) A U2U segment may contain multiple agent and tool spans; the next user call jointly produces a visible utterance and private reaction strategy, which is aligned with the complete preceding segment. (c) Terminal rewards yield trajectory-level outcome advantages, while aligned reactions yield turn-level process advantages. Their additive combination supplies the final advantage for agent optimization. 3 Feedback-Aware Credit Assignment 3.1 Interactive Rollout and Outcome Credit For prompt푥, an agent policy휋 휃 interacts with a tool environment and a frozen user policy휋 푢 . Group-relative optimization [Shao et al., 2024b] samples 퐾 rollouts, 휏 푘 = (푥,푎 푘,1 ,푢 푘,2 ,...,푎 푘,푇 푘 ,푢 푘,푇 푘 +1 ),(1) where푎 푘,푞 is the complete U2U segment between adjacent user messages and may contain multiple language and tool spans. The environment returns verified terminal reward 푅 푘 ∈ 0, 1from the final database or world state. Each U2U unit therefore preserves the full sequence of language and tool actions produced before control returns to the user. Our matched outcome-only Interactive GRPO control computes: 퐴 o 푘 = 푅 푘 − 휇 푅 (푥) 휎 푅 (푥)+ 휖 ,(2) and broadcasts퐴 o 푘 to every trainable agent token in trajectory 푘. It shares the user policy, simulator prompt, agent-visible dialogue, rollout, and optimization with Faca. Components remain matched throughout training. This terminal signal ac- commodates diverse successful trajectories but is temporally flat: useful elicitation, errors, and later repair receive the same advantage. Moreover, a group with constant terminal rewards has퐴 o 푘 = 0. Faca retains this verified outcome branch and adds reaction-grounded credit at the U2U level. 3.2 User Reaction as Local Evidence For both Interactive GRPO and Faca rollouts, the frozen simulator emits a visible utterance푢 푘,푞+1 and private behav- ioral strategy푧 푘,푞+1 in the same call. Only the utterance is appended to the agent context. Interactive GRPO discards 푧 푘,푞+1 ; Faca reads it after rollout construction and aligns it to the immediately preceding agent segment푎 푘,푞 . Temporal adjacency is an inductive bias rather than a claim that every reaction is caused only by that segment. The strategy is read only after the agent yields control, so it never enters the agent-visible context. We map the strategy to ternary evidence: 푧 푘,푞+1 → 푓 푘,푞 ∈ −1, 0,+1.(3) Confirming, closing, or revealing specifically requested information is treated as progress-consistent; asking for clar- ification, challenging a solution, or changing the stated goal is friction-consistent; vague or invalid output is assigned neutral credit. These labels describe interaction movement rather than user sentiment or action correctness. This coarse map preserves ambiguity while retaining the reaction’s di- rection. Table 1 gives the complete mapping, Appendix B details extraction and fallback behavior, and Appendix C gives the shared simulator prompt. An utterance-only ex- tractor could remove reliance on private reaction metadata and extend Faca beyond the instrumented setting toward deployment in less controlled environments. 3 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Strategy eventInterpretation풇 confirmprogress+1 closecompletion+1 reveal-piecerequested state elicited+1 be-vagueambiguous0 invalidmalformed0 ask-clarification local interaction friction−1 challenge-solution proposal challenged−1 change-mindgoal friction; possibly exogenous −1 Table 1: User-strategy mapping used by Faca. Polarity represents coarse evidence of interaction progress or friction, not sentiment or verified task correctness. 3.3 Two-Level Additive Advantage The outcome branch reuses퐴 o 푘 from Equation 2. For prompt 푥and U2U index푞, letI(푥,푞)contain rollouts that reach푞 and have a valid next-user reaction. We construct the aligned reaction group: B 푝 (푥,푞) =푓 푖,푞 | 푖 ∈I(푥,푞),(4) and normalize within that local comparison set: 퐴 p 푘,푞 = 푓 푘,푞 − 휇 푓 (푥,푞) 휎 푓 (푥,푞)+ 휖 .(5) The U2U estimator uses only the immediate next reaction and does not propagate later reactions backward. Singleton or constant anchors receive zero. Every trainable token in 푎 푘,푞 receives: 퐴 푘,푞 = 퐴 o 푘 + 휆퐴 p 푘,푞 .(6) The two terms are complementary:퐴 o 푘 preserves the ter- minal objective, whereas퐴 p 푘,푞 distinguishes local progress from friction. Disabling the reaction branch reduces Equa- tion 6 to the strict outcome-only Interactive GRPO control. In an outcome-homogeneous group, reaction labels can still differ across rollouts; this is a mechanical property rather than an independent performance claim. 3.4 Optimization and Anchoring For assistant tokens, we first clip the GRPO importance ratio [Schulman et al., 2017]: e휌 푘,푡 = clip 휌 푘,푡 , 1− 휀, 1+ 휀 .(7) Substituting the U2U-level advantage퐴 푘,푞 then gives the actor objective: L(휃) =−E 푘,푡 min 휌 푘,푡 퐴 푘,푞(푡) , e휌 푘,푡 퐴 푘,푞(푡) .(8) Generated language and tool-call tokens are trainable; user and raw tool-result tokens are masked. Because reaction metadata is produced in the existing user call, the estimator needs no learned critic, separate evaluator, or additional rollout. We use ordinal U2U index푞as the default anchor because exact dialogue states rarely repeat. This approximation can compare different semantic phases after trajectories diverge, especially at late singleton turns. One-U2U-shifted reactions test temporal adjacency, while random polarity tests whether reaction semantics matter. Equation 6 intentionally changes the training signal and is not potential-based policy-invariant shaping [Ng et al., 1999]. The pre-normalization reward scale cancels under z-normalization; the effective process- strength parameter is휆. Because퐴 p is derived from a simulator reaction rather than the verified terminal state, 휆also limits proxy dominance. We cap its positive value at0.5so that reaction credit remains optimization-relevant while퐴 o stays dominant in aggregate, reducing, but not eliminating, reward-hacking risk from simulator bias or misclassified feedback. 4 Experiments 4.1 Datasets 휏 -bench family. Our main evaluation suite contains nine domains: Airline and Retail from휏-bench; Airline, Retail, and Telecom from휏 2 -bench; and Airline, Retail, Telecom, and Bank from휏 3 -bench [Yao et al., 2025, Barres et al., 2025, Shi et al., 2026]. These benchmarks require agents to resolve multi-turn user requests through tool use and communication. Telecom is a mixed-control setting in which users execute device-side actions, while휏 3 Bank additionally requires grounding in an unstructured policy collection. Crucially, the subsequent RL stage uses only the Airline and Retail training splits from휏-bench; neither Telecom nor Bank appears in RL training. Results on these two domains therefore measure transfer beyond the RL training domains. Pare-Bench. Pare-Bench contains 143 proactive mobile scenarios that require observing user and environment events, inferring latent goals, proposing an intervention, and execut- ing across stateful apps under evolving conditions after user acceptance [Nathani et al., 2026]. It tests whether policies trained on reactive customer-service interactions transfer to a proactive user-agent protocol. Co-Gym. Co-Gym evaluates bidirectional collaboration under dual control and non-turn-taking coordination in shared workspaces [Shao et al., 2024a]. Its simulated condition contains 102 Travel Planning, 100 Related Work Writing, and 110 Tabular Analysis tasks, where the agent and user can act asynchronously in the same environment. 4.2 Experimental Settings We experiment with Qwen3-8B and Qwen3-14B, initialized by SFT on the public MUA-RL release [Zhao et al., 2025a]. From the same SFT checkpoint, we train Interactive GRPO 4 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Model 흉-bench흉 2 -bench흉 3 -bench Avg. Air.Ret.Air.Ret.Tel.Air.Ret.Tel.Ban. Qwen3-8B20.047.014.040.44.414.036.021.13.122.2 + SFT14.039.124.036.02.616.040.428.93.122.7 + Interactive GRPO28.052.232.046.530.739.350.029.83.434.7 + Faca37.343.838.054.441.240.053.553.53.440.6 Qwen3-14B12.059.126.049.12.614.050.023.73.126.6 + SFT14.049.630.047.41.822.043.016.74.125.4 + Interactive GRPO36.058.334.069.026.340.064.650.63.842.5 + Faca38.056.538.062.383.640.765.582.77.252.7 Table 2: Strict pass@1 (%) across nine domains in the휏-bench family. RL scores average three independently trained step-120 runs per method and scale; each seed is evaluated once under one shared protocol. Base and SFT are fixed references. Avg. weights all nine domains equally. Full nominal task sets are used, and missing cases count as failures. Higher is better. “+” marks a training stage; the two RL rows are alternative continuations of the same SFT. Values use one decimal; bold marks column bests within each scale, including ties. and Faca as alternative RL continuations, with three inde- pendent training seeds per method and scale. All headline RL results use the step-120 checkpoints. The comparison isolates credit assignment. Within each scale, the two RL arms share the frozen DeepSeek-V4- Flash simulator and prompt, agent-visible utterances, SFT initialization, training data, rollout construction, optimizer, and horizon. Interactive GRPO uses only terminal outcome advantage, whereas Faca additionally uses the simulator’s private reaction metadata to construct U2U-level process advantage; this metadata is never exposed to the agent. For the휏-bench family, we report strict pass@1 from the verified final environment state. Each of the three indepen- dently trained checkpoints is evaluated once under the same protocol, and the main table averages them domainwise. Avg. equally weights the nine domains; run-level dispersion is reported in Section 5. Base and SFT are fixed pre-RL references. Pare-Bench and Co-Gym are evaluated zero-shot without benchmark-specific training or tuning. Details of the SFT and RL datasets, training hyperparameters, evaluation protocols, and run provenance are provided in Appendix D. 5 Results 5.1 Main Results on the 휏-Bench Family Table 2 shows that reaction-grounded credit improves the matched outcome-only control at both scales. At 8B, the mean±sample standard deviation across three trained runs rises from 34.66±0.25 with Interactive GRPO to 40.57±1.04 with Faca, a 5.91-point gain from unrounded domain means. At 14B, the scores rise from 42.51±0.12 to 52.73±1.53, a 10.22-point gain. Matching runs by training- seed identifier, the gains are 4.67/6.33/6.75 points at 8B and 8.62/10.14/11.91 points at 14B; thus the ordering holds in all runs at both scales. Each checkpoint is evaluated once, so these standard deviations describe run-level dispersion rather than isolating training variation from evaluation stochas- ticity. Appendix D reports the run counts and provenance. Because the two RL arms share their SFT initialization, user- generation path, observations, and optimization setup, this comparison tests reaction-grounded versus outcome-only credit. Consistent with MUA-RL, cold-start SFT can intro- duce domain-specific biases that limit generalization beyond the SFT distribution, whereas subsequent simulator-in-the- loop RL recovers and surpasses the base models [Zhao et al., 2025b]. The improvement is broad but not uniform. Faca leads Interactive GRPO on seven of nine domains at each scale, with its largest gain in all four Telecom scale–benchmark cells. It ties one and trails one domain at 8B, and trails two Retail domains at 14B. The main result is therefore an aggregate cross-scale gain with a repeated Telecom-centered pattern rather than universal domain dominance. The larger absolute gap at 14B than at 8B does not by itself establish a scaling law from two model sizes. The supported conclusion is narrower: the matched advantage persists across both scales despite different domain-level regressions. 5.2 Zero-Shot Transfer The matched ordering persists under two unseen interaction protocols. On Pare-Bench, strict Pass@1 rises from 6.29% to 10.49% at 8B and from 10.49% to 13.29% at 14B; Pass@4 follows the same ordering (Figure 3). All bars use the same 143-scenario full split; each checkpoint is evaluated once, so no uncertainty interval is shown. On Co-Gym, Faca leads 13 of 16 scale–metric cells, including every Related Work, Tabular Analysis, and Overall cell, but Travel DR decreases slightly at both scales (Table 3). Here DR, TP, and CS denote Delivery Rate, Task Performance, and the evaluator-reported overall score. These comparisons test whether Faca retains its ordering over outcome-only training, not an absolute ranking over pretraining or SFT variants. Appendix A records the full protocols, coverage, and comparison scope. 5 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Qwen3-8BQwen3-14B 0 10 20 30 40 Strict pass rate (%) 6.3% 10.5%10.5% 13.3% 18.9% 30.1% 24.5% 32.9% Interactive GRPO Pass@1 FACA Pass@4 Figure 3: Matched zero-shot transfer to Pare-Bench under identical evaluation settings. Solid and diagonally hatched bars report strict Pass@1 and Pass@4, respectively; colors distinguish Interactive GRPO and Faca across both evaluated model scales. Domain Metric Qwen3-8BQwen3-14B I-GRPO FACA I-GRPO FACA Travel DR83.382.479.477.5 TP70.469.167.067.2 Related DR76.087.090.094.0 TP52.954.343.347.6 Tabular DR67.373.683.688.2 TP23.427.623.931.6 Overall DR75.380.884.386.5 CS37.640.936.940.6 Table 3: Zero-shot Co-Gym results on a 0–100 scale. Faca leads 13 of 16 cells. I-GRPO denotes Interactive GRPO; bold marks the better method. DR, TP, and CS denote Delivery Rate, Task Performance, and overall score, respectively. Pare-Bench changes initiative and stateful app execution, whereas Co-Gym changes the coordination structure. Their shared ordering is cross-protocol evidence, but low absolute Pare-Bench rates and Co-Gym Travel regressions keep the claim comparative rather than broadly conclusive. 5.3 Ablation Studies Table 4 reports a single-run controlled ablation from the 8B s1 experiment family; every row uses the same one- shot evaluation protocol. Its Aligned and Outcome-only checkpoints also appear in Table 2, whose headline results average s1–s3. Faca (Aligned,휆 = 0.5) reaches 39.37 overall and 47.37 on Telecom, whereas one-U2U-shifted and randomized reactions erase the gain over outcome- only training. Preserving reaction frequency or introducing generic token-varying advantages is therefore insufficient: both reaction semantics and temporal alignment matter. The Shifted Avg. counts its unfinished휏 3 Banking tail as failures. Among aligned settings, a small positive weight improves over휆 = 0, and the default휆 = 0.5performs best: relative to Outcome-only, it adds 4.67 points to Avg. and 17.11 points to Tel. 2 . Reversing the sign sharply degrades both aggregates, while Shifted and Randomized credit fall below Outcome- only. Together, these controls show that token-level variation alone is insufficient; the signal must preserve reaction content and its temporal association with the preceding agent span. The weight sensitivity further supports reaction credit as an auxiliary, rather than alternative, objective. The control gaps sharpen this interpretation. Relative to Outcome-only, Aligned with휆 = 0.1adds 2.08 points to Avg. and 15.35 points to Tel. 2 , while휆 = 0.5raises these gains to 4.67 and 17.11 points. Both controls also lose the Telecom gain, reinforcing the need for signed, temporally matched credit. These margins show that alignment improves both the broad average and the feedback-rich Telecom subset. 6 Analysis 6.1 When Are User Reactions Informative? The same reaction label need not be equally informative in every task. We use reaction observability descriptively for the extent to which the next user response exposes a state change caused by the preceding U2U segment. When progress occurs mostly inside tools or policy checks, the user may not observe whether an action was correct. When control alternates between agent and user, the user’s report can instead reveal whether an instruction worked, whether validation failed, and what state remains unresolved. The cross-domain pattern in Table 2 is consistent with this distinction. Telecom is particularly feedback-rich. Its tasks repeatedly alternate identity grounding, hidden device-state inspection, agent-side operations, user-executed actions, validation, re- vised diagnosis, and final verification. The next user turn therefore often reports the direct consequence of the pre- ceding segment: progress-consistent reactions can reinforce effective elicitation and repair, while friction-consistent re- actions can localize an unclear instruction or failed proposal. This interaction structure offers a concrete account of why all four Telecom cells show the largest gains under the same estimator. The OOD results extend this account without making it universal. Pare-Bench changes from reactive customer service to proactive mobile assistance, while Co-Gym in- troduces bidirectional, non-turn-taking collaboration. Faca retains its ordering over the outcome-only control at both scales, but Co-Gym’s Travel regression shows that adding reaction credit is not uniformly beneficial. Together, these results suggest that the advantage transfers when subsequent user behavior remains informative about task progress, rather than merely when a task is long or interactive. This account predicts more than a generic benefit from longer conversations. Additional turns create more possible 6 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Condition흀Avg.Tel. 2 Aligned0.5039.3747.37 Aligned0.1036.7845.61 Aligned−0.5015.0817.11 Shifted0.5033.9229.39 Randomized0.5032.6626.32 Outcome-only0.0034.7030.26 Table 4: Single-run controlled 8B credit ablation across the휏- bench family under the shared one-shot evaluation protocol. Avg. is the nine-domain average; Tel. 2 averages 휏 2 and 휏 3 Telecom. credit locations, but they help only when the next user response reveals a consequence of the preceding segment. Reaction observability, rather than interaction length, is the more direct hypothesis suggested by this pattern. 6.2 Training-Signal Dynamics Per-U2U telemetry separates favorable reactions from effec- tive differential credit. A matched 8B replication records all 154,219 U2U segments with exact agreement between telemetry and tensor counts. From the first to the last 40 steps, positive reactions rise from 69.95% to 82.51%, driven in part by moreconfirmand lessbe_vague. Over the same period, nonzero normalized process-advantage coverage decreases from 86.16% to 82.50%. Figure 4 shows segment- weighted phase averages; its dashed coverage series uses the truncated 70–95% right axis. Favorable reactions are therefore not copied one-for-one into reaction credit: anchor- relative normalization retains only differences within the local comparison set. The opposing trends also distinguish signal prevalence from signal discrimination: feedback can become more favorable even as local rollout groups provide fewer nonzero reaction contrasts. Terminal credit is also silent for a substantial share of prompt groups. The step-120 logs contain 1,920 group- steps at each scale, of which 39.27% at 8B and 31.04% at 14B are all-correct or all-wrong and consequently have zero group-normalized outcome advantage. Across training, all-correct groups become more frequent while all-wrong groups decline (Figure 5). These groups identify where reaction credit can supply differential signal; their prevalence alone does not establish that this signal caused the final performance gain. At the logged advantage scale, the process branch remains auxiliary. With휆 = 0.5, it accounts for 36.00% and 33.79% of a pre-optimization L1 advantage-magnitude proxy at 8B and 14B, and its weighted magnitude exceeds the outcome component in only 6/120 and 1/120 steps. Appendix E reports the phasewise curve (Figure 6), segment coverage, and the limits of this proxy. These diagnostics show that the branch is active without dominating terminal credit, but they are descriptive rather than a decomposition of optimizer updates or causal performance gains. 1–4041–8081–120 Training steps 0 20 40 60 80 100 U2U segments (%) 70.0 77.6 82.5 19.7 13.4 10.0 10.4 9.0 7.5 Positive Negative Neutral Nonzero 70 75 80 85 90 95 Nonzero (%) Figure 4: Matched 8B Faca replication under identical train- ing conditions: per-U2U reaction polarity and nonzero process- advantage coverage across 120 steps over three phases. These dynamics support auxiliary reaction credit along- side verified outcomes. In homogeneous groups, terminal normalization provides no differential signal; combining it with reaction credit introduces contrasts while preserving task completion as the governing objective. Its bounded contribution throughout training confirms that the process branch remains active without displacing primary outcome supervision. 6.3 Trajectory-Level Evidence from Telecom We pair the final 14B s1 checkpoints task by task in their single run-level evaluation. On휏 2 Telecom, Faca succeeds on 96/114 tasks versus 30/114 for Interactive GRPO; the paired cells are 23 both-success, 73 FACA-only, 7 outcome- only, and 11 both-fail. On휏 3 , the corresponding totals are 95/114 versus 58/114, with cells 47, 48, 11, and 8. The split is not uniform: Faca trails 20/29 versus 24/29 on휏 3 Service while leading on Mobile and MMS. Appendix F reports issue-family counts, paired tests, and the complete representative trace. A matched Hard mobile-data trace illustrates one recovery pattern, not a general length or multi-turn advantage. Faca reaches verified success in 48 stored events versus 54 for Interactive GRPO. For context, the two traces contain 18 versus 21 assistant messages and 15 versus 17 tool calls, a modest rather than large difference in interaction count. Both attempt user-only tools from the agent side. In this trajectory, Faca recovers from two rejected calls and returns device control to the user, who completes four required state changes and a 275 Mbps test. Interactive GRPO accumulates five such rejections; its final agent-side speed-test attempt reaches the error limit before terminal verification. Across FACA-only wins, error-budget exhaustion ac- counts for 68/73 outcome-only failures on휏 2 and 28/48 on휏 3 ; the rest stop without task closure. This supports a descriptive account in which policies can differ in recovering 7 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context 010203040 Outcome-group rate (%) 8B 14B 5.0 19.7 34.4 27.2 5.6 13.8 25.3 15.9 All-correct First 20 steps All-wrong Last 20 steps Figure 5: Change in outcome-homogeneous group rates from the first to the last 20 training steps. Colors distinguish all-correct and all-wrong groups; hollow markers denote steps 1–20 and filled markers steps 101–120 at both model scales. from tool-authority errors and completing terminal valida- tion, but it does not identify reaction credit as the sole cause: both policies make authority errors, the failure depends partly on a fixed error budget, the trace is one task, and the current checkpoint still loses paired cases and the휏 3 Service slice. Taken together, the domain pattern, telemetry, and final-checkpoint trajectory evidence support a conditional mechanism claim rather than universal improvement. 7 Conclusion The next user turn provides noisy evidence about the pre- ceding U2U segment. Faca converts this reaction into locally normalized process credit and combines it with veri- fied terminal credit, without changing the frozen simulator, exposing labels to the agent, training a critic, or adding rollouts. Under a strict outcome-only Interactive GRPO control, Faca improves the nine-domain휏-family average at both 8B and 14B and retains this ordering on two zero-shot interaction protocols. Gains concentrate in feedback-rich Telecom, while domain regressions and temporal controls show that the benefit depends on informative, aligned re- actions. Faca therefore demonstrates that implicit user feedback can provide actionable local credit for multi-turn user-interacting agents while verified task completion re- mains the governing objective. More broadly, these results suggest that implicit user reactions can provide practical su- pervision for improving long-horizon learning in interactive agents. Limitations Faca is evaluated with a frozen DeepSeek-V4-Flash user simulator that emits private strategy metadata alongside each utterance. We treat this metadata as heuristic evidence for the preceding U2U; event-level causal attribution and recovery from observable utterances or real users remain open. Evidence covers Qwen3 8B/14B agents, nine휏-family domains, and zero-shot Pare-Bench/Co-Gym evaluation. Results establish aggregate gains over outcome-only training across scales and both transfer benchmarks, strongest in feedback-rich Telecom tasks. Validation across simulator families, architectures, and real users, as well as diverse deployment conditions, remains future work. References Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.휏 2 -Bench: Evaluating Conversa- tional Agents in a Dual-Control Environment. arXiv preprint arXiv:2506.07982, 2025. URL https://arxiv.org/ abs/2506.07982. Sanjiban Choudhury. Process Reward Models for LLM Agents: Practical Framework and Directions. arXiv preprint arXiv:2502.10325, 2025. URL https://arxiv.org/ abs/2502.10325. DeepSeek-AI et al. DeepSeek-V4: Towards Highly Effi- cient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348, 2026. doi: 10.48550/arXiv.2606. 19348. URL https://arxiv.org/abs/2606.19348. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group- in-Group Policy Optimization for LLM Agent Training. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2505.10978. Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models. In Advances in Neural In- formation Processing Systems, volume 38, 2025. URL https://papers.nips.c/paper_files/paper/2025/hash/ a6536243037d1e32c20de85137d478da-Abstract-Conference. html. Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xi- aoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, and Arya Talebzadeh. Reinforcement Learning from User Feed- back. arXiv preprint arXiv:2505.14946, 2025. URL https://arxiv.org/abs/2505.14946. Thai Hoang, Kung-Hsiang Huang, Shirley Kokane, Jian- guo Zhang, Zuxin Liu, Ming Zhu, Jake Grigsby, Tian Lan, Michael S. Ryoo, Chien-Sheng Wu, Shelby Hei- necke, Huan Wang, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. LAM SIMULATOR: Ad- vancing Data Generation for Large Action Model Train- ing via Online Exploration and Trajectory Feedback. arXiv preprint arXiv:2506.02298, 2025. URL https: //arxiv.org/abs/2506.02298. 8 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Senkang Hu, Yong Dai, Xudong Han, Zhengru Fang, Yuzhi Zhao, Sam Tak Wu Kwong, and Yuguang Fang. Self-Induced Outcome Potential: Turn-Level Credit As- signment for Agents without Verifiers. arXiv preprint arXiv:2605.04984, 2026a. URL https://arxiv.org/abs/ 2605.04984. Yihao Hu, Zhihao Wen, Xiujin Liu, Pan Wang, Xin Zhang, and Wei Wu. SEAL: Synergistic Co-Evolution of Agents and Learning Environments. arXiv preprint arXiv:2605.24426, 2026b. URL https://arxiv.org/abs/ 2605.24426. Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, and Bang Liu. InfoPO: Information-Driven Policy Optimization for User-Centric Agents. arXiv preprint arXiv:2603.00656, 2026. URL https://arxiv.org/abs/2603. 00656. Yuetai Li, Huseyin A. Inan, Xiang Yue, Wei-Ning Chen, Lukas Wutschitz, Janardhan Kulkarni, Radha Pooven- dran, Robert Sim, and Saravan Rajmohan. Simulating Environments with Reasoning Models for Agent Train- ing. arXiv preprint arXiv:2511.01824, 2025. URL https://arxiv.org/abs/2511.01824. Zeping Li, Hongru Wang, Yiwen Zhao, Guanhua Chen, Yixia Li, Keyang Chen, Yixin Cao, Guangnan Ye, Hongfeng Chai, and Zhenfei Yin. Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Lan- guage Model Agents. arXiv preprint arXiv:2602.02050, 2026. URL https://arxiv.org/abs/2602.02050. Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent Lightning: Train ANY AI Agents with Reinforce- ment Learning. arXiv preprint arXiv:2508.03680, 2025. URL https://arxiv.org/abs/2508.03680. Deepak Nathani, Cheng Zhang, Chang Huan, Jiaming Shan, Yinfei Yang, Alkesh Patel, Zhe Gan, William Yang Wang, Michael Saxon, and Xin Eric Wang. Proactive Agent Re- search Environment: Simulating Active Users to Evaluate Proactive Assistants. arXiv preprint arXiv:2604.00842, 2026. URL https://arxiv.org/abs/2604.00842. Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Pol- icy Invariance under Reward Transformations: Theory and Application to Reward Shaping. In International Conference on Machine Learning, 1999. Akshara Prabhakar, Zuxin Liu, Weiran Yao, Jianguo Zhang, Ming Zhu, Shiyu Wang, Zhiwei Liu, Tulika Awalgaonkar, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay. arXiv preprint arXiv:2504.03601, 2025. URL https:// arxiv.org/abs/2504.03601. Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yu- jia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Tell Me More! Towards Implicit User Intention Understanding of Lan- guage Model Driven Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 1088– 1113, 2024. doi: 10.18653/v1/2024.acl-long.61. URL https://aclanthology.org/2024.acl-long.61/. Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. UserBench: An Interactive Gym Environment for User-Centric Agents. arXiv preprint arXiv:2507.22034, 2025a. URL https://arxiv.org/abs/2507.22034. Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caim- ing Xiong, and Huan Wang. UserRL: Training In- teractive User-Centric Agent via Reinforcement Learn- ing. arXiv preprint arXiv:2509.19736, 2025b. URL https://arxiv.org/abs/2509.19736. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347. Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration. arXiv preprint arXiv:2412.15701, 2024a. URL https: //arxiv.org/abs/2412.15701. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024b. URL https://arxiv.org/abs/2402.03300. Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, and Victor Barres.휏-Knowledge: Evaluat- ing Conversational Agents over Unstructured Knowledge, 2026. URL https://arxiv.org/abs/2603.04370. Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. In- formation Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2510.14967. 9 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. SOTOPIA-휋: Interactive Learning of Socially Intel- ligent Language Agents. arXiv preprint arXiv:2403.08715, 2024. URL https://arxiv.org/abs/2403.08715. Quan Wei, Siliang Zeng, Chenliang Li, William Brown, Oana Frunza, Wei Deng, Anderson Schneider, Yuriy Nevmyvaka, Yang Katie Zhao, Alfredo Garcia, and Mingyi Hong. Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design. arXiv preprint arXiv:2505.11821, 2025. URL https://arxiv.org/abs/2505. 11821. Yutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu, Li Erran Li, and Xiaolong Wang. TIPS: Turn- Level Information-Potential Reward Shaping for Search- Augmented LLMs. arXiv preprint arXiv:2603.22293, 2026. URL https://arxiv.org/abs/2603.22293. An Yang et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025. doi: 10.48550/arXiv.2505. 09388. URL https://arxiv.org/abs/2505.09388. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.휏-bench: A Benchmark for Tool-Agent- User Interaction in Real-World Domains. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2406.12045. Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long T. Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, and Tomas Pfister. Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation. arXiv preprint arXiv:2503.07826, 2025. URL https://arxiv.org/abs/2503.07826. Chen Zhang, Dading Chong, Feng Jiang, Chengguang Tang, Anningzhe Gao, Guohua Tang, and Haizhou Li. Align- ing Language Models Using Follow-up Likelihood as Reward Signal. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 39, pages 25832– 25841, 2025a. doi: 10.1609/aaai.v39i24.34776. URL https://ojs.aaai.org/index.php/AAAI/article/view/34776. Wenyuan Zhang, Tianyun Liu, Mengxiao Song, Xiaodong Li, and Tingwen Liu. SOTOPIA-Ω: Dynamic Strategy Injection Learning and Social Instruction Following Evalu- ation for Social Agents. arXiv preprint arXiv:2502.15538, 2025b. URL https://arxiv.org/abs/2502.15538. Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. RLVMR: Reinforcement Learn- ing with Verifiable Meta-Reasoning Rewards for Ro- bust Long-Horizon Agents. In International Confer- ence on Learning Representations, 2026. URL https: //openreview.net/forum?id=cTbAevdwBE. Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, and Xunliang Cai. MUA-RL Cold-Start Dataset. Hugging Face dataset, 2025a. URL https://huggingface.co/datasets/ zzwkk/MUA-RL-Dataset. Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, and Xunliang Cai. MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for Agentic Tool Use. arXiv preprint arXiv:2508.18669, 2025b. URL https: //arxiv.org/abs/2508.18669. Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. arXiv preprint arXiv:2402.19446, 2024. URL https://arxiv.org/abs/2402. 19446. Yifei Zhou, Song Jiang, Yuandong Tian, Jason We- ston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks. arXiv preprint arXiv:2503.15478, 2025. URL https://arxiv.org/abs/2503. 15478. Rongzhi Zhu, Xiang Huang, Yuchuan Wu, Rui Wang, Zequn Sun, Tao Ren, Weiyao Luo, Bingxue Qiu, Jieping Ye, Yongbin Li, and Wei Hu. EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management. arXiv preprint arXiv:2606.15532, 2026. URL https://arxiv. org/abs/2606.15532. Zefang Zong, Dingwei Chen, Yang Li, Qi Yi, Bo Zhou, Chengming Li, B. O. Qian, Peng Chen, and Jie Jiang. AT 2 PO: Agentic Turn-based Policy Optimization via Tree Search. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24120–24143, 2026. doi: 10. 18653/v1/2026.acl-long.1106. URL https://aclanthology. org/2026.acl-long.1106/. 10 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context A Zero-Shot Transfer Protocols A.1 Pare-Bench The four plotted RL checkpoints were trained only on the Airline and Retail training splits of휏-bench; no Pare-Bench scenario was used for training, checkpoint selection, or prompt tuning. All use the same 143-scenario full split. DeepSeek-V4-Flash is the active user with thinking enabled and high reasoning effort; evaluated agents use temperature 0, thinking disabled, and a 2,048-token response limit. The same checkpoint powers the Observe and Execute roles. Strict denominators retain all exceptions as failures. Inter- active GRPO versus Faca yields Pass@1 counts of 9/143 versus 15/143 at 8B and 15/143 versus 19/143 at 14B. The corresponding Pass@4 counts are 27/143 versus 35/143 and 43/143 versus 47/143. Figure 3 is restricted to the scale-matched RL methods under the same formal protocol. This design tests whether reaction-grounded credit preserves its ordering after a proto- col shift; it is not a comparison of the complete pretraining and optimization ladder. A.2 Collaborative Gym The Co-Gym evaluation uses its simulated collaborative con- dition [Shao et al., 2024a]. Each checkpoint is evaluated on 312 instances: 102 Travel Planning, 100 Related Work Writ- ing, and 110 Tabular Analysis tasks. These environments require the agent and simulated user to communicate while acting in shared editors, search interfaces, or notebooks under a non-turn-taking notification protocol. No Co-Gym instance is used for training, checkpoint selection, or prompt tuning. Table 3 reports the matched Interactive-GRPO/Faca pairs on a 0–100 scale. Overall DR improves by 5.5 points at 8B and 2.2 points at 14B; Overall CS improves by 3.3 and 3.7 points. The evaluator-provided overall values are rescaled by 100 for presentation rather than reconstructed from rounded task-level cells.퐻 init was not produced by this evaluation, so we do not infer initiative balance from these trajectories. As in Pare-Bench, the comparison is restricted to the matched RL pair. B Reaction Mapping and Extraction Missing, malformed, or ambiguous strategy metadata maps to neutral and creates no signed process credit. The mapping is an inductive bias: reveal-piece can reward unnecessary questioning, challenge-solution can follow a correct refusal, change-mind can be exogenous, and close partly overlaps terminal success. The terminal branch remains responsible for verified world-state correctness. C Shared Frozen-User Prompt All main-table Interactive GRPO and Faca runs use the same frozen-simulator path (USERSIM_PLAIN=0) and the following semantic prompt. The simulator emits a private reaction strategy and visible utterance in one response; only the utterance enters the agent context. Frozen User Simulator Prompt You are simulating a USER talking to a customer- service agent. Instruction (your situation and goal): INTENT Your job: Produce the user’s NEXT single reply to the agent, together with a short private reasoning, as a JSON object. Rules: • Reply with ONE short, natural customer message at a time. Do not dump the whole instruction; reveal only what the current step needs. •NEVER invent information that is not in the Instruction. If the agent asks for an absent ID, email, name, or date, say you do not have it. • Only state facts in the Instruction. Stay faithful to the goal. •If the goal is fully satisfied, setutteranceto ###STOP###. •Use your own words; do not quote the Instruction verbatim. Output ONLY a JSON object with exactly these keys: "goal_recap": "true goal including key entities", "progress": "what the agent has learned so far", "withheld": "unrevealed facts, or none", "strategy": "ONE of: be_vague | reveal_piece | ask_clarification | change_mind | challenge_solution | confirm | close", "utterance": "one short natural customer message" Strategy guide: • reveal_piece: give one requested fact. • confirm: agree with the right proposed action. • close: goal satisfied; utterance = ###STOP###. • ask_clarification : ask the agent to explain an unclear response. • challenge_solution: push back on a wrong pro- posal. 11 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Frozen User Simulator Prompt (continued) • change_mind: realize a different need (rare). • be_vague : remain unsure or withhold information (rare). Theutteranceis the ONLY field visible to the agent. Interactive GRPO discards the remaining fields, whereas Faca readsstrategyonly after rollout construction. For the simulator view, original system and raw tool messages are dropped, roles are swapped, empty tool-call-only assistant messages are removed, and prior private reasoning is stripped. Structured tool names, arguments, and raw results are not shown to the user simulator. D Training and Evaluation Details D.1 SFT and RL Configuration We use Qwen3-8B and Qwen3-14B [Yang et al., 2025]. Both backbones are cold-started on the public MUA-RL release, which contains 1,580 annotated multi-turn tool-use trajectories spanning five mock and four MCP tasks [Zhao et al., 2025a]. Each trajectory pairs a dialogue with its tool schema. SFT supervises assistant responses and tool calls while masking user and tool-result tokens, thereby initializing interaction and tool-use behavior before RL. For each scale, Interactive GRPO and Faca are alternative continuations from the same SFT checkpoint, with three independent training seeds per method. Both use only the Airline and Retail training splits of휏-bench, the frozen DeepSeek-V4-Flash user path [DeepSeek-AI et al., 2026], and agent-only optimization. All reported DeepSeek-backed runs use the frozen service version available before the July 31, 2026 update. For 8B, the prompt batch size is 16, rollout group size is퐾 = 8, learning rate is10 −6 , and KL coefficient is10 −3 . All reported RL runs use the step- 120 checkpoint. Faca uses true U2U segments, immediate reactions without backward propagation, ordinal turn-index anchors, and 휆 = 0.5. The matched RL arms share the simulator prompt (USERSIM_PLAIN=0), agent-visible utterances, training data, rollout construction, optimizer, and horizon. The simulator emits the same private strategy metadata and visible utter- ance in both arms. Interactive GRPO ignores the metadata and broadcasts only 퐴 o ; Faca additionally uses 퐴 p . D.2 Evaluation and Run Aggregation For the휏-bench family, pass@1 is determined from the veri- fied final environment state under the official non-thinking protocol. Strict denominators are 50/115 for휏, 50/114/114 for휏 2 , and 50/114/114/97 for휏 3 ; unscored tasks remain failures. Each of the three independently trained step-120 checkpoints is evaluated once under the same protocol and is reused across every휏domain rather than selecting domain- specific peaks. Domain entries in Table 2 are arithmetic means over these runs; Avg. is the unweighted mean of the nine domains. Its sample standard deviation is computed across the three run-level averages. Because every trained checkpoint has one evaluation, this dispersion may include residual evaluation stochasticity and does not separately identify either component. Pare-Bench and Co-Gym use one designated step-120 checkpoint per scale and method and are reported sepa- rately from the three-run휏aggregate. The complete OOD protocols and coverage are given in Appendix A. D.3 Seed-Level Results and Ablations Table 5 gives the complete run-level source counts behind the primary휏comparison. Here s1–s3 denote independently trained RL checkpoints with distinct training seeds, not repeated evaluations of one checkpoint. Each step-120 checkpoint is evaluated once under the same strict protocol. Consequently, the run-level standard deviations reported in Section 5 measure dispersion across complete evaluations and may include residual evaluation stochasticity; they do not separately identify either component. Matching methods by training-seed identifier, the nine- domain improvements are 4.67, 6.33, and 6.75 points at 8B and 8.62, 10.14, and 11.91 points at 14B. The Aligned (휆 = 0.5) and Outcome-only rows in Table 4 are exactly the 8B s1 checkpoints listed here. The remaining ablation conditions use the same reference experiment family and are evaluated once, keeping the ablation internally con- trolled while the main table estimates performance across independent training runs. E Strategy and Credit Telemetry The step-120 Faca logs contain120× 128 = 15,360trajec- tories at each scale. The derived telemetry summaries verify 16 prompt groups per step and reproduce the aggregates in Table 6. The zero consistency/span/debug counts are mechanical implementation checks, not evidence that the reaction labels are semantically correct. A configuration-matched 8B replication augments the rollout logs with one record for every U2U segment. Its 120 telemetry files contain 154,219 records; at every step, the file count equals the tensor-level segment count, with zero embedded-step mismatches or telemetry write errors. Each record includes the training step, prompt group, rollout, U2U ordinal, domain, strategy, mapped polarity, anchor size, normalized process advantage, and a nonzero-credit indicator. Table 7 therefore reports full per-U2U frequencies rather than the terminal representative stored by the original main-run rollout logs. Positive reactions rise from 66.17% to 79.51% in Airline and from 71.71% to 84.18% in Retail. Among positive segments, nonzero-credit coverage never- theless decreases from 85.20% in steps 1–40 to 80.51% in steps 81–120; negative-segment coverage remains above 12 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context (a) Qwen3-8B raw successes / nominal tasks MethodT1 Air.T1 Ret.T2 Air.T2 Ret.T2 Tel.T3 Air.T3 Ret.T3 Tel.T3 BankPooledAvg. Base10/5054/1157/5046/1145/1147/5041/11424/1143/97197/81822.20 MUA-SFT7/5045/11512/5041/1143/1148/5046/11433/1143/97198/81822.68 Interactive GRPO (s1)14/5060/11516/5053/11435/11420/5057/11434/1143/97292/81834.70 Interactive GRPO (s2)15/5058/11515/5055/11433/11420/5059/11436/1143/97294/81834.89 Interactive GRPO (s3)13/5062/11517/5051/11437/11419/5055/11432/1144/97290/81834.39 Faca (s1)17/5049/11518/5062/11447/11418/5061/11461/1143/97336/81839.37 Faca (s2)20/5052/11519/5060/11449/11421/5063/11459/1143/97346/81841.22 Faca (s3)19/5050/11520/5064/11445/11421/5059/11463/1144/97345/81841.14 (b) Qwen3-14B raw successes / nominal tasks MethodT1 Air.T1 Ret.T2 Air.T2 Ret.T2 Tel.T3 Air.T3 Ret.T3 Tel.T3 BankPooledAvg. Base6/5068/11513/5056/1143/1147/5057/11427/1143/97240/81826.63 MUA-SFT7/5057/11515/5054/1142/11411/5049/11419/1144/97218/81825.38 Interactive GRPO (s1)18/5067/11517/5079/11430/11420/5074/11458/1144/97367/81842.64 Interactive GRPO (s2)19/5065/11518/5076/11432/11419/5071/11460/1144/97364/81842.48 Interactive GRPO (s3)17/5069/11516/5081/11428/11421/5076/11455/1143/97366/81842.40 Faca (s1)14/5065/11518/5071/11496/11419/5075/11495/1147/97460/81851.26 Faca (s2)22/5063/11519/5074/11492/11421/5072/11491/1146/97460/81852.62 Faca (s3)21/5067/11520/5068/11498/11421/5077/11497/1148/97477/81854.31 Table 5: Raw counts from the independent training runs underlying Table 2. For each RL method, s1–s3 are distinct training seeds that produce separate step-120 checkpoints, each evaluated once. The seed labels align the two RL methods by training-seed identifier; the main table averages these runs domainwise. Base and SFT are fixed pre-RL references. Pooled counts sum successes across unequal domain sizes and are therefore not the primary metric; Avg. gives each of the nine domains equal weight. All strict scores retain the nominal denominator. 0306090120 Training step 20 30 40 50 60 Process share (%) Qwen3-8BQwen3-14B Figure 6: Process-credit share over the common first 120 training steps. Faint lines show per-step values and solid lines a 10-step moving average. The quantity is a pre-optimization advantage L1 proxy, not optimizer-update or gradient mass. 93%. This further distinguishes raw reaction prevalence from anchor-relative differential credit. Quantities that cannot be reconstructed. The evalu- ated 8B and 14B training logs do not contain the replication’s complete per-U2U schema, so their phasewise reaction distributions cannot be reconstructed. In addition, their singleton-segment counts do not identify the total number of anchors or how many non-singleton anchors have con- stant reactions, so no valid/nonconstant-anchor rate can be reported from the available summaries. Finally,0.5E|퐴 p | andE|퐴 o |precede PPO clipping, log-probability weight- ing, and backpropagation. They are advantage-magnitude proxies, not branch-level update or gradient mass, including within outcome-homogeneous groups, and should not be interpreted as such. F Telecom Audit and Case Study We audit the final 14B s1 Faca and outcome-only Interac- tive GRPO checkpoints in their paired run-level evaluation. Within each benchmark, the two archives share the same 114 task IDs, seeds, initial states, criteria, and scenarios. On휏 2 Telecom, both methods succeed on 23 tasks, Faca alone succeeds on 73, Interactive GRPO alone on 7, and neither on 11 (exact paired McNemar푝 = 5.80× 10 −15 ). On휏 3 Telecom, the corresponding cells are 47, 48, 11, and 8 (푝 = 1.24× 10 −6 ). These are task-level paired audits of the single s1 evaluation; Table 2 instead averages three independently trained checkpoints, each evaluated once. The discordant failures expose two directly recorded modes that clarify how the two training objectives diverge during interaction. Among the Faca-only wins, Interactive GRPO exhausts the environment error budget on 68/73휏 2 tasks and 28/48휏 3 tasks; the remaining 5 and 20 end in a normal user stop without satisfying the task. Conversely, all 7 and 11 Interactive-GRPO-only wins correspond to unsuccessful user stops for Faca, indicating that its gains do not eliminate every failure mode. Table 8 keeps the negative 휏 3 Service slice visible rather than treating the Telecom gain as uniform. To avoid selecting a success that merely receives more 13 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context MetricCount unit8B (120 steps)14B (120 steps) Training coveragesteps× rollouts/step120× 128120× 128 Complete trajectoriestrajectory15,36015,360 All-correct groupsgroup-step224 (11.67%)212 (11.04%) All-wrong groupsgroup-step530 (27.60%)384 (20.00%) Mixed groupsgroup-step1,166 (60.73%)1,324 (68.96%) Outcome-homogeneous groupsgroup-step754 (39.27%)596 (31.04%) U2U segmentssegment145,214116,022 Singleton-anchor segmentssegment5,507 (3.79%)3,760 (3.24%) Multi-span U2U segmentssegment61,965 (42.67%)73,105 (63.01%) Process/outcome L1 proxyratio0.5630.510 Process component shareadvantage L1 proxy36.00%33.79% Weighted process > outcometraining step6/1201/120 Consistency/span/debug failuresevent0 / 0 / 00 / 0 / 0 Table 6: Recoverable telemetry from the step-120 Faca runs. A group-step is one prompt group at one optimizer step (16 groups per step). Singleton statistics count segments assigned to singleton anchors; they are not a nonconstant-anchor rate. Process quantities use0.5E|퐴 p | andE|퐴 o | before PPO clipping and are advantage-magnitude proxies rather than optimizer-update or gradient mass. StrategyPolaritySteps 1–40Steps 41–80Steps 81–120Overall reveal_piecepositive36.3036.9036.1836.46 confirmpositive26.4832.5537.2331.91 be_vagueneutral17.2311.378.1812.41 closepositive7.178.119.108.10 ask_clarificationnegative5.334.113.054.20 challenge_solutionnegative3.102.812.392.78 invalidneutral2.432.071.832.12 change_mindnegative1.952.072.042.02 U2U segmentscount53,93551,39748,887154,219 Table 7: Complete per-U2U strategy distribution (%) in the telemetry-enabled 8B replication. Unlike a terminal representative field, every row here uses the U2U segment as its verified count unit. The largest phasewise changes areconfirm(+10.75points) andbe_vague (−9.06 points). interaction time, the complete trace below uses a휏 2 Faca- only win in which Faca has fewer stored events, assistant messages, and tool calls than Interactive GRPO, providing a stricter qualitative comparison. F.1 Matched Multi-Fault Telecom Case We compare matched trajectories for a휏 2 Telecom task that requires four user-side device-setting changes followed by a user-run speed test. Faca completes the required changes and both terminal checks, reaching 275 Mbps. Interactive GRPO reaches the same four state changes, but its fifth rejected agent-issued call to a user-side device tool exhausts the error budget at the final speed-test step, so the run ends before terminal verification. The two trajectories contain 48 versus 54 stored events, 18 versus 21 assistant messages, and 15 versus 17 tool calls, respectively. F.1.1 Successful FACA trajectory Faca (successful; reward 1) T00 Agent Hi! How can I help you today? T01 User Hi, I’m having some trouble with my mobile data. Sometimes it just stops working altogether, and other times it’s really slow. I want to get this fixed and I absolutely need excellent internet speed on my phone—not just "okay" or "good." Can you help me with that? T02 Agent Sure, I can help you with that. Let’s start by identifying your account. Could you please provide me with your phone number or customer ID? If you don’t have your customer ID, I can also look up your account using your full name and date of birth. T03 User My phone number is 555-123-2002. My name is John Smith. T04 Agent action get_customer_by_phone Arguments: "phone_ number": "555-123-2002" T05 Tool result to agent "customer_id": "C1001", "full_name": "John Smith", "date_of_birth": "1985-06-15", "email": "john.smith@example.com", "phone_number": "555-123-2002", "address": "street": "123 Main St", "city": "Anytown", "state": "CA", "zip_code": "90210", "account_status": "Active", "payment_methods": ["method_type": "Credit Card", "account_number_last_4": "1235", "expiration_date": "12/2026"], "line_ids": ["L1001", "L1002", "L1003"], "bill_ids": ["B1001", "B1002", "B1003"], "created_at": 14 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context BenchmarkSubset푁FacaInt. GRPOFACA-only / GRPO-only 휏 2 Telecom Overall11496/11430/11473 / 7 Mobile3632/3614/3620 / 2 Service2923/298/2919 / 4 MMS4941/498/4934 / 1 휏 3 Telecom Overall11495/11458/11448 / 11 Mobile3633/3622/3613 / 2 Service2920/2924/294 / 8 MMS4942/4912/4931 / 1 Table 8: Final-14B paired Telecom audit on the s1 run-level evaluation. The two methods share task IDs, evaluation seeds, initial states, criteria, and scenarios. The last column reports discordant task counts. The휏 3 Service reversal shows that the checkpoint-level gain is not uniform within Telecom. Faca trajectory (continued) "2025-01-15 10:30:00", "last_extension_date": null, "goodwill_credit_used_this_year": 25.0 T06 Agent action get_details_by_id Arguments: "id": "L1001" T07 Tool result to agent "line_id": "L1001", "phone_number": "555-123-2001", "status": "Active", "plan_id": "P1001", "device_id": "D1001", "data_used_gb": 3.2, "data_ refueling_gb": 0.0, "roaming_enabled": false, "contract_ end_date": "2026-12-31", "last_plan_change_date": "2025- 01-10", "last_sim_replacement_date": null, "suspension_ start_date": null T08 Agent action get_details_by_id Arguments: "id": "L1002" T09 Tool result to agent "line_id": "L1002", "phone_number": "555-123-2002", "status": "Active", "plan_id": "P1002", "device_id": "D1002", "data_used_gb": 8.7, "data_ refueling_gb": 0.0, "roaming_enabled": true, "contract_ end_date": "2026-12-31", "last_plan_change_date": "2024- 12-15", "last_sim_replacement_date": "2025-01-20", "suspension_start_date": null T10 Agent action get_details_by_id Arguments: "id": "L1003" T11 Tool result to agent "line_id": "L1003", "phone_number": "555-123-2003", "status": "Suspended", "plan_id": "P1001", "device_id": "D1003", "data_used_gb": 0.0, "data_ refueling_gb": 0.0, "roaming_enabled": false, "contract_ end_date": "2026-06-30", "last_plan_change_date": "2024- 10-05", "last_sim_replacement_date": null, "suspension_ start_date": "2025-02-01" T12 Agent action get_details_by_id Arguments: "id": "P1001" T13 Tool result to agent "plan_id": "P1001", "name": "Basic Plan", "data_limit_gb": 5.0, "price_per_month": 40.0, "data_refueling_price_per_gb": 5.0 T14 Agent action get_details_by_id Arguments: "id": "P1002" T15 Tool result to agent "plan_id": "P1002", "name": "Premium Plan", "data_limit_gb": 15.0, "price_per_month": 65.0, "data_refueling_price_per_gb": 2.0 T16 Agent action get_data_usage Arguments: "customer_id": "C1001", "line_id": "L1002" T17 Tool result to agent "line_id": "L1002", "data_used_gb": "8.7", "data_limit_gb": "15.0", "data_refueling_gb": "0.0", "cycle_end_date": "2025-02-28" T18 Agent I see. You’re on the **Premium Plan (P1002)** with a **15 GB data limit**, and you’ve used **8.7 GB** so far in this cycle, which is under the limit. However, your data speed is slow or sometimes not working at all. Let’s troubleshoot this step by step. ### Step 1: Check Your Network Mode Preference Your phone might be connecting to a slower network (like 2G or 3G) instead of 4G or 5G. Let’s check your network mode preference. T18 Agent action check_network_mode_preference Arguments: Faca trajectory (continued) T19 Tool result to agent Error: Tool ’check_network_mode_ preference’ not found. T20 Agent I apologize for the confusion earlier. Since I can’t directly access your device, I’l guide you through the steps you can take on your phone to check and adjust your network mode preference. ### How to Check Your Network Mode Preference (Manual Steps): 1. **Open your phone’s Settings**. 2. Go to **Network & Internet** or **Connections** (this may vary slightly depending on your phone’s brand and OS). 3. Look for **Mobile Network** or **Cellular** settings. 4. Tap on **Network Mode** or **Preferred Network Type**. 5. Check what network type is selected (e.g., 5G, 4G, 3G, 2G). ### If You’re on 2G or 3G: - Change the network mode to **4G/5G preferred** to get faster speeds. - After changing, **reboot your phone** to ensure the changes take effect. Would you like me to guide you through other steps to improve your data speed? T21 User Okay, let me check that on my phone. T21 User action check_network_mode_preference Arguments: T22 Tool result to user Network Mode Preference: 2g_only T23 User It says my phone is currently set to "2G only." Is that right? Would changing it to something like "4G/5G preferred" help me get excellent internet speed? T24 Agent Yes, that’s correct. If your phone is set to **"2G only"**, it will only connect to 2G networks, which are significantly slower than 4G or 5G. Changing your network mode preference to **"4G/5G preferred"** will allow your phone to connect to faster networks, which should improve your internet speed. ### Steps to Change Your Network Mode Preference: 1. **Open your phone’s Settings**. 2. Go to **Network & Internet** or **Connections** (this may vary slightly depending on your phone’s brand and OS). 3. Look for **Mobile Network** or **Cellular** settings. 4. Tap on **Network Mode** or **Preferred Network Type**. 5. Select **"4G/5G preferred"** or a similar option. 6. **Reboot your phone** to ensure the changes take effect. After making this change, your phone should connect to a faster network, and your internet speed should improve significantly. Would you like me to help you with anything else related to your mobile data issue? T25 User Let me try changing it to "4G/5G preferred" from my phone. T25 User action set_network_mode_preference Arguments: "mode": "4g_5g_preferred" T26 Tool result to user Preferred Network Mode set to: 4g_5g_ preferred Status Bar: [airplane] Airplane Mode | [battery] 80% T27 User I’ve changed my phone’s setting to "4G/5G preferred," but now I’m seeing a little airplane icon on the top of my screen. Does that mean something is wrong? 15 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Faca trajectory (continued) T28 Agent Yes, the **airplane icon** on your screen means that **Airplane Mode is enabled**, which disables all wireless connections, including mobile data and cellular service. This is likely why you’re still experiencing issues with your internet speed or connectivity. ### How to Turn Off Airplane Mode: 1. **Swipe down from the top of your screen** to open the **Quick Settings panel**. 2. Look for the **Airplane Mode icon** (it looks like a small airplane). 3. Tap on it to **disable Airplane Mode**. Alternatively: 1. **Open your phone’s Settings**. 2. Go to **Network & Internet** or **Connections**. 3. Look for **Airplane Mode** and toggle it **off**. After turning off Airplane Mode, your phone should reconnect to the cellular network, and your mobile data should be restored. Would you like me to help you check anything else related to your mobile data issue? T29 User Oh, I see. That must have been turned on by accident. Let me turn that off now. T29 User action toggle_airplane_mode Arguments: T30 Tool result to user Airplane Mode is now OFF. Status Bar: [signal] 4 Excellent | 5G | [no-cellular] Data Disabled | [battery] 80% T31 User I’ve turned off the Airplane Mode, so that airplane icon is gone now. However, I notice on my screen there’s a symbol that looks like a "no data" sign (data disabled). Does that mean my mobile data is turned off? T32 Agent Yes, the **"no data" symbol** (which looks like a crossed-out phone icon or a "data disabled" message) means that **mobile data is currently turned off** on your phone. This would explain why you’re still not getting internet access or experiencing slow speeds. ### How to Turn On Mobile Data: 1. **Swipe down from the top of your screen** to open the **Quick Settings panel**. 2. Look for the **Data icon** (it looks like a phone with a signal bar or a "Data" label). 3. Tap on it to **enable mobile data**. Alternatively: 1. **Open your phone’s Settings**. 2. Go to **Network & Internet** or **Connections**. 3. Look for **Mobile Network** or **Cellular** settings. 4. Toggle **Mobile Data** to the **ON** position. After enabling mobile data, your phone should reconnect to the cellular network, and your internet speed should improve. Would you like me to help you check anything else related to your mobile data issue? T33 User Let me turn on the mobile data option. T33 User action toggle_data Arguments: T34 Tool result to user Mobile Data is now ON. Status Bar: [signal] 4 Excellent | 5G | [phone] Data Enabled | [data-saver] Data Saver | [battery] 80% T35 User Mobile data is now turned on. I can see 5G signal and it looks good, but I also see a "Data Saver" icon. Should I worry about that? Can I check my internet speed now? T36 Agent Yes, the **"Data Saver" icon** means that **Data Saver mode is enabled** on your phone. This mode restricts background data usage to help conserve data, which can sometimes make your internet feel slower or less responsive, especially for apps that require a lot of data in the background. ### Should You Worry About It? - **No**, you don’t need to worry. However, if you’re looking for **excellent internet speed**, it’s best to **disable Data Saver mode** temporarily while you’re using your phone for data-heavy tasks (like streaming, gaming, or browsing). ### How to Disable Data Saver Mode: 1. **Open your phone’s Settings**. Faca trajectory (continued) 2. Go to **Network & Internet** or **Connections**. 3. Look for **Data Saver** or **Mobile Data** settings. 4. Toggle **Data Saver** to the **OFF** position. Once Data Saver is disabled, your phone will allow full background data usage, which should improve your internet speed. — ### Let’s Check Your Internet Speed I can help you check your internet speed using the **‘run_speed_test‘** tool. This will give us an idea of how fast your connection is. Would you like me to run the speed test now? T37 User Alright, I’l go ahead and disable Data Saver mode first. T37 User action toggle_data_saver_mode Arguments: T38 Tool result to user Data Saver Mode is now OFF. Status Bar: [signal] 4 Excellent | 5G | [phone] Data Enabled | [battery] 80% T39 User Okay, Data Saver mode is now turned off. Should I check my internet speed now to see if it’s excellent? T40 Agent action run_speed_test Arguments: T41 Tool result to agent Error: Tool ’run_speed_test’ not found. T42 Agent I apologize for the confusion earlier. Since I can’t directly run a speed test on your phone, I can guide you through how to check your internet speed manually. ### How to Check Your Internet Speed Manually: 1. **Open a web browser** on your phone (e.g., Chrome, Safari). 2. Go to a **speed test website** like [Speedtest by Ookla](https://w.speedtest.net/). 3. Tap **"Begin Test"** to run the speed test. 4. The test will show your **download speed**, **upload speed**, and **latency**. ### What to Look For: - **Download Speed**: For excellent internet speed, you should see at least **100 Mbps** or higher. - **Latency**: A latency of **under 50 ms** is ideal for smooth browsing and streaming. If your speed is still slow after disabling Data Saver mode and connecting to 5G, we can explore other options like checking your APN settings or looking into your data plan. Would you like me to help you check your APN settings or look into your data plan? T43 User Let me run a speed test using my phone. T43 User action run_speed_test Arguments: T44 Tool result to user Speed Test Result: 275.00 Mbps (Excellent). Connection is very fast. T45 User I ran a speed test, and it says the connection is excellent—275 Mbps! That’s really fast. I think my mobile data is working perfectly now. Thank you so much for your help! T46 Agent You’re very welcome! I’m glad to hear that your mobile data is now working perfectly and that your internet speed is excellent at **275 Mbps**. That’s more than enough for streaming, gaming, and browsing without any issues. If you ever run into any more problems with your mobile data or need help with anything else, feel free to reach out. Have a great day! [smile] T47 User Thank you! I really appreciate your patience and guidance. Have a great day! [smile] ###STOP### 16 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context F.1.2 Failed Interactive-GRPO trajectory Interactive GRPO (failed; reward 0) T00 Agent Hi! How can I help you today? T01 User Hi, I’m hoping you can help me. My mobile data isn’t working right—sometimes it stops working or it’s really slow. I really need great internet speed on my phone when I’m out and about. Can you help me get this fixed? I don’t have Wi-Fi at home, so I’m using my mobile data. T02 Agent action check_network_status Arguments: T03 Tool result to agent Error: Tool ’check_network_status’ not found. T04 Agent I’m sorry for the inconvenience. I need to check your phone’s network status to help resolve the issue. However, I currently don’t have access to that tool. Could you please provide your phone number so I can look up your account and help further? T05 User Sure, my phone number is 555-123-2002. T06 Agent action get_customer_by_phone Arguments: "phone_ number": "555-123-2002" T07 Tool result to agent "customer_id": "C1001", "full_name": "John Smith", "date_of_birth": "1985-06-15", "email": "john.smith@example.com", "phone_number": "555-123-2002", "address": "street": "123 Main St", "city": "Anytown", "state": "CA", "zip_code": "90210", "account_status": "Active", "payment_methods": ["method_type": "Credit Card", "account_number_last_4": "1235", "expiration_date": "12/2026"], "line_ids": ["L1001", "L1002", "L1003"], "bill_ids": ["B1001", "B1002", "B1003"], "created_at": "2025-01-15 10:30:00", "last_extension_date": null, "goodwill_credit_used_this_year": 25.0 T08 Agent action get_details_by_id Arguments: "id": "L1001" T09 Tool result to agent "line_id": "L1001", "phone_number": "555-123-2001", "status": "Active", "plan_id": "P1001", "device_id": "D1001", "data_used_gb": 3.2, "data_ refueling_gb": 0.0, "roaming_enabled": false, "contract_ end_date": "2026-12-31", "last_plan_change_date": "2025- 01-10", "last_sim_replacement_date": null, "suspension_ start_date": null T10 Agent action get_details_by_id Arguments: "id": "L1002" T11 Tool result to agent "line_id": "L1002", "phone_number": "555-123-2002", "status": "Active", "plan_id": "P1002", "device_id": "D1002", "data_used_gb": 8.7, "data_ refueling_gb": 0.0, "roaming_enabled": true, "contract_ end_date": "2026-12-31", "last_plan_change_date": "2024- 12-15", "last_sim_replacement_date": "2025-01-20", "suspension_start_date": null T12 Agent action get_details_by_id Arguments: "id": "L1003" T13 Tool result to agent "line_id": "L1003", "phone_number": "555-123-2003", "status": "Suspended", "plan_id": "P1001", "device_id": "D1003", "data_used_gb": 0.0, "data_ refueling_gb": 0.0, "roaming_enabled": false, "contract_ end_date": "2026-06-30", "last_plan_change_date": "2024- 10-05", "last_sim_replacement_date": null, "suspension_ start_date": "2025-02-01" T14 Agent action get_details_by_id Arguments: "id": "P1001" T15 Tool result to agent "plan_id": "P1001", "name": "Basic Plan", "data_limit_gb": 5.0, "price_per_month": 40.0, "data_refueling_price_per_gb": 5.0 T16 Agent action get_details_by_id Arguments: "id": "P1002" T17 Tool result to agent "plan_id": "P1002", "name": "Premium Plan", "data_limit_gb": 15.0, "price_per_month": 65.0, "data_refueling_price_per_gb": 2.0 T18 Agent I see that you’re using the line with phone number **555-123-2002**, which is associated with the **Premium Plan (P1002)**. This plan provides **15 GB of data per month**. However, your current data usage is **8.7 GB**, which is under the limit. Since your data usage is within the plan’s limit, the issue might be related to **network performance**, **data roaming**, or **device settings**. Let’s check a few things: 1. **Network Mode Preference**: Your phone might be connecting to a slower network (e.g., 2G/3G) instead of 4G/5G. I can check your network mode preference. Interactive GRPO trajectory (continued) 2. **Data Saver Mode**: If Data Saver is enabled, it might be restricting background data usage, which could slow down your internet speed. 3. **Roaming Status**: Your line has roaming enabled, but if you’re in an area with poor signal, it might be affecting your speed. Would you like me to check your network mode preference or data saver status? T19 User Thank you for checking. To be honest, I get a bit lost with all these technical details. Could you please tell me what you’d like me to do first? I’l try my best to follow your instructions. T20 Agent action check_network_mode_preference Arguments: T21 Tool result to agent Error: Tool ’check_network_mode_ preference’ not found. T22 Agent I’m sorry for the inconvenience. I need to check your phone’s network mode preference to help resolve the issue. However, I currently don’t have access to that tool. Could you please try the following steps on your phone? 1. **Check Network Mode Preference**: - Go to your phone’s settings. - Look for "Network & Internet" or "Connections." - Tap on "Mobile Network" or "Cellular." - Check the "Network Mode" or "Preferred Network Type" setting. - Ensure it is set to "4G/5G" or "4G LTE/5G" (not 2G or 3G). 2. **Check Data Saver Mode**: - Go to your phone’s settings. - Look for "Network & Internet" or "Connections." - Tap on "Data Usage" or "Mobile Data." - Check if "Data Saver" is enabled. If it is, disable it. Would you like me to guide you through these steps in more detail? T23 User I’l try to check what you suggested, but I’m not too comfortable navigating through the settings on my own. Could you please guide me one step at a time? I’d rather not get confused with too many steps at once. T24 Agent Sure, I can guide you step by step. Let’s start with checking your **Network Mode Preference**. ### Step 1: Check Network Mode Preference 1. Open the **Settings** app on your phone. 2. Scroll down and tap on **Network & Internet** or **Connections** (the exact name may vary slightly depending on your phone’s manufacturer). 3. Tap on **Mobile Network** or **Cellular**. 4. Look for an option called **Network Mode**, **Preferred Network Type**, or **Network Selection**. 5. Tap on it to see what network type your phone is currently using (e.g., 5G, 4G, 3G, 2G). 6. If it’s set to **2G** or **3G**, we can change it to **4G/5G** or **4G LTE/5G** for better speed. Would you like me to guide you through this step-by-step on your phone? T25 User Let me have a look at what my phone shows for the network mode. T25 User action check_network_mode_preference Arguments: T26 Tool result to user Network Mode Preference: 2g_only T27 User I just checked, and it looks like my phone is set to 2G only. I’m not really sure how that happened. What should I do next? T28 Agent action set_network_mode_preference Arguments: "mode": "4g_5g_preferred" T29 Tool result to agent Error: Tool ’set_network_mode_ preference’ not found. T30 Agent I’m sorry for the inconvenience. I need to change your phone’s network mode preference to **4G/5G** to improve your data speed. However, I currently don’t have access to that tool. Could you please follow these steps on your phone to change the network mode preference manually? ### Step 2: Change Network Mode Preference 1. Open the **Settings** app on your phone. 2. Scroll down and tap on **Network & Internet** or **Connections**. 3. Tap on **Mobile Network** or **Cellular**. 4. Look for an option called **Network Mode**, **Preferred Network Type**, or **Network Selection**. 5. Tap on it and select **4G/5G** or **4G LTE/5G** (not 2G or 3G). 6. Save your changes. 17 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context Interactive GRPO trajectory (continued) After changing the network mode, your phone should connect to a faster network, which should improve your data speed. Would you like me to guide you through this step-by-step on your phone? T31 User Let me try to change that on my phone now. T31 User action set_network_mode_preference Arguments: "mode": "4g_5g_preferred" T32 Tool result to user Preferred Network Mode set to: 4g_5g_ preferred Status Bar: [airplane] Airplane Mode | [battery] 80% T33 User I think I managed to change it to 4G/5G preferred. But I noticed something on my phone—there’s an airplane icon showing at the top of the screen. Could that be causing any problems? T34 Agent action check_status_bar Arguments: T35 Tool result to agent Error: Tool ’check_status_bar’ not found. T36 Agent I’m sorry for the inconvenience. I need to check your phone’s status bar to see if **Airplane Mode** is enabled, which could be causing your mobile data issues. However, I currently don’t have access to that tool. Could you please check the top of your phone’s screen for an **Airplane Mode icon** (it looks like a small airplane or "[airplane] Airplane Mode" text)? If you see it, that means Airplane Mode is enabled, which disables all wireless connections, including mobile data. ### Step 3: Check and Disable Airplane Mode 1. Look at the top of your phone’s screen. 2. If you see an **Airplane Mode icon** or the text "[airplane] Airplane Mode," it means Airplane Mode is enabled. 3. To disable it: - Tap the **Airplane Mode icon**. - Alternatively, go to **Settings** > **Network & Internet** > **Airplane Mode** and toggle it off. After disabling Airplane Mode, your phone should reconnect to the cellular network, and your mobile data should be restored. Would you like me to guide you through this step-by-step on your phone? T37 User You’re right, I can see a little airplane icon at the top of my screen, so it looks like Airplane Mode is on. How do I turn it off—should I just tap it or go to the settings? T38 Agent You can **tap the airplane icon** directly to toggle Airplane Mode off. However, if you’re unsure or the icon doesn’t respond, you can also go to your phone’s **Settings** and disable it from there. ### Step-by-Step Instructions to Turn Off Airplane Mode: 1. **Option 1 (Tap the Icon):** - Tap the **airplane icon** at the top of your screen. - This should toggle Airplane Mode off, and the icon should disappear. 2. **Option 2 (Settings):** - Open the **Settings** app. - Scroll down and tap on **Network & Internet** or **Connections**. - Tap on **Airplane Mode**. - Toggle the switch to **Off**. After turning off Airplane Mode, your phone should reconnect to the cellular network, and your mobile data should be restored. Would you like me to guide you through this step-by-step on your phone? T39 User action toggle_airplane_mode Arguments: T40 Tool result to user Airplane Mode is now OFF. Status Bar: [signal] 4 Excellent | 5G | [no-cellular] Data Disabled | [battery] 80% T41 User Okay, I turned off Airplane Mode by tapping the icon. Now the airplane icon is gone, but I also see something that says "Data Disabled" on my screen. Does that mean my mobile data is turned off? T42 Agent Yes, the message **"Data Disabled"** means that your **mobile data is currently turned off**. This would prevent your phone from using cellular data for internet access, even if you’re connected to a 4G/5G network. ### Step 4: Enable Mobile Data Let’s enable your mobile data so that your phone can use the cellular network for internet access. Interactive GRPO trajectory (continued) #### Option 1: Toggle Mobile Data from the Status Bar 1. Look at the top of your phone’s screen. 2. Tap the **"Data Disabled"** icon (it might look like a phone signal with a slash through it or a similar symbol). 3. This should toggle mobile data **ON**, and the icon should change to "Data Enabled" or similar. #### Option 2: Enable Mobile Data from Settings 1. Open the **Settings** app. 2. Scroll down and tap on **Network & Internet** or **Connections**. 3. Tap on **Mobile Network** or **Cellular**. 4. Look for an option called **"Mobile Data"** or **"Data Usage"**. 5. Toggle the switch to **ON**. After enabling mobile data, your phone should reconnect to the cellular network, and your internet speed should improve. Would you like me to guide you through this step-by-step on your phone? T43 User action toggle_data Arguments: T44 Tool result to user Mobile Data is now ON. Status Bar: [signal] 4 Excellent | 5G | [phone] Data Enabled | [data-saver] Data Saver | [battery] 80% T45 User I’ve turned mobile data on. Now I see "Data Enabled" at the top, but there’s also an icon that says "Data Saver." Could that be slowing things down? T46 Agent Yes, the **"Data Saver"** icon can affect your data speed and performance. When **Data Saver mode is ON**, it restricts background data usage, which can make some apps or services seem slow or unresponsive. ### Step 5: Check and Disable Data Saver Mode Let’s check if **Data Saver mode is ON** and disable it if needed. #### Option 1: Toggle Data Saver from the Status Bar 1. Look at the top of your phone’s screen. 2. Tap the **"Data Saver"** icon (it might look like a downward arrow or a similar symbol). 3. This should toggle **Data Saver mode OFF**, and the icon should disappear or change to indicate it’s disabled. #### Option 2: Disable Data Saver from Settings 1. Open the **Settings** app. 2. Scroll down and tap on **Network & Internet** or **Connections**. 3. Tap on **Data Usage** or **Mobile Data**. 4. Look for an option called **"Data Saver"** or **"Data Saver Mode"**. 5. Toggle the switch to **OFF**. After disabling **Data Saver mode**, your phone should use mobile data more freely, which should improve your internet speed. Would you like me to guide you through this step-by-step on your phone? T47 User action check_data_restriction_status Arguments: T48 Tool result to user Data Saver mode is ON (limits data usage). T49 User Yes, I see that Data Saver is on. Let me try turning it off. T49 User action toggle_data_saver_mode Arguments: T50 Tool result to user Data Saver Mode is now OFF. Status Bar: [signal] 4 Excellent | 5G | [phone] Data Enabled | [battery] 80% T51 User Okay, I’ve turned Data Saver mode off. The icon is gone, so I think it’s disabled now. Can I check my internet speed now to see if it’s working as it should? T52 Agent action run_speed_test Arguments: T53 Tool result to agent Error: Tool ’run_speed_test’ not found. Case interpretation and boundary. This pair removes a simple extra-interaction explanation: Faca succeeds with six fewer stored events, three fewer assistant messages, and two fewer tool calls. Both policies nevertheless make execution-authority mistakes. After an invalid agent-side network-mode check, Faca hands the device action to the user, who changes the network preference, disables airplane 18 ANT INTERNATIONAL RESEARCHThe Next User Turn Is More Than Context mode, enables mobile data, and disables Data Saver. When Faca later attempts the speed test from the agent side, it again recovers by asking the user to run it; the resulting 275 Mbps measurement satisfies both terminal assertions. Interactive GRPO also elicits the four state-changing user calls, but intersperses five rejected agent-side calls to user- only tools. Its final agent-side speed-test attempt exhausts the error budget before terminal validation. The trace therefore illustrates recovery from repeated execution-authority errors and task closure without a longer trajectory. It remains one task, both policies make such errors, and the baseline failure depends partly on the environment’s fixed error limit; the pair does not by itself identify the causal contribution of reaction-grounded training. G Algorithm and Reproducibility 1.Sample퐾complete user–agent–tool rollouts per prompt and compute terminal rewards. 2. Group all assistant spans between adjacent user mes- sages into one U2U segment and attach the immediately following reaction. 3.Normalize terminal rewards by prompt to obtain퐴 o and reactions by prompt/ordinal U2U to obtain 퐴 p . 4. Broadcast퐴 o + 휆퐴 p to assistant language and generated tool-call tokens; mask user and raw tool-result tokens. 5.Log reaction validity, homogeneous groups, singleton anchors, advantage magnitudes, and segment invariants. Compute and configuration. Each main training run uses one node with eight NVIDIA B200 GPUs (180 GB HBM3e each; 1.44 TB aggregate). Training uses VERL with SGLang rollouts, and evaluation follows the official benchmark im- plementations. Matched arms share the optimizer, batch and rollout settings, horizon, simulator endpoint, and container. We fix휆 = 0.5for the main Faca runs; Table 4 compares 휆 ∈ −0.5, 0, 0.1, 0.5. All methods use the fixed step-120 checkpoint, with no benchmark-specific peak selection. Frozen simulator. Every reported arm trains only the agent; the frozen user server isolates credit assignment from user-policy drift and fabricated task entities. 19

Instant research discovery