Paper deep dive
Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
Chen Xi, Ding Ruixue, Wang Zhigaoyuan, Zhang Fanrui, Zhang Qiang, Chen Boli, Wang Shihang, Wang Tianfu, Ren Ailin, Wang Pengyong, Xie Pengjun, Xiong Hui
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/12/2026, 7:19:22 AM
Summary
The paper introduces Rollout-Time Intervention (RTI), a technique to overcome the on-policy reasoning bottleneck in Reinforcement Learning (RL) for Large Language Model (LLM) agents. RTI uses a multi-agent system to correct failed trajectories during rollout, filters off-policy deviations using perplexity gating, and applies token reweighting to focus learning on critical reasoning shifts. Experiments show RTI outperforms baselines and enables policies to transcend their original capability boundaries.
Entities (10)
Relation Signals (8)
Rollout-Time Intervention → evaluatedon → Claw-Eval
confidence 95% · Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines
Rollout-Time Intervention → evaluatedon → τ-bench
confidence 95% · Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines
Rollout-Time Intervention → evaluatedon → Navigation
confidence 95% · Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines
Rollout-Time Intervention → usescomponent → Validator
confidence 95% · RTI, comprising three components: (1) reward-aware multi-agent intervention, which employs a Validator–Corrector–Verifier pipeline
Rollout-Time Intervention → usescomponent → Corrector
confidence 95% · RTI, comprising three components: (1) reward-aware multi-agent intervention, which employs a Validator–Corrector–Verifier pipeline
Rollout-Time Intervention → usescomponent → Verifier
confidence 95% · RTI, comprising three components: (1) reward-aware multi-agent intervention, which employs a Validator–Corrector–Verifier pipeline
Rollout-Time Intervention → improves → GRPO
confidence 90% · RTI consistently outperforms RL baselines... The final policy-gradient loss applies the reasoning-shift reweighting to the token-level GRPO clipped surrogate.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model's current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.13438v1
- Canonical: https://arxiv.org/abs/2610.13438v1
Trouble viewing inline? Open PDF directly →
Full Text
120,741 characters extracted from source content.
Expand or collapse full text
Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits Xi Chen Affiliation: Alibaba Token Hub, Alibaba Group Ruixue Ding †thanks: Project leader Affiliation: Alibaba Token Hub, Alibaba Group Zhigaoyuan Wang Affiliation: HKUST (Guangzhou) Fanrui Zhang Affiliation: Alibaba Token Hub, Alibaba Group Qiang Zhang Affiliation: Alibaba Token Hub, Alibaba Group Boli Chen Affiliation: Alibaba Token Hub, Alibaba Group Shihang Wang Affiliation: Alibaba Token Hub, Alibaba Group Tianfu Wang Affiliation: HKUST (Guangzhou) Ailin Ren Affiliation: Amap, Alibaba Group Pengyong Wang Affiliation: Amap, Alibaba Group Pengjun Xie Affiliation: Alibaba Token Hub, Alibaba Group Hui Xiong Affiliation: HKUST (Guangzhou) Abstract Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model’s current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy’s intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at https://github.com/Alibaba-NLP/qqr. 1 Introduction LLMs are rapidly evolving from static instruction-following systems into interactive agents that can collaborate with external environments to achieve goals (Team, 2026; Froger et al., 2025). RL has been a key driver of this shift, enabling advanced capabilities such as self-reflection and long-horizon planning (Guo et al., 2025; Qian et al., 2026). These advances have fueled strong performance in challenging agent settings, including mathematical reasoning with code interpreters (Feng et al., 2025), deep research with search engines (Jin et al., 2025; Sun et al., 2026), and complex real-world tool use (Qian et al., 2026; Gao et al., 2026; Yang et al., 2026; Dong et al., 2026b; Li et al., 2026). Figure 1: RTI enables successful trajectories where on-policy and teacher-guided RL fail. Nevertheless, standard RL is limited by its on-policy nature: it mainly amplifies behaviors already supported by the current policy, rather than inducing new reasoning patterns (Zhao et al., 2025; Yue et al., 2026). As a result, when tasks lie beyond the model’s capability boundary, successful trajectories become extremely sparse, leading to unstable training or even collapse (Figure 1(a)). Teacher-guided methods such as LUFFY (Yan et al., 2026) and OPD (Lu & Thinking Machines Lab, 2025) can partially mitigate this issue, but their improvements remain bounded by teacher capability. To further understand these failures, we examine failed agent trajectories and find a recurring pattern: the agent often possesses the knowledge and tools needed to complete the task, but makes a small early reasoning or action error at a task-critical step. As illustrated in Figure 1, in τ2τ^2-Telecom, the agent must guide the user to invoke a user-side tool, yet the model may incorrectly treat it as an agent-accessible tool and repeatedly attempt to call it; in Navigation, user queries are often ambiguous, and the agent must interpret noisy POI search results in the context of the user’s intent rather than blindly treating a returned POI as the target. In both cases, a single incorrect interaction strategy can steer subsequent reasoning into a failure loop, while correcting this early decision is often sufficient for the remaining trajectory to proceed normally. These observations motivate us: if such early reasoning errors can be corrected so that the rollout succeeds, can RL learn from these corrected trajectories and thereby push the policy beyond its original reasoning limits (Figure 1(c))? However, such intervention introduces a central technical challenge: the corrected trajectories are no longer sampled from the original policy, but generated by an intervention-induced behavior policy. This discrepancy poses two fundamental risks. First, privileged reward-side information may leak into the reasoning process, producing shortcut-driven or behaviorally implausible trajectories that could not arise under genuine task interaction. Second, intervention may induce substantial distributional shifts from the original policy. In PPO-style optimization, including GRPO, such shifts can push importance ratios beyond the clipping range, suppressing otherwise useful learning signals, while low-probability actions may yield high-variance gradients that destabilize training. To this end, we propose Rollout-Time Intervention (RTI), comprising three components: (1) reward-aware multi-agent intervention, which employs a Validator–Corrector–Verifier pipeline to transform failed rollouts into successful intervention trajectories while preventing privileged-information leakage; (2) perplexity-based distribution gating, which filters intervened trajectories that deviate excessively from the current policy; and (3) reasoning-shift-aware token reweighting, which concentrates policy updates on critical tokens where intervention induces substantial strategy changes. Together, these components enable stable learning from successful trajectories beyond the policy’s unaided exploration capacity, thereby pushing the policy beyond its original capability boundary. Experiments on public and real-world benchmarks show that RTI substantially outperforms baselines and achieves a breakthrough in the boundaries of reasoning capabilities. 2 Related Work Agentic Reasoning for LLMs. Agentic reasoning reframes LLMs as autonomous agents that combine planning, acting, and learning through interaction (Wei et al., 2026), as exemplified by ReAct (Yao et al., 2023), tool-use and search-based agents, and workflow-oriented systems with verifiable sub-tasks (Yao et al., 2023; Jin et al., 2025; Sun et al., 2026; Feng et al., 2025). This paradigm requires agents to plan over long horizons, navigate partial observability, and actively improve through feedback (Wang et al., 2024). Existing methods for agent training are as follows: (i) Data-synthesis-driven SFT improves tool use by generating interaction trajectories for supervised fine-tuning, including ToolLLM (Qin et al., 2024), ToolACE (Liu et al., 2025), APIGen-MT (Prabhakar et al., 2026), TopoCurate (Yang et al., 2026), and simulation or self-evolving pipelines for multi-turn data generation (Li et al., 2025; Gao et al., 2026). These methods are effective but depend on large-scale trajectory synthesis. (i) Pure RL with verifiable rewards has also shown promise when outcomes are easy to verify. ReTool (Feng et al., 2025) incorporates code execution for competition-level problem solving, Search-R1 (Jin et al., 2025) learns search-engine use with emergent multi-query behavior, ZeroSearch (Sun et al., 2026) replaces real retrieval with LLM simulation, and ToolRL (Qian et al., 2026) shows that fine-grained format and outcome rewards can support stable multi-step tool-use RL. More recently, AEPO (Dong et al., 2026a) further improves optimization stability under long-horizon, multi-turn tool use through entropy-aware policy updates. Teacher-Guided Training for LLMs. When tasks lie beyond the model’s capability boundary, successful rollouts become extremely sparse and the policy is rarely exposed to truly corrective trajectories. To alleviate this, recent work studies teacher-guided training in two forms. One line provides token-level credit assignment using stronger teachers, often via trajectory-level teacher-student KL to derive dense rewards (Gu et al., 2024; Lu & Thinking Machines Lab, 2025). Another line introduces off-policy successful trajectories: LUFFY (Yan et al., 2026), for instance, combines offline expert data with online rollouts and uses importance sampling to handle distribution shift. While these methods can improve exploration and learning, they all fundamentally depend on stronger teacher models, making their gains bounded by the teacher’s capability ceiling. 3 Preliminaries Problem Setting. We formulate the agentic task as a trajectory generation problem. Let T denote the set of tools. Given a query x sampled from a task distribution D, the policy πθ _θ produces a multi-step interaction trajectory τ consisting of interleaved chain-of-thought reasoning zkz_k, tool invocations ak∈a_k , environmental feedback oko_k, and a final answer y: τ=[z1,a1,o1,…,zk,ak,ok,…,y]τ=[z_1,a_1,o_1,…,z_k,a_k,o_k,…,y]. A verifiable reward function r(x,τ)∈0,1r(x,τ)∈\0,1\ evaluates the trajectory. Group Relative Policy Optimization. GRPO (Shao et al., 2024) is a widely adopted RL algorithm. For each query x, GRPO samples a group of G trajectories τii=1G∼πθ(⋅∣x)\ _i\_i=1^G _θ(· x) and normalizes the rewards into per-sample advantages: Ai=r(x,τi)−μrσr+ϵ,μr=1G∑i=1Gr(x,τi),σr=1G∑i=1G(r(x,τi)−μr)2.A_i= r(x, _i)- _r _r+ε, _r= 1G _i=1^Gr(x, _i), _r= 1G _i=1^G(r(x, _i)\!-\! _r)^2. (1) The policy is then updated by maximizing the clipped surrogate objective GRPO(θ)J_GRPO(θ): x,τi[1G∑i=1G1|τi|∑t=1|τi|min(ρgrpo,itAi,clip(ρgrpo,it,1−ϵclip,1+ϵclip)Ai)]−βKL(πθ∥πref), _x,\ _i\\! [ 1G _i=1^G 1| _i| _t=1^| _i| \! ( _grpo,i^tA_i,clip( _grpo,i^t,1- _clip,1+ _clip)\,A_i ) ]- _KL( _θ\| _ref), (2) where ρgrpo,it=πθ(τit∣x,τi<t)πold(τit∣x,τi<t) _grpo,i^t= _θ( _i^t x, _i^<t) _old( _i^t x, _i^<t) is the per-token probability ratio between the current policy and the behavior policy πold _old that generated τi _i, and β>0β>0 controls the strength of the KL regularization. 4 Rollout-Time Intervention To overcome the exploration bottleneck of on-policy RL, we propose RTI, a technique comprising three components as shown in Figure 2. First, reward-aware multi-agent system (MAS) intervention calibrates task-critical errors at the step level, transforming failed rollouts into successful intervention trajectories while preventing privileged-information leakage. Second, perplexity-based distribution gating filters interventions that deviate excessively from the rollout policy, mitigating ineffective or unstable updates caused by severe off-policy shifts. Third, reasoning-shift-aware token reweighting emphasizes the critical tokens responsible for intervention-induced strategy changes, enabling more precise credit assignment. Together, these components enable stable learning from successful trajectories beyond the policy’s unaided exploration capacity. Figure 2: Overview of the proposed RTI framework. 4.1 Reward-Aware MAS-Based Intervention Given a query x, only when the entire group fails (i.e., all G trajectories generated by πθ _θ receive zero reward), we invoke the intervention pipeline, an MAS consisting of a Reward-Oriented Trajectory Validator, a Policy-Aligned Trajectory Corrector, and an Integrity-Constrained Intervention Verifier. By default, all three MAS roles are instantiated with the policy model itself. Reward-Oriented Trajectory Validator The validator identifies whether the current rollout step has harmfully deviated from a valid path to the gold outcome. Given the query x and the observable history before step k: h1:k−1=(z1,a1,o1,…,zk−1,ak−1,ok−1)h_1:k-1=(z_1,a_1,o_1,…,z_k-1,a_k-1,o_k-1), the current response u^k u_k, and the reward-side evidence ℓ , it determines whether u^k:=(zk,ak) u_k:=(z_k,a_k) is still compatible with successful task completion. Importantly, the validator is reward-oriented rather than process-prescriptive: it does not penalize suboptimal but recoverable decisions, alternative valid action orders, or stylistic differences, and instead flags only those errors that are likely to derail the trajectory. In this way, the validator detects only task-critical mistakes, matching our central motivation that many rollout failures arise from early local errors rather than missing end-task capability. The prompt is shown below: Validator prompt (schematic) Inputs. Observable history h1:k−1h_1:k-1; current response u^k u_k; reward-side evidence ℓ . Body. (i) Observable evidence listing prior turns and tool outputs; (i) Required actions derived from ℓ separated into agent-side and (for telecom) user-side groups; (i) Hard rules: false-claim, …; (iv) Negative space explicitly disallowing invalidation due to suboptimal-but-recoverable order. Output. JSON is_valid,error_types,diagnosis,suggest_next\ is\_valid,\, error\_types,\, diagnosis,\, suggest\_next\. The validator returns a structured judgment including whether the step is valid, the error type, a brief diagnosis, and a next-step suggestion. The diagnosis explains why the current step is inconsistent with the gold trajectory, while the suggested next step only indicates the direction of repair. Policy-Aligned Trajectory Corrector When the validator detects a harmful deviation, the corrector rewrites the current step into an intervened response u~k u_k. The key requirement is that the corrected step should remain a plausible continuation of the agent’s own reasoning process. Concretely, the corrector takes as input the history h1:k−1h_1:k-1, tool schema T, the original draft u^k u_k, and the validator feedback, and produces a revised response that fixes the critical error while preserving the original response style, action format, and local reasoning structure as much as possible. Corrector prompt (schematic) Inputs. History h1:k−1h_1:k-1; tool schemas T; prior draft u^k u_k (style reference); validator feedback (diagnosis and suggested next step); reward-side evidence ℓ (privileged reference, never echoed). Body. (i)“silent calibrator” specification; (i) four generation principles—style alignment, progressive correction, leakage suppression, minimal intervention; (i) domain-specific requirements; (iv) explicit instruction that the intervened response must be derivable from visible history alone. Output. JSON reasoning,tool_calls,content\ reasoning,\, tool\_calls,\, content\. Our goal is not to replace the trajectory with an oracle solution, but to generate a minimally corrected version that stays close to the policy’s native behavior while removing the decisive error. Therefore, the corrector is instructed to follow the visible history and imitate the original reasoning pattern. Integrity-Constrained Intervention Verifier Finally, the verifier checks whether the intervened response is suitable. Given the original context and the corrected response u~k u_k, it verifies three properties: (i) the correction remains consistent with the gold requirement, (i) it does not leak privileged information, and (i) it is still a reasonable continuation of the original policy behavior. Verifier prompt (schematic) Inputs. User query; history h1:k−1h_1:k-1; tool schemas; prior draft u^k u_k; intervened response u~k u_k; evaluation criteria from ℓ . Body. (i) visibility specification (only the user query, tools, and history may justify u~k u_k); (i) four rejection criteria: leak of privileged keywords, fabricated tool result, wrong tool arguments contradicting history or violating explicitly specified parameter values, unreasonable intervention; (i) an enumerated allow-list of legitimate behaviors. Output. JSON ok,reasons,diagnosis\ ok,\, reasons,\, diagnosis\. The verifier closes the intervention loop by ensuring that accepted trajectories are aligned with the gold path, do not leak privileged information, and stay behaviorally close to the original policy. 4.2 Perplexity-Based Distribution Gating Although the intervention verifier removes trajectories with information leakage or implausible reasoning, the accepted trajectories may still deviate substantially from the rollout policy. When an intervened trajectory has low likelihood under the current policy, its probability ratios may fall outside the PPO clipping range, suppressing useful learning signals; meanwhile, learning from such low-probability behavior can produce poorly conditioned, high-variance gradient estimates and destabilize optimization. We therefore introduce perplexity-based distribution gating to filter intervention trajectories with excessive off-policy deviation before policy optimization. Specifically, after obtaining an intervened response, we remove all intervention-specific information and rescore it under the rollout policy conditioned solely on the original observable history. For u~k=(u~k1,…,u~kTk) u_k=( u_k^1,…, u_k^T_k) with TkT_k tokens at step k, its perplexity is computed as PPLroll(u~k)=exp(−1Tk∑t=1Tklogπθroll(u~kt∣h1:k−1,u~k<t)),PPL_roll( u_k)= (- 1T_k _t=1^T_k _θ^roll ( u_k^t h_1:k-1, u_k^<t ) ), (3) where TkT_k is the number of tokens in the intervened response. Higher perplexity indicates a larger deviation from the behavior supported by the policy model. We retain u~k u_k with a threshold δ: PPLroll(u~k)≤δ.PPL_roll( u_k)≤δ. (4) 4.3 Reasoning-Shift-Aware Token Reweighting The intervention replaces a single failed rollout in the group with an intervened trajectory τ~ τ, which is sampled from a behavior distribution πcalπ^cal different from the rollout distribution πθroll _θ^roll. Importantly, not all corrected tokens contribute equally to learning: in many cases, a small subset of tokens is critical for changing the reasoning direction and turning a failed trajectory into a successful one. We therefore introduce a reasoning-shift-aware token reweighting, which focuses the update on those intervention-induced tokens that carry the key reasoning correction. Consider an intervened response segment u~k=(u~k1,…,u~kTk) u_k=( u_k^1,…, u_k^T_k) with TkT_k tokens at step k. We define a token-level reweighting coefficient based on the likelihood ratio between the two conditions: ρt=exp(logπcal(u~kt∣h1:k−1,u~k<t)−logπθroll(u~kt∣h1:k−1,u~k<t)).ρ^t= \! ( π^cal( u_k^t h_1:k-1, u_k^<t)- _θ^roll( u_k^t h_1:k-1, u_k^<t) ). (5) where πcalπ^cal and πθroll _θ^roll denote the token likelihoods under the correction prompt pcalp_cal (which additionally includes the validator feedback) and the rollout prompt prollp_roll, respectively. A larger ratio means that the token is much more characteristic of the corrected reasoning path than of the original rollout path and should therefore receive greater emphasis during learning. To avoid unstable updates caused by extreme ratios, we apply a two-sided clip around 11 so that the ratio acts as a bounded per-token reweighting coefficient: ρ~calt=clip(ρt, 1−ϵcal, 1+ϵcal), ρ_cal^t=clip\! (ρ^t,\;1- _cal,\;1+ _cal ), (6) where ϵcal∈(0,1) _cal∈(0,1), treated as a stop-gradient constant that only reweights the advantage of the tokens it emphasizes. Note that the reweighting coefficient ρ~calt ρ_cal^t is only computed on tokens from intervened trajectories; for purely on-policy trajectories (i.e., when at least one rollout in the group already succeeds and no intervention is invoked), we set ρ~calt≡1 ρ_cal^t≡ 1, so that RTI reduces exactly to GRPO on those groups. The final policy-gradient loss applies the reasoning-shift reweighting to the token-level GRPO clipped surrogate. Concretely, expanding GRPOJ_GRPO from Eq. (2) yields: RTI(θ)=x,τ~i[1G∑i=1G1|τ~i|∑t=1|τ~i|ρ~caltmin(ρgrpo,itAi,clip(ρgrpo,it,−ϵclip,+ϵclip)Ai)]. _RTI(θ)=E_x,\ τ_i\\! [ 1G _i=1^G 1| τ_i| _t=1^| τ_i| ρ_cal^t \! ( _grpo,i^t\,A_i,\;clip( _grpo,i^t,1\!-\! _clip,1\!+\! _clip)\,A_i ) ]. (7) 5 Experiments 5.1 Experimental Setup Benchmarks. We evaluate RTI across five benchmarks. First, we construct Navigation, a multi-waypoint navigation benchmark derived from real-world map scenarios, where agents must plan routes and complete tasks by interacting with real APIs from the Amap Open Platform. Second, we adopt the publicly available multi-turn dialogue benchmark τ2τ^2-Bench (Barres et al., 2025) and evaluate on its Telecom, Retail, and Airline domains. The corresponding RL training instances are drawn from the AReaL dataset (Fu et al., 2026). Finally, we consider Claw-Eval (Ye et al., 2026), a human-verified benchmark comprising highly diverse tasks grounded in real-world environments. To evaluate whether RTI can improve performance beyond the policy’s native capabilities, we select 20 challenging T-series tasks on which the base policy rarely succeeds, forming Claw-Eval-Hard20, and synthesize task-specific training data for these tasks. Evaluation Metrics. We report Pass@k and Passk for k=1,…,4k=1,…,4. Pass@k measures the proportion of tasks for which at least one of the k independent rollouts succeeds, while Passk measures the proportion of tasks for which all k rollouts succeed. The p@1 is the mean Pass@11 over the four evaluations, while avg p@k and avg pk additionally average the corresponding metric over k=1,…,4k=1,…,4. For the publicly available benchmarks, we follow their original evaluation protocols to determine whether a trajectory is successful. For Navigation, a trajectory is considered successful if the agent invokes the direction tool with the correct parameters. Baselines. We compare RTI against: Reference models include the backbones (Qwen3-8B / Qwen3.5-9B), larger same-family variants (Qwen3-235B and Qwen3.5-397B-A17B), and contemporaneous proprietary models, paired by generation: GPT-4.1 and Claude-4-Sonnet for Qwen3-8B, and GPT-5.1 and Claude-4.6-Sonnet for Qwen3.5-9B. On-policy RL baselines include GRPO (Shao et al., 2024), GSPO (Zheng et al., 2025), and AEPO (Dong et al., 2026a). Teacher-guided methods include OPD (Lu & Thinking Machines Lab, 2025) and LUFFY (Yan et al., 2026). Further details on the benchmarks and baselines are provided in Appendices C.2 and C.3. 5.2 Main Results Tables 1 and 2 show that RTI consistently improves agent performance, with the largest gains on tasks where the backbone and existing RL methods struggle. Three findings stand out. First, RTI overcomes bottlenecks in domain-specific interaction. With Qwen3-8B, RTI improves Pass@1 over GRPO by 24.56 points on τ2τ^2-Telecom and 22.00 points on Navigation. Telecom requires the agent to guide users through actions it cannot perform itself; Navigation requires it to recognize noisy POI results and refine its search before requesting directions. Corrected rollouts expose the policy to successful strategies for both failure modes (Appendix C.2). Second, updating trajectories with a teacher does not necessarily overcome the teacher’s capability limit. OPD and LUFFY provide additional learning signals but achieve only 26.25% and 30.00% Pass@1, respectively, on Claw-Eval-Hard20 with Qwen3-8B. RTI reaches 50.00%, indicating that its gains cannot be explained solely by supplying trajectories beyond unaided on-policy exploration. Third, RTI extends these gains to diverse hard tasks and stronger backbones. On Claw-Eval-Hard20, RTI exceeds GRPO by 27.50 points with Qwen3-8B. Although Qwen3.5-9B already performs strongly on several other benchmarks, its baseline Pass@1 on this subset is 26.25%; RTI raises it to 85.00%, 23.75 points above the strongest competing method, LUFFY. These results suggest that RTI expands the set of tasks the policy can solve across different backbone capability levels. Table 1: Main results on five benchmarks with the Qwen3-8B backbone. Navigation τ2τ^2-Telecom τ2τ^2-Retail τ2τ^2-Airline Claw-Eval-Hard20 Method p@1 avg p@k avg pk p@1 avg p@k avg pk p@1 avg p@k avg pk p@1 avg p@k avg pk p@1 avg p@k avg pk Qwen3-8B 22.67 28.50 14.17 36.84 57.02 21.49 45.61 71.93 35.53 31.00 50.50 16.00 18.75 24.38 15.94 Qwen3-235B-A22B 28.00 35.50 16.67 44.74 64.04 28.07 64.04 79.61 47.15 50.50 64.50 32.00 20.00 20.00 12.50 GPT-4.1 23.33 31.33 18.67 39.47 58.99 23.68 71.93 83.55 52.19 53.00 66.50 34.00 30.00 30.00 18.75 Claude-4-Sonnet 26.00 32.83 20.67 49.12 67.76 32.02 74.56 85.09 53.95 54.00 68.50 35.50 40.00 46.25 30.00 GRPO 35.33 40.00 29.00 36.84 55.48 21.71 67.54 80.92 50.22 46.50 57.50 26.50 22.50 24.17 20.83 GSPO 28.00 34.50 18.00 44.96 64.48 30.66 63.16 74.78 46.27 39.00 55.50 22.50 26.25 31.67 19.90 AEPO 34.00 37.67 26.17 45.68 62.41 29.23 63.16 81.14 49.78 51.00 65.50 33.50 20.00 27.50 16.25 OPD 37.33 41.50 32.00 55.26 68.64 36.62 68.42 82.24 46.71 47.50 56.50 30.50 26.25 31.46 21.98 LUFFY 36.00 51.50 26.67 49.12 72.59 27.19 71.93 84.87 49.34 46.00 59.50 26.50 30.00 32.81 27.19 RTI 57.33 62.33 46.67 61.40 81.14 44.30 73.68 85.09 55.92 56.00 70.00 35.50 50.00 62.29 37.71 Table 2: Main results on five benchmarks with the Qwen3.5-9B backbone. Navigation τ2τ^2-Telecom τ2τ^2-Retail τ2τ^2-Airline Claw-Eval-Hard20 Method p@1 avg p@k avg pk p@1 avg p@k avg pk p@1 avg p@k avg pk p@1 avg p@k avg pk p@1 avg p@k avg pk Qwen3.5-9B 39.33 53.00 26.17 92.11 96.49 89.69 82.46 93.64 71.05 60.50 73.00 47.00 26.25 31.67 19.90 Qwen3.5-397B-A17B 44.00 53.67 38.00 95.61 98.25 93.64 84.21 94.74 73.90 63.00 77.00 51.00 30.00 40.00 26.25 GPT-5.1 42.00 54.00 30.17 95.61 98.03 93.64 77.19 92.11 62.28 66.50 79.00 53.00 40.00 47.50 40.00 Claude-4.6-Sonnet 58.00 64.17 53.83 97.37 98.90 96.05 91.23 96.93 85.53 80.00 88.50 71.50 51.25 59.27 44.17 GRPO 71.33 74.83 66.33 95.61 97.59 93.64 84.21 94.74 73.90 75.00 82.50 63.00 52.50 57.71 45.42 GSPO 68.67 72.33 63.17 95.61 98.03 94.30 85.09 94.96 75.44 79.00 85.50 68.00 51.25 58.75 44.69 AEPO 70.67 77.17 64.83 96.49 98.47 94.52 85.96 95.18 76.75 78.50 86.50 70.00 51.25 56.66 44.90 OPD 70.00 75.00 58.67 96.49 98.68 95.18 86.84 95.61 78.29 79.50 84.50 65.00 51.25 55.94 47.50 LUFFY 71.33 76.17 65.17 96.49 98.68 95.18 85.96 95.18 76.75 78.00 86.50 67.00 61.25 66.67 54.90 RTI 74.67 78.50 68.67 96.49 98.68 95.18 88.60 96.05 81.14 82.00 87.50 70.50 85.00 94.17 73.96 5.3 Ablation Studies Table 3: Ablation study on Navigation and τ2τ^2-Telecom with the Qwen3-8B backbone. Navigation τ2τ^2-Telecom Method pass@1 pass@2 pass@3 pass@4 pass1 pass2 pass3 pass4 pass@1 pass@2 pass@3 pass@4 pass1 pass2 pass3 pass4 GRPO 35.33 40.00 41.33 43.33 35.33 29.33 26.67 24.67 36.84 52.63 63.16 69.30 36.84 21.93 17.54 10.53 RTI w/o Post-Check 26.00 33.33 36.00 38.00 26.00 14.00 9.33 7.33 38.60 53.51 59.65 67.54 38.60 24.56 16.67 14.91 RTI w/o PPL Gating 50.00 60.00 64.00 65.33 50.00 37.33 29.33 24.00 54.39 68.42 75.44 79.82 54.39 27.19 21.05 18.42 RTI w/o Reweight 55.33 58.67 60.67 62.00 55.33 45.33 42.00 33.33 60.53 79.82 85.96 88.60 60.53 38.60 30.70 20.18 RTI 57.33 61.33 64.67 66.00 57.33 48.00 42.00 39.33 61.40 83.33 87.72 92.11 61.40 46.49 37.72 31.58 To isolate each component’s contribution, we conduct ablation studies on Navigation and τ2τ^2-Telecom with Qwen3-8B, evaluating three variants: RTI w/o Post-Check, which removes the Integrity-Constrained Intervention Verifier; RTI w/o PPL Gating, which removes the PPL-Based Distribution Gating; and RTI w/o Reweight, which removes the Reasoning-Shift-Aware Token Reweighting. Effect of Post-Check. Removing the Post-Check leads to a large degradation on both benchmarks. On τ2τ^2-Telecom, the variant loses almost all of RTI’s gains and falls back to the GRPO level; on Navigation, it even drops below GRPO (26.00 vs. 35.33 in Pass@1). Without integrity verification, the admitted interventions are frequently leak-contaminated or otherwise unlearnable, and the corrupted supervision can actively harm the policy rather than merely fail to help. A representative failure case is given in Appendix D.2. Effect of PPL Gating. Removing the PPL Gate yields the second-largest degradation on τ2τ^2-Telecom: pass@1 decreases from 61.40 to 54.39, and the all-of-k metrics drop even more sharply (pass2: 46.49→ 27.19, pass4: 31.58→ 18.42). This confirms that bounding the distributional shift of intervened responses is essential: interventions that deviate excessively from the rollout policy provide unstable learning signals, so filtering them out preserves consistent optimization. Effect of Reweight. Removing the token reweighting results in the mildest overall change, yet the degradation is systematic: RTI is at least as good on all sixteen metrics, with the largest gaps on the all-of-k metrics. This suggests that reweighting mainly sharpens credit assignment toward the decisive intervention tokens, improving the stability with which the corrected reasoning is reproduced across repeated attempts, whereas the exploration breadth is already supplied by the intervention. 5.4 Hyperparameter Study We further study the gating threshold δ in the PPL-based distribution gating (Section 4.2, Eq. (4)), which controls the maximum deviation from the rollout policy that an intervened response may exhibit before being discarded. We sweep δ∈0,3,5,20,+∞δ∈\0,3,5,20,+∞\ on Navigation with the Qwen3-8B backbone. At one extreme (δ=0δ=0), the gate rejects all interventions, so training falls back to the no-intervention limit; at the other (δ=+∞δ=+∞), no filtering is applied at all. As shown in Figure 3, the two extremes behave as expected: with δ=0δ=0, training degenerates into the GRPO run and the success rate stays nearly flat, whereas δ=+∞δ=+∞ improves steadily but converges to a clearly lower level. Performance improves as δ grows from 33 to 55 and degrades at δ=20δ=20: a threshold that is too small discards too many corrective trajectories to sustain late-stage gains, while a threshold that is too large admits severely shifted interventions, whose low-probability, high-variance updates destabilize optimization—the δ=20δ=20 curve indeed exhibits the highest volatility among all runs. The best trade-off is achieved at δ=5δ=5, which preserves informative corrections while filtering out excessively deviated ones: it tracks δ=3δ=3 closely in early training, pulls ahead in later stages, and converges to the highest final success rate. We therefore adopt δ=5δ=5 in all experiments. 5.5 Further Analysis Figure 3: Effect of the gating threshold δ on Pass@4 for Navigation with Qwen3-8B. Figure 4: Training dynamics on Navigation under GRPO vs. RTI with the Qwen3-8B backbone. Training Dynamics Comparison. Figure 4 compares evaluation Pass@4 (solid lines) with training rollout Pass@8 (dashed lines) on Navigation. All curves are Gaussian-smoothed for visualization. For RTI, the two dashed curves measure rollout success before and after intervention. The large gap at early stage between them shows that intervention converts many failed rollout groups into successful trajectories, providing training signals where GRPO receives few. Crucially, RTI’s pre-intervention Pass@8 also rises over training and exceeds GRPO’s rollout success, while its evaluation Pass@4 improves in tandem. Thus, the corrected trajectories support effective policy updates: the model increasingly succeeds without intervention, rather than relying on intervention only at rollout time. (a) Dynamics of Intervention PPL under RTI. (b) Per-iteration distribution of interaction turns. Figure 5: Further analysis of intervention PPL dynamics and resulting reasoning-pattern distributions. Dynamics of Intervened-Step Perplexity and Reasoning Patterns. To further verify whether the intervened behaviors are truly internalized by the policy, we examine the average PPL under the policy model per successful intervened step, which measures the distributional gap between the intervened trajectory and the policy’s native generation distribution. As shown in Figure 5, the curve drops steadily during training: those that are still triggered yield intervened responses that are increasingly close to the policy’s own output distribution. It provides direct evidence that the policy is not merely exploiting the intervention at training time but is actually absorbing the intervened reasoning pattern into its native behavior. Additionally, we analyze Navigation to examine how training under RTI versus GRPO changes the model’s resulting reasoning patterns. Figure 5 contrasts the per-iteration distribution of interaction turns under GRPO and RTI on Navigation. Compared with GRPO, RTI maintains a broader and more multi-modal turn distribution throughout training. In particular, trajectories exceeding 3 turns continue to retain non-trivial mass even at later stages, indicating that the policy preserves the capacity for self-reflective re-querying rather than collapsing into a single short interaction template. A case is provided in Appendix D.4. Figure 6: Training curves on Claw-Eval-Hard20 with the Qwen3.5-9B backbone. Generalization Across Hard Agent Tasks. All preceding RTI experiments use self-intervention. To examine whether a stronger intervening model improves learning, we additionally use Qwen3.5-397B-A17B to correct rollouts from the Qwen3.5-9B policy. Figure 6 compares this variant with standard RTI and GRPO on Claw-Eval-Hard20. GRPO improves briefly before its Pass@k and Passk scores collapse. Both RTI variants instead sustain gains across the two metric families: higher Pass@k reflects success on more tasks within k attempts, while higher Passk reflects more consistent success across attempts. Standard RTI achieves the strongest results, showing that corrections from the much larger model deviate further from the policy’s distribution and are therefore harder to learn from, despite the teacher’s greater capability. These findings show that self-intervention suffices to improve performance across diverse hard tasks without relying on a stronger teacher. Additionally, we supplement the experimental results on the full set of unimodal (text-only) tasks in Appendix D.6. 6 Conclusion We introduced RTI to enable agent policies to learn beyond their unaided exploration capacity. RTI uses reward-aware self-intervention to correct failed rollouts, a perplexity-based gate to filter excessively shifted trajectories, and reasoning-shift-aware token reweighting to focus policy updates on consequential decisions. Across five benchmarks and two backbones, RTI substantially improves task success, with particularly large gains on Navigation, τ2τ^2-Telecom, and Claw-Eval-Hard20. Training analyses show that intervention supplies successful trajectories early in training and that the policy increasingly succeeds without intervention as learning proceeds. An additional comparison using a much larger intervening model confirms that RTI does not require a stronger teacher to achieve these gains, supporting its applicability across diverse agent tasks. Acknowledgments This work was supported by Alibaba Research Intern Program. We would like to thank the Qwen Team at Alibaba Token Hub (ATH), Alibaba Group, for providing the computational resources and foundation models (Qwen) used in this research. AI use statement We used generative AI tools as an assistive aid in the preparation of this manuscript, specifically for proofreading, grammar checking, and language refinement to improve clarity and readability. All research ideas, experimental designs, theoretical claims, and final content were conceived, verified, and approved by the authors. Ethics statement We will abide by the laws, rules, and regulations of our community, school, work, and country. We will conduct ourselves with integrity, fidelity, and honesty. We will openly take responsibility for our actions and only make agreements that we intend to keep. All data used in this study are intended for research purposes. No personally identifiable information (PII) was collected. Reproducibility statement We have included comprehensive details to support reproducibility of this work. The complete RTI algorithm, implementation details, hyperparameters, benchmark descriptions, baseline configurations, and theoretical proofs are provided in the appendix. Representative prompts and case studies are also included in the appendix. References Barres et al. (2025) Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ2τ^2-bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506.07982. Dong et al. (2026a) Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jinghan Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Toward generalized web agent training: A deep dive into entropy-balanced reinforcement learning. In Proceedings of the ACM Web Conference 2026, p. 2126–2137, 2026a. URL https://doi.org/10.1145/3774904.3792301. Dong et al. (2026b) Guanting Dong, Junting Lu, Junjie Huang, Wanjun Zhong, Longxiang Liu, Shijue Huang, Zhenyu Li, Yang Zhao, Xiaoshuai Song, Xiaoxi Li, Jiajie Jin, Yutao Zhu, Hanbin Wang, Fangyu Lei, Qinyu Luo, Mingyang Chen, Zehui Chen, Jiazhan Feng, Ji-Rong Wen, and Zhicheng Dou. Agent-world: Scaling real-world environment synthesis for evolving general agent intelligence, 2026b. URL https://arxiv.org/abs/2604.18292. Feng et al. (2025) Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URL https://arxiv.org/abs/2504.11536. Froger et al. (2025) Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Grégoire Mialon, and Thomas Scialom. Are: Scaling up agent environments and evaluations, 2025. URL https://arxiv.org/abs/2509.17158. Fu et al. (2026) Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, WANG JIASHU, Tongkai Yang, Binhang Yuan, and Yi Wu. AREAL: A large-scale asynchronous reinforcement learning system for language reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=X9diEuva9R. Gao et al. (2026) Jiaxuan Gao, Jiaao Chen, Chuyi He, Shusheng Xu, Di Jin, and Yi Wu. From self-evolving synthetic data to verifiable-reward rl: Post-training multi-turn interactive tool-using agents, 2026. URL https://arxiv.org/abs/2601.22607. Gu et al. (2024) Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5h0qf7IBZZ. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Jin et al. (2025) Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=Rwhi91ideu. Li et al. (2026) Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong, Jiajie Jin, Yinuo Wang, Hao Wang, Yutao Zhu, Ji-Rong Wen, Yuan Lu, and Zhicheng Dou. Deepagent: A general reasoning agent with scalable toolsets. In Proceedings of the ACM Web Conference 2026, p. 2219–2230, New York, NY, USA, 2026. Association for Computing Machinery. URL https://doi.org/10.1145/3774904.3792460. Li et al. (2025) Yuetai Li, Huseyin A Inan, Xiang Yue, Wei-Ning Chen, Lukas Wutschitz, Janardhan Kulkarni, Radha Poovendran, Robert Sim, and Saravan Rajmohan. Simulating environments with reasoning models for agent training, 2025. URL https://arxiv.org/abs/2511.01824. Liu et al. (2025) Weiwen Liu, Xu Huang, Xingshan Zeng, xinlong hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Wang Xinzhi, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. ToolACE: Winning the points of LLM function calling. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=8EB8k6DdCU. Lu & Thinking Machines Lab (2025) Kevin Lu and Thinking Machines Lab. On-policy distillation. 2025. URL https://thinkingmachines.ai/blog/on-policy-distillation. Prabhakar et al. (2026) Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Manoj Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Quoc Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, and Caiming Xiong. APIGen-MT: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. URL https://openreview.net/forum?id=qk6ORqQ4Cu. Qian et al. (2026) Cheng Qian, Emre Can Acikgoz, Qi He, Hongru WANG, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. ToolRL: Reward is all tool learning needs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=eOLdGbXT6t. Qin et al. (2024) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=dHng2O0Jjr. Reddi et al. (2016) Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczós, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, p. 314–323. JMLR.org, 2016. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. Sun et al. (2026) Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching, 2026. URL https://arxiv.org/abs/2505.04588. Team (2026) Kimi Team. Kimi k2: Open agentic intelligence, 2026. URL https://arxiv.org/abs/2507.20534. Wang et al. (2024) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), March 2024. ISSN 2095-2236. doi: 10.1007/s11704-024-40231-1. URL http://dx.doi.org/10.1007/s11704-024-40231-1. Wei et al. (2026) Tianxin Wei, Ting-Wei Li, Zhining Liu, Xuying Ning, Ze Yang, Jiaru Zou, Zhichen Zeng, Ruizhong Qiu, Xiao Lin, Dongqi Fu, Zihao Li, Mengting Ai, Duo Zhou, Wenxuan Bao, Yunzhe Li, Gaotang Li, Cheng Qian, Yu Wang, Xiangru Tang, Yin Xiao, Liri Fang, Hui Liu, Xianfeng Tang, Yuji Zhang, Chi Wang, Jiaxuan You, Heng Ji, Hanghang Tong, and Jingrui He. Agentic reasoning for large language models, 2026. URL https://arxiv.org/abs/2601.12538. Yan et al. (2026) Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=vO8LLoNWWk. Yang et al. (2026) Jinluan Yang, Yuxin Liu, Zhengyu Chen, Chengcheng Han, Yueqing Sun, Qi Gu, Hui Su, Xunliang Cai, Fei Wu, and Kun Kuang. Topocurate:modeling interaction topology for tool-use agent training, 2026. URL https://arxiv.org/abs/2603.01714. Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. Ye et al. (2026) Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, Qi Liu, Zhifang Sui, and Tong Yang. Claw-eval: Towards trustworthy evaluation of autonomous agents, 2026. URL https://arxiv.org/abs/2604.06132. Yue et al. (2026) Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=4OsgYD7em5. Zhao et al. (2025) Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining, 2025. URL https://arxiv.org/abs/2504.07912. Zheng et al. (2025) Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization, 2025. URL https://arxiv.org/abs/2507.18071. Appendix A More Details of RTI A.1 RTI MAS Prompt Templates In this section, we present the full prompt templates for the three MAS agents used in RTI. A.1.1 τ2τ^2-Bench Prompts (Telecom / Retail / Airline) Reward-Oriented Trajectory Validator. The Validator assesses whether the current response is a valid next step that keeps the agent on track. It checks against a set of hard rules and marks the response invalid only when a clear violation is detected. Reward-Oriented Trajectory Validator (τ2τ^2-Bench) You are a trajectory supervisor for a tool-calling agent task. Your job is to judge whether the CURRENT response is a valid next step that keeps the agent on track toward completing the task. Task is complete only when completion_note. [Observable History] history_json [Current Response](Formatted as json) response_text [Required actions to complete the task] Agent-side actions (the agent must call these tools itself): agent_actions user_actions_block (telecom only: lists user-side device actions the agent must guide) How to tell if an action is done: - agent-side action: a matching assistant tool call + successful result in history. - user-side action [telecom]: a matching user tool call output in history. RULES --- mark invalid if ANY rule is violated: 1) FABRICATED_ARGUMENTS: A tool argument value is not present in history and not derivable from the task description. (Values from earlier tool outputs ARE valid.) 2) POINTLESS_REPEAT: Calling the exact same tool with the exact same arguments that already succeeded. (Same tool, different arguments = allowed.) 3) FALSE_CLAIM: The response claims information was retrieved without a corresponding tool call. 4) [domain_specific]: [Telecom] USER_SIDE_DEVICE_ACTION: The agent emits a tool call for a user-device action. The agent must NEVER call these --- it must instruct the user to perform them on their device. USER_SIDE_TOOLS = [check_status_bar, check_network_status, ..., reboot_device] AGENT_SIDE_TOOLS = [get_customer_by_phone, ..., refuel_data] [Retail] MISSING_CONFIRMATION: Before any tool call that updates the database (including modify_user_address and any cancel/modify/return/exchange action), the agent must first list the exact action details and obtain an explicit user confirmation ‘yes’. [Airline] MISSING_CONFIRMATION: Before taking any actions that update the booking database (booking, modifying flights, editing baggage, changing cabin class, or updating passenger information), the agent must first list the action details and obtain an explicit user confirmation ‘yes’. 5) PREMATURE_TERMINATION: The response ends the conversation (transfer to human, closing message) while one or more required actions are not yet done. 6) CONTRADICTORY_ACTION: The response calls a tool or instructs the user in a way that directly contradicts a required action. 7) GOAL_DRIFT: The response ignores an obviously pending required action and pursues something unrelated. 8) ARGUMENT_MISMATCH: For actions that update the database, if [Required actions] specifies exact argument values, the agent’s tool call must match these values exactly. DO NOT mark invalid for: - Suboptimal but recoverable action order - Diagnostic / information-lookup steps before acting - Re-attempting a failed action with corrected arguments - A valid path that differs from the reference [Evaluation_Criteria --- for context only] evaluation_criteria Output JSON only: "diagnosis": "...", "is_valid": true/false, "error_types": [...], "suggest_next": "..." Policy-Aligned Trajectory Corrector. The Corrector generates an optimized next action that corrects errors while concealing privileged information. It receives the Validator’s diagnosis together with the visible-only conversation history and produces an intervened response. Policy-Aligned Trajectory Corrector (τ2τ^2-Bench) You are a silent calibrator for a tool-calling agent task. Your role is to minimally correct the agent’s next action to bring it closer to the successful path, WITHOUT exposing any privileged knowledge. [Conversation History] history [Available Tools] tools [Reference / evaluation criteria (for calibration only; MUST NOT appear in output)] evaluation_criteria CORE INSTRUCTIONS: Generate the optimized next action based on [Conversation History], NOT by continuing from [Prior Draft]. The Prior Draft is provided only as a style reference. GENERATION STRATEGY: 1. Style alignment: match the reasoning style, verbosity, and expression habits of the Prior Draft. 2. Progressive correction: guide toward a more reasonable path; corrections should feel like natural evolution of thought. 3. Concealment: NEVER mention ‘reference’, ‘groundtruth’, ‘required actions’, ‘evaluation criteria’, ‘prior draft’, ‘calibrated response’, ‘previous version’, or ‘the correct answer’. All reasoning must appear naturally derived from conversation history. 4. Minimal change: only correct clearly wrong steps; preserve correct parts of the prior draft. CORRECTION GUIDELINES: - If a tool argument is wrong, use the correct value from history (e.g., correct payment_id from user profile) - If the agent skipped a necessary step, add a message to gather required info or confirm with user - If the agent made an unfounded claim, revise to acknowledge uncertainty or check first - If multiple valid approaches exist, prefer the one closer to the original intent - DO NOT introduce information that cannot be derived from visible history - If no valid agent-side tool call is appropriate, set ‘tool_calls’ to an empty list. - Before any tool call that updates the database (including modify_user_address and any cancel/modify/return/exchange action), the agent must first list the exact action details and obtain an explicit user confirmation ‘yes’. [domain_specific] [Telecom] If an action must be done by the user on their device, do not emit a tool call for it; guide the user to perform the action on their device instead. USER_SIDE_DEVICE_ACTIONS = [check_status_bar, ..., reboot_device] AGENT_SIDE_TOOLS = [get_customer_by_phone, ..., refuel_data] Note: ‘tool_calls’ must be an empty list if no agent-side tool is to be called. [Prior Draft (style reference only)] original_response [Supervisor Feedback (problem indication)] feedback_json Output JSON: "reasoning": "...", "tool_calls": [...], "content": "..." Field rules: ‘reasoning’ is always required. ‘tool_calls’ and ‘content’ are mutually exclusive: use tool_calls for agent-side actions, content for direct user replies. Integrity-Constrained Intervention Verifier. The Verifier validates that the corrected response does not leak privileged information or introduce hallucinated content. It serves as the final quality gate before an intervened response enters the training batch. Integrity-Constrained Intervention Verifier (τ2τ^2-Bench) You are a post-calibration safety & faithfulness inspector. Decide whether the CALIBRATED reply should be accepted. VISIBLE INFO = user question + tool list + conversation history (including tool outputs). Nothing else is allowed. [User Question] user_prompt [Evaluation Criteria] evaluation_criteria [Available Tools] tools [Agent History (visible)] history [Prior Draft] original [Calibrated Reply] corrected Reject (ok=false) ONLY if ANY holds: 1) leak: explicitly mentions ‘reference’, ‘groundtruth’, ‘evaluation criteria’, ‘required actions’, ‘answer key’, ‘labels’, ‘seeing the solution’, ‘the prior draft’, ‘previous version’, ‘calibrated version’, or any phrasing that implies the model is comparing its output to a hidden draft or ground truth. 2) fabricated_tool_result: claims a specific tool result/value that does NOT appear in history. Example: claiming ‘flight price is $500’ when no tool returned that exact value. 3) wrong_tool_arguments: uses tool arguments that contradict visible history or deviate from explicit constraints defined in [Evaluation Criteria]. Mandatory Alignment: if the criteria explicitly specifies a parameter value, the agent’s argument must match exactly. Contextual Discretion: if not specified, the agent may derive values logically. 4) unreasonable_calibration: the calibrated reply is not a reasonable next step given the visible history, fails to improve the prior draft, or introduces unnecessary or less appropriate actions. ALLOWED (not an issue) --- DO NOT REJECT for: - Adding reasonable next steps implied by task context (e.g., asking for user confirmation before database updates) - Adding policy-compliant behaviors (e.g., ‘I need to verify with the user first’) - Changing the action sequence to be more cautious or thorough - Adding intermediate steps that logically follow from the conversation The calibration goal is to IMPROVE the agent’s behavior, not just paraphrase it. Adding missing confirmation steps or policy-compliant actions is the INTENDED behavior of calibration. Output JSON: "ok": true/false, "reasons": [...], "diagnosis": "..." A.1.2 Navigation Domain Prompts The Navigation domain follows the same three-agent Validate–Correct–Verify pipeline as τ2τ^2-Bench, sharing the identical architectural logic: the Validator detects deviations from a valid path using the reward-side evidence as a silent reference; the Corrector applies the same four generation principles (style alignment, progressive correction, leakage suppression, minimal intervention); and the Verifier enforces the same integrity constraints (no leakage, no hallucination). Reward-Oriented Trajectory Validator (Navigation). The Navigation Validator applies the same goal-oriented, process-agnostic philosophy as its τ2τ^2-Bench counterpart: it does not penalize suboptimal-but-recoverable action orders or stylistic differences, and only flags violations of the hard rules already stated in the agent’s system prompt. Reward-Oriented Trajectory Validator (Navigation) You are a trajectory supervisor for an ‘Agent tool-calling task’. Judge whether the current response constitutes a reasonable next step. [Conversation History (observable evidence)] history_json [Current Response] response_text [Hard Rules]: - No fabricating coordinates/key parameters (must come from historical tool returns) - Do NOT call direction before all POI coordinates are collected - No redundant poi_search with identical parameters in the same turn - Parallel tool calls are allowed [Reference (for internal comparison only; do NOT judge invalid based solely on reference)] order: order ref: ref groundtruth: gt Mark invalid ONLY if: A) Hard rule violation (fabrication / premature direction / redundant search) B) Stubbornly repeating on clearly failed tool returns C) Irreversible wrong critical call with clearly untenable arguments Output JSON only: "is_valid": true/false, "error_types": [], "diagnosis": "...(cite observable evidence/hard rules)...", "suggest_next": "..." Policy-Aligned Trajectory Corrector (Navigation). The Navigation Corrector shares the same four generation principles as τ2τ^2-Bench (style alignment, progressive correction, leakage suppression, minimal intervention). Policy-Aligned Trajectory Corrector (Navigation) You are the ‘silent calibrator’ in a navigation task --- your role is to minimally optimize an agent’s action trajectory toward the successful path, WITHOUT exposing any additional knowledge. You have a global view (knowing how the task should ultimately be completed), but must translate this insight into seemingly natural, progressive corrections. [Conversation History] history [Available Tools] tools [Reference/groundtruth (for calibration only; MUST NOT appear in output)] ref: ref order: order groundtruth: gt CORE INSTRUCTIONS: Generate the optimized next action based on [Conversation History], NOT based on [Prior Draft]. The prior draft is provided only as a style reference. GENERATION STRATEGY: 1. Style alignment: analyze the reasoning style, detail level, and expression habits of the Prior Draft; new reasoning should match in style. 2. Progressive correction: leverage your global view to embed a more reasonable reasoning path; corrections should feel like natural evolution, not a 180-degree turn. 3. Concealment: NEVER mention or imply you know the correct answer, final goal, reference behavior, or full process. All reasoning must appear naturally derived from history. 4. Minimal change: only correct clearly wrong key points; preserve correct parts of the prior draft. [Prior Draft (style reference)] original_response [Supervisor Feedback (problem indication)] feedback_json KEY REMINDERS: - You are generating the next action based on [Conversation History], not continuing from [Prior Draft] - Your output should look like the agent’s own naturally improved reasoning Integrity-Constrained Intervention Verifier (Navigation). The Navigation Verifier enforces the same two core rejection criteria as the τ2τ^2-Bench Verifier—leakage of privileged information and hallucination of underivable content. Integrity-Constrained Intervention Verifier (Navigation) You are a ‘safety & explainability post-checker’. A calibrated reply is given; decide whether it should be accepted. [User Question] user_prompt [Available Tools] tools [Agent Conversation History] history [Prior Draft] original [Calibrated Reply] corrected Rejection criteria (any one triggers ok=false): 1) leak: mentions reference/groundtruth/standard answer/labels, or clearly uses invisible information. 2) not_derivable (hallucination): the calibrated reply adds/modifies key facts or tool call parameters that come neither from ‘user question + visible history’ nor qualify as ‘allowed rewrites’ below. Allowed rewrites (NOT hallucination): - Equivalent rewriting of already-appeared information (synonyms, format normalization, unit conversion) - Incorporating already-specified context constraints into parameters (region, time window, mode) - Conservative narrowing for disambiguation (combining category words with keywords, removing uncertain components) - Decomposing complex requests into verifiable sub-steps (but must not claim tool results that don’t exist in history) Output JSON only: "ok": true/false, "reasons": ["leak", "not_derivable"], "diagnosis": "must cite observable evidence (specific fragments from history / missing points)" A.2 The Full RTI Algorithm Algorithm 1 integrates the intervention pipeline and the on-policy GRPO loop into a single training procedure. At every iteration we sample a batch of queries from D and, for each query, first run a fully on-policy GRPO group; if any rollout in this group succeeds, the group is used as-is and only the on-policy portion of RTIJ_RTI is exercised (with ρ~calt≡1 ρ_cal^t≡ 1). Only when the entire group fails do we re-run an intervened rollout with the MAS intervention pipeline enabled; every accepted correction must additionally pass the perplexity gate of Eq. (4), and only if the intervened rollout succeeds is a single failed member of the group replaced by the intervened trajectory, for which we additionally cache the rollout-prompt and correction-prompt logprobs needed by Eq. (7). A single backward pass over RTIJ_RTI then updates θ with a gradient that is on-policy on easy instances, reweighted on intervened tokens of hard instances, and unbiased on every other token. This design preserves GRPO’s behavior whenever the policy can solve a task on its own and adds capability-expanding signals exactly where vanilla on-policy RL stalls. Algorithm 1 Rollout-Time Intervention (RTI) Input: Policy πθ _θ (which also instantiates all MAS roles), reference πref _ref, task distribution D, group size G, reweighting clip ϵcal _cal. 1 for each training iteration do 2 Sample batch xn∼\x_n\ 3 for each query xnx_n do // Stage 1: on-policy rollouts 4 Sample τii=1G∼πθ(⋅∣xn)\ _i\_i=1^G _θ(· x_n) with intervention disabled 5 if maxir(xn,τi)>0 _ir(x_n, _i)>0 then 6 τ~i←τi\ τ_i\←\ _i\, set ρ~calt≡1 ρ_cal^t≡ 1 on all tokens // purely on-policy update 7 end if 8 else // Stage 2: reward-aware MAS intervention 9 τ~i←τi\ τ_i\←\ _i\ 10 Run an intervened rollout: at each step k generate u^k∼πθ u_k _θ; 11 pass (h1:k−1,u^k,ℓ)(h_1:k-1, u_k, ) through Validator → Corrector → Verifier 12 if the correction is accepted by the Verifier then 13 Rescore u~k u_k under the rollout prompt and compute PPLroll(u~k)PPL_roll( u_k) by Eq. (3) 14 if PPLroll(u~k)≤δPPL_roll( u_k)≤δ then 15 Advance with u~k u_k, cache logπθroll _θ^roll and logπcal π^cal 16 end if 17 else 18 Discard u~k u_k, advance with u^k u_k 19 end if 20 end if 21 else 22 Advance with u^k u_k 23 end if 24 if the intervened rollout achieves reward 11 then 25 Replace a single failed member of τ~i\ τ_i\ with the intervened trajectory 26 Compute clipped weights ρ~calt ρ_cal^t via Eq. (6) on intervened tokens; set ρ~calt≡1 ρ_cal^t≡ 1 on the remaining on-policy tokens 27 end if 28 else 29 Keep τ~i\ τ_i\ unchanged, so that no correction enters the update 30 end if 31 end if 32 Compute group advantages Ai\A_i\ from rewards r(xn,τ~i)\r(x_n, τ_i)\ as in GRPO 33 end for 34 Update θ by ascending the RTI objective (Eq. 7) 35 end for Appendix B Theoretical Analysis of RTI B.1 Convergence Analysis of RTI RTI differs from standard on-policy GRPO in two aspects: it incorporates MAS-intervened trajectories, which are off-policy relative to the current policy, and it further applies the reasoning-shift-aware token reweighting in Eq. (7). These two components raise a natural concern: although they may improve exploration and credit assignment, they also introduce additional distribution shift into the optimization process. Based on the theoretical analysis of stochastic gradient descent in nonconvex optimization (Reddi et al., 2016; Yan et al., 2026), we give a convergence analysis in Theorem 1 to show that, under the alignment and decoupled boundedness conditions stated below, RTI admits an O(1/K)O(1/ K) stationarity guarantee, where K is the total number of iterations. For notational simplicity, we write (θ):=RTI(θ)J(θ):=J_RTI(θ) throughout this section. Theorem 1 (Conditional Convergence under Alignment and Decoupled Boundedness). RTI constructs the policy gradient using a ratio mixture λ∈(0,1)λ∈(0,1) of off-policy intervened rollouts and on-policy rollouts, where λ denotes the expected mixing ratio over training (since intervention is triggered only on fully failed groups, λ is a data-dependent random quantity and we take its expectation as a constant for the analysis). Accordingly, we decompose the per-step update as Δθk=λΔθcalk+(1−λ)Δθgrpok θ^k=λ\, _cal^k+(1-λ)\, _grpo^k. Suppose that the objective (θ)J(θ) is L-smooth and that ⋆:=supθ(θ)<∞J := _θJ(θ)<∞. Assume further that the expected update direction is aligned with the true gradient with a lower bound governed by the on-policy clip margin 1−ϵclip1- _clip (this alignment lower bound arises from the on-policy term Δθgrpok _grpo^k under the classic clipped surrogate; the intervened term Δθcalk _cal^k is not required to contribute alignment and is instead absorbed into the second-moment bound Dcal2D_cal^2 in Definition 5), i.e., [⟨∇(θk),Δθk⟩]≥αk(1−ϵclip)‖∇(θk)‖2.E\! [ (θ^k), θ^k ]≥ _k(1- _clip)\| (θ^k)\|^2. (8) For K iterations, let the learning rate be set to αk=c/K _k=c/ K, where c=2[⋆−(θ0)+L2λDcal2]L(1+ϵclip)2σ2c= 2 [J -J(θ^0)+ L2λ D_cal^2 ]L(1+ _clip)^2σ^2 (9) Then the iterates satisfy the following (1/K)O(1/ K) convergence bound: min0≤k≤K−1[‖∇(θk)‖2]≤1(1−ϵclip)2[⋆−(θ0)+L2λDcal2]L(1+ϵclip)2σ2K, _0≤ k≤ K-1E [\| (θ^k)\|^2 ]≤ 1(1- _clip) 2 [J -J(θ^0)+ L2λ D_cal^2 ]L(1+ _clip)^2σ^2K, (10) where θ0θ^0 denotes the initial policy parameter, σ2σ^2 controls the base gradient scale and Dcal2=4ϵcal2η2σ2D_cal^2=4\, _cal^2\,η^2\,σ^2 with η>0η>0 as a parameter-space perturbation constant abstracting the bounded contribution of the intervened rollouts (formalized in Definition 5). B.2 Proof of Theorem 1 We first state the standard definitions and assumptions for non-convex stochastic optimization. Definition 1 (Stochastic Oracle). For a smooth functional J, we assume the algorithm has access to a stochastic oracle that, given an index i∈[n]i∈[n] and a parameter θ∈ℝdθ ^d, returns the pair (i(θ),∇i(θ))(J_i(θ), _i(θ)). Definition 2 (Lipschitz Smoothness). We say :ℝd→ℝJ:R^d is L-smooth if there exists a constant L>0L>0 such that ‖∇(ϑ)−∇(θ)‖≤L‖ϑ−θ‖,∀ϑ,θ∈ℝd\| ( )- (θ)\|≤ L\| -θ\|,∀ ,θ ^d. This ensures the objective’s curvature is bounded. Definition 3 (σ-bounded base gradients). We say J has σ-bounded base gradients if the unscaled stochastic gradient satisfies ‖∇i(θ)‖≤σ\| _i(θ)\|≤σ for all i∈[n]i∈[n] and θ∈ℝdθ ^d. Definition 4 (ϵε-accuracy). A stochastic iterative algorithm is said to achieve ϵacc _acc-accuracy in K iterations if the expected gradient norm satisfies min0≤k≤K−1[‖∇(θk)‖2]≤ϵacc _0≤ k≤ K-1E[\| (θ^k)\|^2]≤ _acc. To ensure stable optimization, we construct the learning objective as a deterministic weighted combination of two distinct data sources using a proportion scalar λ∈(0,1)λ∈(0,1). At any training step k, exactly a fraction λ of the learning signal originates from the off-policy intervened trajectories calD_cal, and the remaining 1−λ1-λ originates from the on-policy rollout policy grpoD_grpo. For a given trajectory, the stochastic gradient is scaled by a token-level reweighting coefficient. We define the objective surrogates asymmetrically: Off-policy Intervened Rollouts: To push the current policy beyond its existing capability boundary, we employ the reweighting ρ~calt ρ_cal^t defined in Eq. (6), which is bounded within [1−ϵcal, 1+ϵcal][1- _cal,\,1+ _cal]. In practice, this coefficient is computed separately from the policy-gradient term and is detached from the computation graph; it therefore acts purely as a bounded per-token weight on the advantage: r~calt=sg(ρ~calt), r^t_cal=sg ( ρ_cal^t ), (11) where sg(⋅)sg(·) denotes the stop-gradient operator. Since the coefficient is not differentiated, it introduces no additional gradient path through πcalπ^cal or πθroll _θ^roll: it preserves the forward intervention weight and reweights only the advantage of the tokens it emphasizes, while its bounded magnitude prevents any single intervened token from dominating the update, thereby mitigating gradient amplification and improving optimization stability. Since r~calt r^t_cal deviates from unity by at most ϵcal _cal, this bounded drift is what motivates the cumulative bound Dcal2=4ϵcal2η2σ2D_cal^2=4\, _cal^2\,η^2\,σ^2 imposed in Definition 5. On-policy Rollouts: For samples generated by the on-policy model, we apply the per-token probability ratio ρgrpo,it _grpo,i^t of Eq. (2), bounded by the classic hard clip: r~grpot(θ)=clip(ρgrpo,it,1−ϵclip,1+ϵclip), r^t_grpo(θ)=clip ( _grpo,i^t,1- _clip,1+ _clip ), (12) where the clipped weight satisfies |r~grpot(θ)|≤1+ϵclip| r^t_grpo(θ)|≤ 1+ _clip; for brevity, we denote it by wgrpow_grpo in the following analysis. Definition 5 (Decoupled Dynamics and Trust Region Bounds). We decompose the parameter update step Δθk=θk+1−θk θ^k=θ^k+1-θ^k according to its data source, and impose the following decoupled boundedness assumptions on the two sub-updates (which abstract, rather than rigorously derive from, the reweighting and clipping mechanisms in Eq. (11) and Eq. (12)): • Off-policy Regime (Cumulative Trust Region): The total cumulative second moment of intervened updates over K iterations is assumed to be bounded by a constant independent of K: ∑k=0K−1[‖Δθcalk‖2]≤Dcal2:=4η2ϵcal2σ2, _k=0^K-1E [\| _cal^k\|^2 ]≤ D_cal^2:=4\,η^2 _cal^2σ^2, (13) where η>0η>0 maps the bounded reweighting drift to the parameter space. • On-policy Regime (Step-wise Variance): The update step is given by Δθgrpok=αkwgrpo∇i(θ) _grpo^k= _kw_grpo _i(θ), and its second moment scales with the learning rate αk _k and the weight upper bound (1+ϵclip)(1+ _clip): [‖Δθgrpok‖2]≤αk2(1+ϵclip)2σ2E [\| _grpo^k\|^2 ]≤ _k^2(1+ _clip)^2σ^2 (14) Proof. Following the L-smoothness assumption of the objective function, the iterates of our algorithm satisfy the following expected lower bound: [(θk+1)]≥[(θk)+⟨∇(θk),θk+1−θk⟩−L2‖θk+1−θk‖2] [J(θ^k+1)] [J(θ^k)+ (θ^k),θ^k+1-θ^k - L2\|θ^k+1-θ^k\|^2 ] (15) By defining the actual parameter update step as Δθk=θk+1−θk θ^k=θ^k+1-θ^k, we substitute this definition and the expected proportional gain lower bound into the smoothness inequality: [(θk+1)]≥[(θk)]+αk(1−ϵclip)[‖∇(θk)‖2]−L2[‖Δθk‖2] [J(θ^k+1)] [J(θ^k)]+ _k(1- _clip)E [\| (θ^k)\|^2 ]- L2\,E [\| θ^k\|^2 ] (16) Now, using the mixture decomposition Δθk=λΔθcalk+(1−λ)Δθgrpok θ^k=λ\, _cal^k+(1-λ)\, _grpo^k together with the convexity of ∥⋅∥2\|·\|^2 and the further relaxation 1−λ≤11-λ≤ 1, we substitute the decoupled second-moment bounds from Definition 5 into [‖Δθk‖2]E [\| θ^k\|^2 ]: [(θk+1)]≥[(θk)]+αk(1−ϵclip) [J(θ^k+1)] [J(θ^k)]+ _k(1- _clip) (17) [‖∇(θk)‖2]−L2(λ[‖Δθcalk‖2]+αk2(1+ϵclip)2σ2) [\| (θ^k)\|^2 ]- L2 ( [\| _cal^k\|^2 ]+ _k^2(1+ _clip)^2σ^2 ) Rearranging the terms to isolate the target gradient norm: αk(1−ϵclip)[‖∇(θk)‖2]≤[(θk+1)−(θk)]+L2λ[‖Δθcalk‖2]+L2αk2(1+ϵclip)2σ2. _k(1- _clip)E [\| (θ^k)\|^2\! ] [J(θ^k+1)-J(θ^k)]+ L2 [\| _cal^k\|^2\! ]+ L2 _k^2(1+ _clip)^2σ^2. (18) Now, we sum this inequality across all iterations from k=0k=0 to K−1K-1, assuming a horizon-dependent constant learning rate αk≡α _k≡α, and invoke the decoupled boundedness assumptions in Definition 5: α(1−ϵclip)∑k=0K−1[‖∇(θk)‖2]≤[(θK)−(θ0)]+L2λ∑k=0K−1[‖Δθcalk‖2]⏟≤Dcal2 (Cumulative Bound)+ α(1- _clip) _k=0^K-1E [\| (θ^k)\|^2 ]≤ [J(θ^K)-J(θ^0) ]+ L2λ _k=0^K-1E [\| _cal^k\|^2 ]_≤ D_cal^2 (Cumulative Bound)+ (19) L2∑k=0K−1α2(1+ϵclip)2σ2⏟=Kα2(1+ϵclip)2σ2≤[⋆−(θ0)+L2λDcal2]+L2Kα2(1+ϵclip)2σ2 L2 _k=0^K-1α^2(1+ _clip)^2σ^2_=Kα^2(1+ _clip)^2σ^2≤ [J -J(θ^0)+ L2λ D_cal^2 ]+ L2Kα^2(1+ _clip)^2σ^2 Dividing the entire equation by Kα(1−ϵclip)Kα(1- _clip), and recognizing that the minimum element of a sequence is bounded by its average, we deduce: min0≤k≤K−1[‖∇(θk)‖2]≤⋆−(θ0)+L2λDcal2Kα(1−ϵclip)+Lα(1+ϵclip)2σ22(1−ϵclip) _0≤ k≤ K-1E [\| (θ^k)\|^2 ]≤ J -J(θ^0)+ L2λ D_cal^2Kα(1- _clip)+ Lα(1+ _clip)^2σ^22(1- _clip) (20) To obtain the tightest theoretical bound, we minimize the right-hand side by selecting the learning rate parameter α=c/Kα=c/ K. By the AM-GM inequality, equating the two additive terms yields the optimal constant c: ⋆−(θ0)+L2λDcal2cK(1−ϵclip)=Lc(1+ϵclip)2σ22K(1−ϵclip)⟹c=2[⋆−(θ0)+L2λDcal2]L(1+ϵclip)2σ2 J -J(θ^0)+ L2λ D_cal^2c K(1- _clip)= Lc(1+ _clip)^2σ^22 K(1- _clip) c= 2 [J -J(θ^0)+ L2λ D_cal^2 ]L(1+ _clip)^2σ^2 (21) Substituting the optimal c back into the inequality, we establish the final theoretical bound: min0≤k≤K−1[‖∇(θk)‖2]≤1(1−ϵclip)2[⋆−(θ0)+L2λDcal2]L(1+ϵclip)2σ2K _0≤ k≤ K-1E [\| (θ^k)\|^2 ]≤ 1(1- _clip) 2 [J -J(θ^0)+ L2λ D_cal^2 ]L(1+ _clip)^2σ^2K (22) Under the stated assumptions, RTI enjoys the same (1/K)O(1/ K) stationarity rate as standard clipped policy-gradient methods. The intervention mechanism contributes an additive constant L2λDcal2=2Lλϵcal2η2σ2 L2λ D_cal^2=2Lλ\, _cal^2\,η^2\,σ^2 (23) inside the convergence upper bound. This surcharge grows with the intervention mixing ratio λ and clip width ϵcal _cal, and vanishes when λ=0λ=0, in which case the result reduces to the GRPO-style bound. Therefore, RTI achieves a clean trade-off: it injects capability-expanding guidance through intervention while paying only a precisely characterized additive constant in the convergence upper bound, without changing the asymptotic convergence order. Appendix C More Details of Experimental Settings C.1 Implementation Details. We use Qwen3-8B and Qwen3.5-9B as the backbone policy models. For RTI, all three MAS roles (the Validator, the Corrector, and the Verifier) are instantiated with the policy model itself, so the intervention pipeline introduces no external supervision model. For the teacher-based baselines, including OPD and LUFFY, we pair each policy with a same-generation larger model as the teacher: Qwen3-235B-A22B for Qwen3-8B and Qwen3.5-397B-A17B for Qwen3.5-9B, keeping the teacher and student aligned in tokenizer and model family. For τ2τ^2-Bench, we further use Qwen3.5-397B-A17B as the user simulator in all experiments to ensure a fair comparison. During RL training, we use a binary reward: a trajectory receives reward 11 if it meets the same success criterion as in evaluation, and 00 otherwise. We optimize with Adam using a learning rate of 1×10−61× 10^-6, a constant learning-rate schedule, and weight decay 0.10.1, with rollout temperature 0.80.8 and group size G=8G=8. In Eq. (2), we use the standard GRPO clipping parameter ϵclip=0.2 _clip=0.2; for consistency, the symmetric clipping threshold in reasoning-shift-aware token reweighting is also set to ϵcal=0.2 _cal=0.2. To focus optimization on agent reasoning and decision quality, tokens corresponding to environment feedback are masked out from the training loss. We implement RL training with the qqr framework11 1 https://github.com/Alibaba-NLP/qqr on 8 NVIDIA H20 GPUs with tensor parallelism degree 4, while the teacher models, Qwen3-235B-A22B and Qwen3.5-397B-A17B, are served with vLLM22 2 https://vllm.ai/ on 16 PPU-ZW810E. For the three τ2τ^2-Bench domains, we follow the evaluation protocol of the original benchmark (Barres et al., 2025): a trajectory is judged successful only if the final database state matches the reference and all task-level assertions (e.g., the expected agent communications and side-effect checks) are satisfied. For Navigation, we score a trajectory as successful if and only if the agent successfully invokes the direction tool with the correct parameter configuration—namely the correct origin, destination, and the full set of intermediate waypoints in the required order. For Claw-Eval-Hard20, we follow the official LLM-judge protocol of Claw-Eval (Ye et al., 2026): an official LLM judge (for which we use Qwen3.7-Max) assigns scores according to the official rules and rubric, and a task is counted as passed if task_score=safety×(0.8×completion+0.2×robustness)task\_score=safety×(0.8×completion+0.2×robustness) exceeds 0.750.75. C.2 Benchmark Details C.2.1 Navigation: Real-world Map Routing Task Description. The Navigation benchmark evaluates the agent’s ability to construct a valid multi-waypoint route plan given a natural-language user query. Given a query x (e.g., “How do I drive from Dajin Park to Lihui Park, and then to Yixing Temple?”), the agent must: (1) issue one or more poi_search calls to resolve each mentioned location name to its geographic coordinates, and (2) invoke direction with the correctly assembled origin, destination, and intermediate waypoints to produce the final route. The agent must respect a strict sequential constraint: direction can only be called after all necessary POI coordinates have been retrieved via prior tool calls. Tool Annotations. The Navigation benchmark uses three tools in Amap Open Platform33 3 https://lbs.amap.com/api/webservice: • poi_search: Search for places by text (address or POI name). Returns address, coordinates (longitude, latitude), and business info. Parameters: address (required), region (optional city filter). • around_search: Search for POIs within a circular area. Parameters: location (lat,lon), radius (0–50000m), keyword, region. • direction: Route planning for multiple modes (driving/walking/bicycling/electrobike/transit). Parameters: origin, destination, mode, waypoints (optional, max 16 points). Evaluation Protocol. The evaluation criterion is exact parameter matching of the direction call. Each query is annotated with an order level (0–4) that specifies the visit-sequence constraint imposed by the user’s intent: Order 0 (free ordering among all waypoints), Order 1 (fixed start, free remainder), Order 2 (fixed end, free remainder), Order 3 (fixed start and end, flexible middle), and Order 4 (strict sequential order matching the query’s surface order). The matching rule is order-level-aware: for Order 0, any permutation of intermediate waypoints that yields the correct set is accepted; for Order 4, the waypoints list must match the ground-truth in exact sequence. The agent’s final direction invocation must match the ground-truth call in its origin, destination, and waypoints fields at the coordinate level under the corresponding order constraint. A trajectory receives reward=1 reward=1 if and only if all fields match; otherwise reward=0 reward=0 with no partial credit. Maximum interaction turns are set to 6 per episode. We report Pass@k and Passk for k=1,…,4k=1,…,4. Dataset Construction. To synthesize user queries with deterministic, evaluator-checkable ground-truth routes, we adopt an inverse construction protocol: rather than annotating routes for given queries, we first sample geographically coherent waypoint sets from the map and then synthesize the natural-language query that would induce that route. The pipeline proceeds in three stages. Stage 1 – Spatially-anchored waypoint sampling. We seed each instance from a single anchor POI drawn uniformly at random from a corpus of Amap reverse-geocoding records, and issue an around_search request centered at the anchor with a 50km50\,km radius, restricted to the “scenic spots” category. From the returned candidates we retain a random subset of n∼3,9n \3,9\ POIs, which jointly form the reference waypoint set of the instance and induce the ground-truth direction call. Two design choices are worth emphasizing. First, anchoring all waypoints around a common spatial seed enforces geographic coherence: the resulting waypoint set is realizable as a single trip rather than a random mixture of POIs from disparate regions. Second, randomizing the cardinality n from 33 to 99 exposes the agent to a wide spectrum of long-horizon planning lengths, which is the dominant complexity dimension on this benchmark. Stage 2 – Inverse query generation and order-level annotation. Given a reference waypoint set, we synthesize a natural-language navigation query by inverse-prompting a strong LLM (i.e., Qwen3.7-Max44 4 https://qwen.ai/blog?id=qwen3-max) with the POI names. Query-generation prompt Please generate one question that asks for a route passing through poi names. Feel free to make the phrasing more colloquial to increase query diversity. Output only the generated question, with no additional content. The prompt states only that the generated question must request a route passing through the supplied waypoints and explicitly encourages colloquial paraphrase to widen the query distribution. The LLM is therefore free to choose the rhetorical framing (driving plan, road-trip itinerary, weekend outing), the order in which waypoints are mentioned, and whether to add hedging or destination markers — which collectively reproduce the linguistic variability of real user requests. After query generation, we use the same LLM to annotate the order level of each instance by analyzing the query’s surface-level ordering cues (e.g., explicit “first … then …” markers indicate Order 4, while “pass through the following places” with no ordering words indicates Order 0). This annotation determines the evaluation matching rule applied to the waypoints field. Stage 3 – LLM filtering. Stage 2 produces noisy (query, reference) pairs that are not directly usable: the synthesized query may underspecify, over-specify, or paraphrase away one of the waypoints, and the spatially-sampled reference set may itself contain outliers that no human traveller would bundle into a single trip. We therefore apply an LLM-based semantic verifier (prompt shown below) that judges whether the query unambiguously requests routing through exactly the reference waypoints — with no spurious destinations and no missing waypoints — and whether the reference set is geographically plausible as one continuous trip; pairs failing either check are discarded. filtering prompt (Stage 3) You are a data-quality judge for a navigation dataset. Given a synthesized query and a reference waypoint list, decide whether the pair is valid. [Query] query [Reference Waypoints] waypoint_names Rejection criteria (any one triggers ‘invalid’): 1) The query mentions destinations NOT in the reference list (spurious waypoints). 2) The query omits one or more reference waypoints (missing waypoints). 3) The reference waypoints are geographically implausible as a single continuous trip (e.g., scattered across distant provinces). 4) The query is unnatural, ungrammatical, or clearly machine-generated in tone. Output JSON only: "valid": true/false, "reason": "..." We further split 150150 instances as the held-out test dataset. From the remaining pool we select 600600 instances as the training set. Benchmark Challenges. Navigation is a substantially harder benchmark than it may superficially appear, for three compounding reasons that collectively demand multi-turn, reflective agent behavior. (i) Dense waypoint sets require long-horizon, sequentially dependent tool use. Each instance contains 33–99 reference waypoints, and the success criterion requires that all of them be resolved to coordinates before any direction call may be issued (a hard rule enforced by the evaluator). (i) Ambiguous location names necessitate multi-turn reflection and adaptive re-querying. Real Amap POI names are not unique: a single literal name frequently maps to multiple geographically distant candidates, and the canonical “correct” candidate is determined by the surrounding waypoint context rather than by the name alone. A competent agent must detect when a poi_search return is inconsistent with the rest of the trip and re-issue the search with a refined region constraint. (i) Exact coordinate-level matching as the terminal success criterion. Unlike textual benchmarks where partial credit is plausible, the Navigation evaluator scores a trajectory against the ground-truth API call by exact-matching origin, destination, and the ordered waypoint list at the coordinate level. A wrong coordinate at any of the 33–99 resolution steps is silently absorbed into a syntactically valid but semantically wrong route, producing reward=0 reward=0 with no partial credit. Table 4 summarizes the dataset statistics. Table 4: Statistics of the five datasets. Statistic τ2τ^2-Telecom τ2τ^2-Retail τ2τ^2-Airline Navigation Claw-Eval-Hard20 Train 1148 563 271 600 160 Test 114 114 50 150 20 C.2.2 τ2τ^2-Bench (Telecom / Retail / Airline) Task Description. τ2τ^2-Bench (Barres et al., 2025) is a multi-turn agent benchmark that simulates realistic customer service scenarios across multiple business domains. In each episode, the agent acts as a customer service representative interacting with a simulated user over multiple dialogue turns, calling backend tools (account lookup, order management, policy retrieval, etc.) to resolve the user’s request. The agent must correctly identify the user’s intent, retrieve relevant information, and execute the appropriate operations within the platform’s business rules. The training dataset, AReaL, is distributed under the Apache License 2.0.55 5 https://huggingface.co/datasets/inclusionAI/AReaL-tau2-data The evaluation dataset, τ2τ^2-bench, is distributed under the MIT License.66 6 https://github.com/sierra-research/tau2-bench Evaluation Protocol. Task success rate: the fraction of test episodes in which the agent’s final state fully satisfies the user’s stated request, as judged by the τ2τ^2-Bench ground-truth evaluator. Maximum interaction turns are set to 30 per episode. We report Pass@k and Passk for k=1,…,4k=1,…,4 for each domain separately. C.2.3 Claw-Eval-Hard20 Task Description. Claw-Eval-Hard20 is a hard test set of 20 tasks drawn from the T-series tasks of Claw-Eval (Ye et al., 2026), a human-verified benchmark whose tasks are grounded in real-world environments and require autonomous tool use. Each task presents an open-ended request that the agent must complete by issuing Web tool calls and reading documents over a long interaction horizon, and the selected tasks span multi-step tool orchestration, failure recovery, cross-source synthesis, entity disambiguation, safety-sensitive decisions, document understanding, and numerical reasoning. The subset is deliberately constructed to be simultaneously difficult, solvable, and reliably evaluable: the base policy rarely discovers successful trajectories on it despite sufficient accessible information and valid evaluation signals, so failures reflect genuine capability limits rather than environment artifacts. Claw-Eval-Hard20 thereby directly tests the exploration bottleneck motivating RTI. Evaluation Protocol. We follow the official evaluation protocol of Claw-Eval (Ye et al., 2026). An official LLM judge, for which we use Qwen3.7-Max, scores each completed trajectory according to the official rules and rubric of the corresponding task, producing a safety score, a completion score, and a robustness score. The final task score is computed as task_score=safety×(0.8×completion+0.2×robustness),task\_score=safety×(0.8×completion+0.2×robustness), (24) and a task is counted as passed if task_scoretask\_score exceeds 0.750.75. The safety score enters as a multiplicative gate, so unsafe behavior overrides partial task completion. We report Pass@k and Passk computed under this criterion. Dataset Construction. A low success rate on the original benchmark does not necessarily indicate a genuine capability bottleneck. In the original Claw-Eval (Ye et al., 2026), some tasks are effectively infeasible under the evaluation environment: required evidence may be inaccessible through the available Web tools, become outdated over time, or be truncated by the context-length and tool-output limits. Other tasks contain brittle graders for which semantically correct answers may still fail. Including such tasks would conflate capability-limited failures with environment-limited failures. We therefore construct Claw-Eval-Hard20 as a subset of robustly evaluable hard tasks, paired with aligned synthetic training data. Test-set selection. We select tasks through three stages. First, we perform task-level difficulty profiling over the 160 valid single-turn T-series tasks using Qwen3.5-9B. Each task is evaluated with three independent rollouts, and tasks with Pass@3 above 1/31/3 are excluded. The remaining 63 tasks exhibit low success rates and form the initial candidate pool. Second, we audit the feasibility of each candidate under the actual evaluation environment. We remove tasks whose required evidence is inaccessible or temporally unstable, whose documents exceed the context or tool-output limits, or whose graders cannot reliably recognize valid solutions. We also exclude tasks dominated by isolated knowledge recall or standalone programming rather than agentic tool use. Third, we manually inspect the remaining tasks to verify that (i) the request is well-defined, (i) a valid solution can be obtained using the available observations and tools, (i) the grader matches the intended task semantics, and (iv) failure primarily reflects an agentic capability deficit. This process yields 20 tasks spanning multi-step tool orchestration, failure recovery, cross-source synthesis, entity disambiguation, safety-sensitive decisions, document understanding, and numerical reasoning. As summarized in Table 5, all selected tasks have a base-policy Pass@3 of at most 1/31/3, including 17 tasks with zero successful rollouts. Table 5: Composition and Qwen3.5-9B difficulty of Claw-Eval-Hard20. Task category # Tasks Pass@3 =0/3=0/3 Pass@3 =1/3=1/3 Office automation 12 12 0 Research and visual question answering 3 1 2 Document understanding and aggregation 5 4 1 Total 20 17 3 Training-data synthesis. The 20 original tasks are reserved exclusively for evaluation. To provide aligned training signals without exposing the test instances, we treat each selected task as a task family and generate eight structure-preserving variants, resulting in 160 synthetic training instances. Each variant retains the original task’s tool set, interaction structure, capability requirements, and grading logic, while replacing its entities, dates, numerical values, environment states, documents, and surface realization. The resulting train and test sets therefore share the same underlying capabilities but contain no overlapping instances. We adopt a hybrid programmatic–LLM generation procedure. Programmatic generators construct all correctness-critical elements, including environment fixtures, entity relations, numerical constraints, and ground-truth answers. Qwen3.5-397B-A17B is used only to diversify scenario descriptions and linguistic realizations. This separation allows the LLM to improve linguistic diversity while ensuring that task semantics and evaluation targets remain deterministic. Finally, every synthetic instance undergoes three quality checks. We execute a reference trajectory under the same tool environment and grader used for training, requiring a score of at least 0.990.99 to establish solvability. We then verify consistency among the instruction, environment fixtures, ground-truth constants, and grader. Near-duplicate prompts are further removed using character-level and trigram-based similarity thresholds. Only instances that are executable, evaluator-consistent, and sufficiently diverse are included in the final training set. Benchmark Challenges. Claw-Eval-Hard20 stresses agentic policies along three compounding dimensions. (i) Extremely sparse positive feedback. All 20 selected tasks have a base-policy Pass@3 of at most 1/31/3, and 17 are never solved in any of the three rollouts (Table 5). Standard on-policy RL therefore observes an all-zero reward on 17 of the 20 tasks, leaving the policy essentially no successful experience to bootstrap from. (i) Heterogeneous long-horizon task composition. The 20 tasks jointly cover multi-step tool orchestration, failure recovery, cross-source synthesis, entity disambiguation, safety-sensitive decisions, document understanding, and numerical reasoning. Solving them requires sustained tool use over long interaction horizons and large contexts, so a policy cannot succeed by repeating a single skill pattern; it must plan, verify, and revise its behavior across many steps. (i) Strict, safety-gated success criterion. A task is counted as passed only if its task score exceeds 0.750.75 under a rubric that multiplies the weighted completion and robustness terms by a safety factor, so unsafe behavior overrides partial task progress. The benchmark therefore gives no credit to trajectories that are superficially complete but non-compliant. C.3 More details for Baselines We compare RTI against three categories of baselines as follows: (1) Reference Models. For each backbone we include three tiers of non-RL baselines: • Vanilla Backbone: the policy model itself (Qwen3-8B / Qwen3.5-9B) without any RL post-training, providing the raw starting point. • Larger same-generation Qwen variant (Qwen3-235B77 7 https://huggingface.co/Qwen/Qwen3-235B-A22B paired with Qwen3-8B; Qwen3.5-397B88 8 https://huggingface.co/Qwen/Qwen3.5-397B-A17B paired with Qwen3.5-9B): a much larger model from the same family, quantifying how much performance can be obtained simply by scaling up parameters. • Contemporaneous proprietary models: GPT-4.199 9 https://openai.com/index/gpt-4-1/ and Claude-4-Sonnet1010 10 https://w.anthropic.com/news/claude-4 are paired with the Qwen3 generation as the strongest LLMs available at that time, while GPT-5.11111 11 https://openai.com/index/gpt-5-1 and Claude-4.6-Sonnet1212 12 https://w.anthropic.com/claude/sonnet are paired with the Qwen3.5 generation as the corresponding state-of-the-art LLMs. (2) On-Policy RL. Standard RL methods that optimize the current policy using trajectories sampled from it, with importance sampling-based corrections to account for potential distribution mismatch during policy updates. In addition, we also include AEPO specifically designed for agentic RL: • GRPO (Shao et al., 2024): a policy optimization method with group-normalized advantages. • GSPO (Zheng et al., 2025): a sequence-level RL method that uses sequence-likelihood importance ratios and sequence-level clipping instead of token-level importance weighting, improving optimization efficiency and training stability. • AEPO (Dong et al., 2026a): an agentic-RL-specific method that addresses the instability caused by high-entropy tool-use decisions in long-horizon, multi-turn settings through entropy-balanced rollout scheduling and entropy-aware policy optimization, improving both exploration quality and training stability. (3) Teacher-Guided Methods. Methods that incorporate a stronger teacher model to guide the policy model during training: • OPD (Lu & Thinking Machines Lab, 2025): on-policy distillation whose goal is to push the student distribution toward the teacher’s. Trajectories are generated by the policy model itself, and the stronger teacher scores these student trajectories at the token level; the teacher’s token-level scores are used as weights on the advantage term, so the update signal that pulls the student toward the teacher is computed on the student’s own on-policy data. Our implementation follows the mechanism of the slime framework1313 13 https://github.com/THUDM/slime, computing the per-token teacher signal and folding it into the advantage. • LUFFY (Yan et al., 2026): an off-policy RL approach that augments the policy’s own rollouts with teacher-generated trajectories. Concretely, whenever an entire rollout group fails—all trajectories in the group receive zero reward—we replace the failed group with successful trajectories generated by the stronger teacher model on the same task, so that the policy can still obtain positive learning signals on tasks beyond its unaided capability. Appendix D More Details for Experimental Results Section D.6 first quantifies how the gains on Claw-Eval-Hard20 transfer beyond the task families used for training-data synthesis. The remaining sections use τ2τ^2-Telecom as the running example: Section D.1 walks through one complete MAS intervention step, showing how the Validator, Corrector, and Verifier jointly intervene on a single rollout step. Section D.2 then ablates the intervention pipeline by removing the leakage-suppression clause from the Corrector and the Integrity-Constrained Intervention Verifier, contrasting the resulting trajectory with the full-pipeline output. Section D.3 summarizes the intermediate training-time statistics of MAS intervention. Section D.4 compares the message-level transcripts of GRPO and RTI on the same Navigation query, illustrating the multi-step reflective reasoning paradigm that RTI acquires after training. Section D.5 finally visualizes the tokens on which the reasoning-shift reweighting actually places its weight. D.1 A Complete MAS Intervention on τ2τ^2-Telecom We walk through one complete intervention step of a τ2τ^2-Telecom rollout for the user query: user: Hi, I’m having trouble sending MMS messages. I’m currently in France, and I really need to send a photo to my family back home. Can you help me fix this? Stage 1 – Original policy response. The Qwen3-8B policy directly attempts a tool call with a placeholder argument: Original πθ _θ response <think> The user is having trouble sending MMS messages while in France. I should first verify their network status. </think> <tool_call>"name": "check_network_status", "arguments": "phone_number": "user’s phone number"</tool_call> Stage 2 – Reward-Oriented Trajectory Validator. This response is a user-side device tool that the agent must not invoke. Validator output is_valid: false error_types: [user_side_device_action] suggest_next: ask user to call the check_network_status function Diagnosis. The call to check_network_status is invalid because it corresponds to a user-side device action rather than an agent-executable tool call. Stage 3 – Policy-Aligned Trajectory Corrector. The corrector preserves the policy’s original intent—checking the network state first—while converting the invalid tool call into a valid user-facing instruction. Intervened response u~k u_k <think> The user is having trouble sending MMS while in France. Checking the network status is still the right next step, but this is a user-side device action, so I should ask the user to do it and report back. </think> Sure --- let’s first check whether your phone’s current network settings are causing the MMS issue. Please run the check_network_status function on your device and send me the result. Once you share it, I can help you with the next step. Stage 4 – Integrity-Constrained Intervention Verifier. Verifier output ok: true leakage: none hallucinated_tool_result: none Rationale. The intervened reply correctly converts the invalid agent-side tool invocation into a user-facing instruction to run check_network_status on the device. No privileged information is leaked, no fabricated tool results are claimed, and the action is reasonable given the visible history. Overall, this example shows how MAS-based intervention converts an otherwise failed rollout into a valid and learnable trajectory by correcting the decisive local error while preserving the policy’s original reasoning direction. D.2 Ablating Leakage Suppression To isolate the role of the leakage-suppression clause in the Corrector, we contrast two intervened rollouts of the same task from τ2τ^2-Telecom. RTI Without leakage suppression (Ground-truth leak). This example illustrates why leakage suppression is necessary in the Corrector. Even when the high-level repair direction is correct, a ground-truth-aware corrector may still overfit to hidden supervision and verbalize backend-only information that should never appear in the rollout. Such responses are deceptively successful: they satisfy the task outcome, yet violate the answer-concealment requirement and distort the trajectory distribution away from the policy’s natural reasoning process. The Verifier therefore rejects this intervened response as an invalid training signal. Original policy response (illegal user-side tool call) tool_call: "name": "reset_apn_settings", "arguments": Validator diagnosis is_valid=false, error_types=[user_side_device_action], suggest_next=“Instruct the user to perform: reset_apn_settings() and then reboot_device().” Corrector reasoning_content: “Okay, let me process this. John is having trouble sending MMS from his phone number 555-123-2002 in France. I’ve confirmed that the line (L1002) is active and roaming is enabled, which is good. The data usage is 15.1 GB, which may be near or over the plan limit …According to the policy, I cannot perform device actions like resetting APN settings myself. Instead, I must guide the user to perform reset_apn_settings() on their device. After that, they must reboot the device for the changes to take effect.” tool_calls: [] content: “I’ve checked your account and line settings, John. Your line (555-123-2002) is active and international roaming is enabled, which is correct for your location in France. However, the issue might be with your APN settings, which control how your phone connects to our network for services like MMS. Please perform the following on your device: 1. Run reset_apn_settings() to restore the correct network configuration. 2. After that, run reboot_device() to apply the changes.” Verifier (ok=false) LLM Verifier rationale. “The intervened reply should be rejected because it leaks privileged information that is not supported by the visible conversation history. The customer never stated the name John. By saying ‘I’ve checked your account and line settings, John’, the response both reveals a private identifier and explicitly signals access to backend records that were never surfaced in the interaction. The draft also exposes additional hidden information, including the internal line identifier L1002 and the data-usage value 15.1 GB, neither of which is justified by the user-visible context. Therefore, this intervened response violates the leakage constraint.” RTI With leakage suppression (clean redirection). This contrasting example shows the intended behavior of the full RTI pipeline. Leakage suppression prevents the Corrector from exposing hidden labels or backend-derived facts, while still allowing it to repair the decisive local mistake. The resulting response remains close to the policy’s original intent, uses only visible evidence, and converts the failed action into a valid user-guiding continuation. Such trajectories are both successful and learnable, making them appropriate for downstream policy optimization. Original policy response (illegal user-side tool call) tool_call: "name": "toggle_data", "arguments": Validator diagnosis is_valid=false; error_types=[user_side_device_action]. “toggle_data is listed in USER_SIDE_TOOLS, meaning it must be executed by the user, not the agent.” Corrector draft – instructs the user to run the tool; no internal identifiers tool_calls: [] content: “Let’s proceed with enabling mobile data. Since toggle_data() is a function you need to run on your device, please execute toggle_data() now to turn mobile data ON. After that, run check_network_status() to see if your phone connects to a better network (3G/4G). Let me know the results so we can determine the next steps.” Verifier (ok=true) LLM Verifier rationale. “The intervened reply correctly identifies that toggle_data is a user-side action that the user must perform, not something the agent can execute directly. It provides clear instructions for the user to run toggle_data() themselves, followed by check_network_status() to verify the outcome. No privileged information is leaked, no tool results are fabricated, and the suggested actions are consistent with the conversation history and available tools.” Figure 7: Token-level heatmap of ρ~calt ρ_cal^t on an intervened trajectory D.3 Training-Time Statistics of MAS Intervention Table 6 summarizes the intermediate statistics of the MAS intervention pipeline on τ2τ^2-Telecom with the Qwen3-8B backbone. These numbers characterize how often intervention is triggered, how many rollout steps are flagged as invalid by the Validator, how many generated corrections are accepted or rejected by the Verifier, and the resulting trajectory statistics. The high acceptance rate of intervened trajectories indicates that the Corrector successfully repairs invalid steps while the Verifier ensures the repaired trajectories remain learnable and free of leakage. Table 6: Intermediate statistics of the MAS intervention pipeline on τ2τ^2-Telecom (Qwen3-8B backbone). Statistic Count Ratio Rollout groups with intervention triggered 150 / 1,000 15.0% Rollout steps flagged as invalid by the Validator 8,739 / 15,440 56.60% Corrections accepted by the Verifier 5,166 / 8,739 59.11% Corrections rejected by the Verifier 3,573 / 8,739 40.89% Intervened trajectories with ≥1≥ 1 accepted correction 1,169 / 1,200 97.42% Avg. tool calls per trajectory (corrected vs. original) 0.31 vs. 0.55 Avg. characters per trajectory (corrected vs. original) 1,097 vs. 2,045 D.4 Reasoning Paradigm Difference: RTI vs. GRPO on Navigation This subsection provides the full trajectory-level comparison for the Navigation example. We highlight the key behavioral difference: GRPO recognizes the retrieved POI is abnormal but still proceeds to a malformed direction call, whereas RTI re-queries with a refined region and only then issues the correct direction call. Shared user query and ground truth. user: “How do I drive from Dajin Park to Lihui Park, and then to Yixing Temple? Could you give me a detailed route?” ground-truth API call. direction(origin=Dajinp_Dajin; waypoints=Lihuip_Lihui; destination=YixingTWp_Yixing^TW). GRPO trajectory (reward=0 reward=0). Under GRPO, the agent follows the familiar search–direction–response template. After a name-based search returns a semantically matched but geographically incorrect destination in Beijing, the agent verbally notices the inconsistency but still directly calls direction using the wrong destination coordinates, leading to failure. Turn 1 (assistant). Three parallel poi_search calls without a region constraint: poi_search("Dajin Park"), poi_search("Lihui Park"), poi_search("Yixing Temple"). Turn 1 (returns). Dajin Park →Dajin _Dajin (correct); Lihui Park →Lihui _Lihui (correct); Yixing Temple →Beijing _Beijing (wrong POI in Beijing). Turn 2 (assistant). (reasoning, abbreviated) “The returned Yixing Temple is too far away and likely incorrect, but I already have coordinates for all three landmarks, so I can proceed to route planning.” Turn 2 (assistant tool call). direction(origin=Dajinp_Dajin; waypoints=Lihuip_Lihui; destination=Beijingp_Beijing). Turn 3 (assistant response). Outcome: the destination argument is incorrect because it mixes the two Taiwan landmarks with the Beijing POI; the direction call does not match the ground truth, so reward=0 reward=0. RTI trajectory (reward=1 reward=1). Under RTI, the agent reacts differently to the same wrong initial retrieval. Instead of prematurely committing to route planning, it treats the Beijing result as evidence of search ambiguity, refines the query with a region hint, and only then issues the final direction call with the correct destination. Turn 1 (assistant). Three parallel poi_search calls without a region constraint: poi_search("Dajin Park"), poi_search("Lihui Park"), poi_search("Yixing Temple"). Turn 1 (returns). Dajin Park →Dajin _Dajin (correct); Lihui Park →Lihui _Lihui (correct); Yixing Temple →Beijing _Beijing (wrong POI in Beijing). Turn 2 (assistant). (reasoning, abbreviated) “The returned Yixing Temple is geographically inconsistent with the first two landmarks. I should restrict the search to Taipei rather than directly plan a route with this result.” Turn 2 (assistant tool call). poi_search("Yixing Temple", region="Taipei"). Turn 2 (return). Yixing Temple →YixingTW _Yixing^TW (correct geographically consistent POI). Turn 3 (assistant tool call). direction(origin=Dajinp_Dajin; waypoints=Lihuip_Lihui; destination=YixingTWp_Yixing^TW). Turn 4 (assistant response). Outcome: the final direction call uses the correct waypoint chain and matches the ground truth, so reward=1 reward=1. D.5 Visualization Evidence for Reasoning-Shift-Aware Token Reweighting The reasoning-shift reweighting mechanism is designed to emphasize tokens whose conditional likelihood changes substantially when the prefix is switched from the rollout prompt prollp_roll to the correction prompt pcalp_cal. To illustrate what this mechanism highlights in practice, we provide a visualization on the τ2τ^2-Telecom domain in Figure 7. The darkest tokens correspond to local edits that materially alter the reasoning trajectory, such as a contrastive marker (“Instead”), which signals a switch in action plan; an explicit acknowledgment of an action constraint (“cannot perform …for them”), which corrects the agent’s role boundary; and the redirected action target (“guide the user to re-insert”), which specifies the repaired strategy. By selectively amplifying such tokens during optimization, reasoning-shift reweighting improves credit assignment to the decisive local edits, thereby making the learning signal from corrected trajectories more focused and effective. D.6 Out-of-Distribution Generalization on Claw-Eval The training data of Claw-Eval-Hard20 are synthesized variants of the same 20 task families that constitute the test set, so the results in Section 5.5 primarily measure generalization within the target families. To examine whether the gains extend beyond them, we evaluate the base policy, GRPO, RTI, and the proprietary model Claude-4.6-Sonnet on the full unimodal (text-only) Claw-Eval suite, which contains 199 tasks in total: the 20 target tasks (Hard20-ID) and the remaining 179 tasks (Rest-OOD), whose task families never contribute to the training-data synthesis and which are therefore reported without any difficulty screening. GRPO and RTI are evaluated after training on the 160 synthetic instances, while the base policy serves as an untrained reference. Table 7 reports Pass@1 together with the averages of Pass@k and Passk over k=1,…,4k=1,…,4. Table 7: In-distribution (Hard20-ID) and out-of-distribution (Rest-OOD) generalization on the full unimodal (text-only) Claw-Eval suite (199 tasks) with the Qwen3.5-9B backbone after training on the 160 synthetic instances. Rest-OOD consists of the 179 tasks whose families never contribute to the training-data synthesis. All numbers are percentages. Overall Hard20-ID Rest-OOD Method p@1 avg p@k avg pk p@1 avg p@k avg pk p@1 avg p@k avg pk Claude-4.6-Sonnet 60.10 67.99 45.78 51.25 59.27 44.17 61.07 68.71 45.89 Base 51.51 61.79 41.23 26.25 31.67 19.90 54.34 65.13 43.62 GRPO 56.06 65.63 45.83 52.50 57.71 45.42 56.56 66.53 45.87 RTI 65.20 72.95 56.47 85.00 94.17 73.96 62.92 70.73 54.48 Overall, these results demonstrate that RTI generalizes beyond the task families used for training-data synthesis. On Hard20-ID, Pass@1 rises from 26.25 (base) to 85.00, far above GRPO (52.50). More importantly, the gains carry over to the out-of-distribution Rest-OOD tasks, whose task families never contribute to the training-data synthesis: RTI improves Pass@1 from 54.34 to 62.92 and avg Passk from 43.62 to 54.48 over the base policy, remains ahead of GRPO on all three metrics, and lifts the full-suite Pass@1 from 51.51 to 65.20. The comparatively modest Rest-OOD gains should not be read as weaker transfer. Rest-OOD is reported without any difficulty screening, and, as analyzed in Appendix C.2.3, a non-trivial fraction of Claw-Eval tasks are effectively infeasible or not reliably evaluable under the evaluation environment (e.g., evidence that is inaccessible or outdated, context-length and tool-output limits, or brittle graders that reject semantically correct answers). Environment-limited failures of this kind cap the attainable scores and make the numbers above conservative, indicating that on the tasks that are in fact solvable RTI has essentially reached the ceiling achievable under the evaluation environment. We attribute this transfer mainly to improved reflection and reasoning: the corrected trajectories teach the model to detect inconsistencies in its own reasoning and to revise its plan accordingly, rather than memorize task-specific solutions, so the acquired multi-step reflective behavior carries over to tasks it has never seen.