Paper deep dive
ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction
Yongqi Tong, Tan Li Hui Faith, Choy Zhen Wen Marcus, Zhou Jin, Kewei Fu, Jiang-Ming Yang, Jianshe Li, Xin Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/17/2026, 3:51:26 AM
Summary
The paper introduces ARC (Advantage Regularization via Conditioning), a training method for open-ended agent interaction that addresses the 'reward fairness problem' where comparing diverse interaction strategies within a single group biases relative advantage estimation. ARC uses strategy-conditioned rollout grouping to ensure fair comparison. It is evaluated within the INTER3 framework, which decouples user-visible communication from latent reasoning, and the INTER3-86K dataset, a strategy-annotated corpus. Results show ARC improves tool-use benchmarks and INTER3 reduces time-to-first-token significantly.
Entities (11)
Relation Signals (11)
ARC → solves → Reward Fairness Problem
confidence 95% · We formalize this as a reward fairness problem and propose ARC (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison
ARC → uses → strategy-conditioned rollout grouping
confidence 94% · ARC restores fairer relative comparison through strategy-conditioned rollout grouping
INTER3 → provides → INTER3-86K
confidence 93% · INTER3 also provides the annotation and distillation pipeline for constructing INTER3-86K
INTER3 → reduces → Time-to-First-Token
confidence 92% · INTER3 reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline.
ARC → improves → tool-use benchmarks
confidence 91% · Empirically, ARC substantially strengthens the core τ/τ2 tool-use benchmarks
INTER3 → supports → Direct Answer
confidence 90% · We organize these behaviors into four high-level strategy families: ... Direct Answer ... INTER3 ... supports explicit strategy control
INTER3 → supports → Clarify First
confidence 90% · We organize these behaviors into four high-level strategy families: ... Clarify First ... INTER3 ... supports explicit strategy control
INTER3 → supports →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a \textit{reward fairness problem} and propose \textbf{ARC} (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed \inter, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. \inter\ also provides the annotation and distillation pipeline for constructing \inter-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core $\tau/\tau^2$ tool-use benchmarks, while \inter\ reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and \inter-86K training data will be released.
Tags
Links
- Source: https://arxiv.org/abs/2608.13622v1
- Canonical: https://arxiv.org/abs/2608.13622v1
Trouble viewing inline? Open PDF directly →
Full Text
93,092 characters extracted from source content.
Expand or collapse full text
TECHNICAL REPORT August 13, 2026 ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction Yongqi Tong* Tan Li Hui Faith* Choy Zhen Wen Marcus* Zhou Jin Kewei Fu Jiang-Ming Yang Jianshe Li Xin Zhang Ant International tongyongqi.yq, faith.t, marcus.choy, xiaocao.zj, fukewei.fkw, jmyang, zhouran.ljs, evan.zx@ant-intl.com Framework Code: ASRI Dataset: ARC Abstract Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a reward fairness problem and propose ARC (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed INTER3, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. INTER3 also provides the annotation and distillation pipeline for constructing INTER3-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core τ/τ2τ/τ^2 tool-use benchmarks, while INTER3 reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and INTER3-86K training data will be released. Keywords: Reinforcement learning, AI agents, Tool Use, Interaction Strategies, Advantage Estimation 1 Introduction Figure 1: Overview of ARC. ARC modifies group-based RL by pairing each example with a strategy instruction during training, so rollouts are compared only within strategy-conditioned groups, yielding cleaner relative advantages by eliminating cross-strategy contamination. At inference, the instruction is removed and the policy selects strategies autonomously, making ARC a general recipe for fairer advantage estimation in open-ended interaction. Group-based RL and RLHF methods learn from relative rewards within sampled groups and have become a standard post-training recipe for language models and agents 41; 1; 42; 63; 27. Their signal is most interpretable when the compared rollouts occupy a locally comparable response region, so that centered rewards primarily reflect quality rather than qualitatively different behavioral modes. In practice, however, reward models are known to depend on spurious attributes such as response length and stylistic form, and RL policies can exploit these biases during optimization 6; 12. When a rollout group spans sharply different behaviors, these biases need not be uniform across the group: samples may fall in different regions of the reward model’s preference landscape, so the centered reward can absorb both quality differences and region-specific reward preference. The comparison remains relative, but it is no longer fair. This problem is especially acute in authentic real-world interaction, where many tasks are open-ended at the turn level and a single dialogue state may admit several appropriate next actions 62; 60; 4. An agent may answer directly, ask for clarification, provide progress updates while tools run, or confirm before an irreversible action; which choice is best can depend on missing information, action reversibility, and the evolving user-agent state 60; 4; 11; 51. Unlike short-form reasoning domains with a single verifiable target, these interaction choices often do not have a unique standard answer even when they are all task-appropriate 60; 4. Comparing them inside one rollout group therefore entangles strategic diversity with reward-model preference. We formulate this confounding as the reward fairness problem: when multiple strategies are valid for the same prompt, cross-strategy comparison biases group-relative advantage estimation and can skew optimization toward reward-preferred interaction styles. We address this problem with ARC (Advantage Regularization via Conditioning). ARC assigns a training-time strategy instruction to each example, samples multiple rollouts within that strategy-conditioned comparison class, computes relative advantages only within the group, and updates the policy with our hybrid reward and entropy-regularized objective. Figure 1 gives a high-level overview. At inference, the strategy instruction is removed and the policy selects an interaction strategy autonomously. This differs from hint- or guidance-based RL methods, which use auxiliary information to reveal solution structure, privilege stronger traces, or improve exploration 23; 18; 59; 66. ARC uses auxiliary information for a different purpose: not to make the answer easier to find, but to compare rollouts fairly with one another. However, existing agentic frameworks remain a weak substrate for studying this problem under authentic interaction, where users may interrupt, redirect, pause, or lose patience before execution completes, and where trajectories that achieve the same task outcome can still induce materially different interaction experiences 60; 4; 68; 24; 67. Think-then-act agents typically postpone user-visible communication until the hidden reasoning-and-tool trajectory terminates, while ReAct agents intermingle reasoning, acting, and communication in a single visible trace 62; 22; 57. Both paradigms are optimized primarily for backbone capability and task success. They therefore primarily accumulate trajectories with verifiable outcomes, rather than open-ended interaction data in which multiple valid interleaving strategies may all solve the task yet induce different user experiences. We therefore build INTER3 (Interplay of Internal Reasoning, Tool Usage, and Interaction), a channel-separated framework whose system prompt explicitly defines interaction strategies aligned with the strategy families used by ARC. By separating <answer> spans from latent reasoning and tool execution, INTER3 turns progress updates, clarification, alignment checks, and mid-execution steering into first-class, controllable, and annotatable behaviors; this both broadens dialogue-style diversity and gives ARC cleaner rollout groups for fairer comparison. The same interface reduces TTFT from 4.91s to 1.27s relative to a think-then-act baseline. After deployment, we collect real online interaction traces in this format, augment them with curated public data, and synthesize large-scale diversified interaction trajectories aligned with ARC’s strategy families, yielding INTER3-86K, a strategy-annotated corpus of 86K examples and a realistic substrate for fair cross-strategy comparison. Our theoretical analysis isolates inter-strategy variance as a source of estimator error in standard group-based RL and shows that ideal strategy conditioning removes this term from the centered-advantage variance decomposition. Our empirical results then show that, when trained on INTER3-86K, ARC substantially improves open-ended agentic RL for tool-usage settings. Additional diagnostics make the mechanism more concrete: strategy hints matter during training, while omitting them at inference recovers the best overall behavior, indicating that ARC is not merely fitting to prompt-side instructions but learning a more general interaction policy. Together, INTER3 and ARC address both sides of the problem: INTER3 provides a realistic interaction framework in which diverse valid behaviors can be observed and collected, while ARC resolves how those behaviors should be compared during optimization. We view this combination as a practical step toward open-ended real-world agent training, where the central challenge is to learn from realistic interaction data while preserving fair relative comparison when multiple valid behaviors admit no single exact answer. Our contributions are threefold: • We propose ARC, a strategy-conditioned group-RL recipe for mitigating unfair advantage comparison in open-ended real-world agent interaction, and we analyze how it reduces inter-strategy variance in group-based advantage estimation. • We build INTER3, an async streaming interaction framework that separates user-visible communication from latent reasoning and tool execution, supports explicit strategy control and user interruption, and makes open-ended interaction observable for training. • We construct INTER3-86K, a strategy-annotated training substrate from real online traces, curated public data, and synthetic trajectories, and show it enables ARC to improve open-ended agentic RL while generalizing best to hint-free inference. 2 Related Work Tool-augmented agents have made rapid progress in reasoning, planning, and external action 40; 34; 62; 11; 35; 49, while recent interleaving work has improved responsiveness by alternating between thought and partial output 22; 57. Our departure is architectural: INTER3 separates visible interaction from latent reasoning and tool execution, so the user need not wait for the entire internal trajectory to finish before the system can communicate. This turns responsiveness from a decoding behavior into an interface property, and makes strategy diversity operational in a realistic agent loop. On the RL side, our work builds on policy-gradient and RLHF estimators 41; 1; 37; 13, especially group-relative methods such as GRPO and its descendants 42; 63; 27. These methods are effective when samples within a group are meaningfully comparable. Our focus is a different failure mode: in open-ended agent interaction, valid rollouts may differ in strategy rather than only in quality, so cross-strategy comparison can inject reward-model preferences directly into relative advantage estimates. ARC targets this comparability problem at rollout construction time. The closest methodological neighbors are guidance-augmented RL approaches such as SAGE, Scaf-GRPO, LUFFY, and ExGRPO 23; 18; 59; 66, which introduce auxiliary signals to improve exploration, mitigate sparse rewards, or reuse successful experience. ARC also uses an auxiliary instruction, but for a different reason: not to make the task easier, but to define a cleaner comparison class for group-relative optimization. Additional discussion of adjacent tool-use, interaction, and guidance literatures is deferred to Appendix B. 3 The INTER3 Setting 3.1 Interaction Interface ARC is evaluated in INTER3, a channel-separated interaction setting for open-ended agents. The core design choice of INTER3 is to separate the interaction channel from the execution channel. Plain text outside <answer> is treated as latent reasoning, <answer> spans are streamed to the user, and tool calls remain structured internal actions whose results are returned to context. This makes user-visible communication available before the hidden reasoning-and-tool trajectory terminates, and it allows the same underlying task to be paired with different valid communication strategies. The mechanism is model-agnostic and requires only two implementation changes: adding <answer> tags to the tokenizer, and post-processing generated text to extract the user-visible spans. Figure 2 illustrates the resulting execution pattern. Figure 2: Standard think-then-answer (left) completes reasoning and tool execution before answering, resulting in high TTFT and no steering, whereas INTER3 (right) interleaves reasoning, answering, and tool use, allowing real-time steering and faster TTFT. 3.2 Runtime Design Rationale INTER3 is instantiated in a lightweight async streaming agent runtime because standard agent traces are a weak substrate for the interaction behaviors studied in this paper. Strict think-then-act pipelines delay user-visible communication until execution completes, while monolithic ReAct-style traces intermingle reasoning, acting, and communication in a single stream. Neither produces clean open-ended interaction data in which behaviors such as progress updates, clarification, alignment checks, and redirection are explicit and separately annotatable. Because <answer> spans can be emitted before hidden execution completes, channel-separated interaction enables a substantially lower-latency operating regime. As shown in Figure 3, this reduces perceived latency while preserving continuous interaction during execution. Our runtime preserves these behaviors as first-class events within a unified session. A single agent loop can continue hidden reasoning, emit user-visible <answer> spans, issue tool calls, and respond to user interruptions, making interaction strategy observable throughout execution rather than only in the final outcome. Two design choices are key for training. First, prompt-side strategy definitions are explicit and aligned with ARC conditioning families, so runtime control, annotation, and rollout grouping share consistent behavioral variables. Second, partial outputs, interruptions, tool results, and resumed continuations are normalized into a unified trace format, enabling post-deployment logs, curated public data, and synthetic trajectories to be converted into the same strategy-conditioned training substrate. Figure 3: Latency-capability trade-off for interactive agent architectures. Channel-separated interaction shifts the operating point into a substantially lower-latency regime, while ARC improves capability within that regime rather than causing the horizontal TTFT shift itself. Non-INTER3 baselines are schematic architectural placements rather than exact benchmark claims. This separation makes interaction strategy explicit: the same task can be completed through different valid communication patterns while sharing the same execution substrate. That is precisely the regime in which cross-strategy comparison becomes measurable. We organize these behaviors into four high-level strategy families: Progress Update, Clarify First, Alignment Check, and Direct Answer. ARC uses these families only as training-time comparison classes, not as inference-time requirements. We instantiate this setting with INTER3-86K, a strategy-annotated corpus of 86.8K examples spanning tool use, multi-hop QA, and logical reasoning (57.9K SFT, 28.9K RL with strategy instructions). Since INTER3 decouples communication from computation, it reduces user-visible latency; we treat this as an architectural property rather than evidence for ARC. Full strategy definitions and dataset construction details are deferred to Section 4. 4 INTER3-86K Construction This section presents the construction of INTER3-86K, including strategy taxonomy, data sources, curation, strategy annotation, and dataset statistics. 4.1 Strategy Taxonomy INTER3 organizes interaction behavior into four high-level families and nine concrete strategies as seen in Table 1. ARC conditions rollout groups on the high-level family during training; the finer-grained strategies are useful for analysis, and interface documentation. Table 1: INTER3 strategy taxonomy used for ARC conditioning, analysis, and data construction. Category Strategy Description Progress Update Tool Execute Execute single or sequential tool calls; return the final answer upon completion Parallel Tools Execute independent tool calls simultaneously; synthesize combined results Multi-Step Update Stream incremental progress updates to the user between sequential sub-tasks Silent Execution Execute tool calls without emitting user-visible output when intermediate steps are irrelevant Error Recovery Handle tool failures gracefully and recover with corrective actions Clarify First Clarify First Ask the user for missing or ambiguous information before executing any tool call Alignment Check Alignment Check Restate the user’s intent for confirmation before executing an irreversible action Decision Support Present options and trade-offs for the user to make a judgment Direct Answer Direct Answer Answer immediately from internal knowledge without invoking any tools 4.2 Data Sources and Curation We construct INTER3-86K from two sources. The first comes from real-world deployment in customer service on a large-scale global payment platform under the INTER3 runtime, where we collect online interaction traces exhibiting interruption, redirection, clarification, progress updates, and multi-step tool use. After de-identification and normalization into the INTER3 format, these traces provide realistic open-ended interaction patterns that are difficult to recover from standard offline benchmarks alone. The second source combines curated public benchmarks with teacher-driven augmentation, distillation, and synthesis. Public tool-use and reasoning data are rewritten into the INTER3 format, expanded with strategy-conditioned variants, and supplemented with diversified synthetic interaction trajectories produced with a strong teacher model, Qwen3.5-397B-A17B 36. The resulting SFT split contains 57.9K examples and mixes tool-use, multi-hop QA, and logical reasoning: 34.2K tool-use examples (59.1%) from the ToolMind collection, which aggregates public function-calling and agent benchmarks 10; 26; 60; 69; 2; 39; 8; 17.2K multi-hop QA examples (29.7%) from Musique Long Content 52; and 6.4K logical-reasoning or high-quality interleaved examples (11.1%) from KnightsAndKnaves 56 and Opus Distilled 46. The RL split contains 28.9K examples and is drawn entirely from tool-use data, to which we add strategy-conditioned prompts for ARC training. Figure 4 summarizes the resulting domain and strategy distributions. 4.3 SFT Data Construction The SFT set draws from four data families with complementary reasoning demands. The largest component (∼ 34.2K, 59.1%) is the ToolMind collection, which aggregates seven publicly available tool-use benchmarks spanning diverse agentic scenarios. Since these benchmarks include ground-truth chain-of-thought annotations, we use a 397B LLM (Qwen3.5-397B-A17B 36) to rewrite each assistant turn into the INTER3 format—interleaving internal reasoning with <answer> blocks and explicit tool calls—then filter malformed dialogues (invalid turn orderings, empty <answer> blocks). Three supplementary sources diversify the model’s interleaved reasoning beyond tool calling: KnightsAndKnaves (∼ 6.2K, 10.7%) for multi-step deductive inference, Musique Long Content (∼ 17.2K, 29.7%) for multi-hop reasoning over extended passages, and Opus Distilled (∼ 250, 0.4%) for high-quality interleaved conversations generated via Claude Opus. 4.4 RL Data Construction The RL set (∼ 28.9K) reuses the seven ToolMind sub-datasets but introduces training-time strategy instructions essential for ARC training. Each conversation undergoes a two-phase process: (1) a strategy-annotation pipeline assigns one plausible strategy instruction from our taxonomy—Progress Update, Direct Answer, Clarify First, or Alignment Check—based on the query, context, and response; (2) the selected instruction is injected into the system prompt, providing an explicit behavioral condition for RL+ARC training. These prompt-side strategy definitions are aligned with the high-level strategy families used by ARC, so the interaction interface, data annotation pipeline, and rollout comparison classes share the same control variables. In contrast, baseline RL training is performed on the same data without any injected strategy instruction. Figure 4: Left: INTER3 strategy distribution in the RL training dataset. Progress Update dominates at 75.3%, reflecting the prevalence of multi-step tool-use scenarios that benefit from incremental user updates. Right: data domain composition of the full training set (SFT + RL). Tool Use accounts for 72.8% of examples, with Multi-hop QA (19.8%) and Logical Reasoning (7.1%) providing complementary reasoning diversity. 4.5 Strategy Annotation A core contribution of our dataset is the explicit strategy annotation for each example. Unlike prior work that treats agent behavior as monolithic, we recognize that effective interaction requires behavioral diversity adapted to context. We develop a rigorous annotation pipeline combining model-based labeling with collaborative verification. We annotate examples on the four high-level strategies—Progress Update, Clarify First, Alignment Check, and Direct Answer. 4.5.1 Collaborative Annotation Pipeline We employ a collaborative annotation approach using two large language models (Qwen3-235B-Instruct 45 and Qwen3.5-27B 36) to ensure annotation quality and consistency. This dual-model verification captures the inherent subjectivity in strategy assignment while maintaining high inter-annotator agreement. Stage 1: Independent Annotation. For each example, both models independently predict a plausible primary strategy given: • The user query • The conversation history • The reference assistant response • The 4 high-level strategies taxonomy with detailed definitions and examples Each model outputs: (1) a predicted strategy assignment, (2) a confidence score, and (3) a brief justification. Stage 2: Agreement and Disagreement Resolution. • Agreement cases: When both models predict the same strategy with confidence >>0.85, we accept the annotation directly. • High-confidence disagreement: When models disagree but one has significantly higher confidence (Δ>0.15 >0.15), we accept the higher-confidence prediction. • Low-confidence or ambiguous disagreement: We escalate to human review. Three trained annotators independently label the example. We determine the final annotation using a majority vote. 4.6 Dataset Statistics Table 2 presents comprehensive statistics of the final dataset. Table 2: Inter-86K summary statistics. Token counts use whitespace tokenization. Statistic Value Total examples 86,796 Token Statistics Average input tokens 702.1 Average output tokens 264.5 Average total tokens 966.6 Tool Statistics Examples with tool calls 54,856 (63.2%) Average tools per example 4.9 Max tools in single example 38 Conversation Statistics Single-turn conversations 55,376 (63.8%) Multi-turn conversations 31,420 (36.2%) Average turns (multi-turn) 12.5 <answer> Tag Statistics Examples with <answer> tags 86,780 (100.0%) Average <answer> segments 3.48 Average tokens in <answer> 25.3 5 Our Methodology: Advantage Regularization via Conditioning While the INTER3 setting makes interaction strategy observable, it does not solve the core RL challenge: how to compare responses fairly when they follow different communication patterns. We present ARC (Advantage Regularization via Conditioning), a conditioning-based RL method for open-ended agent training. ARC’s core mechanism is strategy-conditioned rollout grouping, which changes the comparison unit in group-based RL: relative advantages are computed within a strategy-conditioned subspace rather than across heterogeneous behaviors. 5.1 Problem: Unfair Advantages in Multi-Strategy RL In group-based RL, given prompt x, we sample N responses y1,…,yN\y_1,…,y_N\ from πθ(⋅|x) _θ(·|x) and compute advantages A^i=ri−r¯ A_i=r_i- r, assuming comparable samples. When responses follow different strategies, this breaks: reward model bias contaminates the advantage signal, skewing policy updates toward reward-preferred behaviors regardless of task appropriateness. Definition 1 (δ-Reward Fairness). A reward model ℛR is δ-fair w.r.t. strategy set S if for any prompt x and responses yi,yjy_i,y_j following different strategies with equal quality: |ℛ(x,yi)−ℛ(x,yj)|≤δ|R(x,y_i)-R(x,y_j)|≤δ. A reward model is fair if δ=0δ=0, and unfair when δ>0δ>0 introduces systematic bias across strategies. In practice, reward models violate this fairness property due to exposure, length, and style bias—especially pronounced in agent settings where interaction appropriateness lacks clear ground truth. 5.2 Strategy-Conditioned Advantage Estimation The following results analyze a stylized prompt-conditional reward decomposition of the form r=μs(x)+ϵr= _s(x)+ε. They isolate one source of estimator variance in group-relative advantages and should not be read as end-to-end convergence guarantees for GRPO. This estimator-centric viewpoint is closer to classical variance-reduction analyses for policy gradients 16 and recent RLHF-side discussions of estimator behavior 1; 27 than to a full convergence theory. For clarity, the theoretical comparison uses equal group size across the conditioned and unconditioned cases. Theorem 5.1 (Idealized Variance Amplification). Under standard group-based RL sampling, the advantage variance is Var[A^i]=(σinter2+σintra2)(1−1/N)Var[ A_i]=(σ^2_inter+σ^2_intra)(1-1/N), where σintra2σ^2_intra captures within-strategy noise and σinter2σ^2_inter captures between-strategy variance from reward model bias. The inter-strategy component σinter2σ^2_inter persists under standard sampling: no amount of oversampling can eliminate it. Our solution is to condition each rollout group on a specific strategy. Definition 2 (Strategy-Conditioned Sampling). For each training example, we construct a strategy-conditioned prompt p∗p^* by attaching a strategy instruction s∗s^* from our interaction taxonomy, then sample M responses from πθ(⋅|x,s∗) _θ(·|x,s^*). Theorem 5.2 (Idealized Variance Reduction via Conditioning). Under strategy-conditioned sampling with the same group size: Var[A^i|s∗]=σintra2⋅(1−1/N)Var[ A_i|s^*]=σ^2_intra·(1-1/N). Under perfect conditioning, the between-strategy mean-shift term is absent from this centered-advantage variance decomposition. Intuition. Heuristically, strategy conditioning emphasizes within-strategy reward dependence I(r;y∣x,s)I(r;y x,s) while suppressing the strategy-linked term I(r;s∣x)I(r;s x) that can contaminate cross-strategy comparison. Unlike hint-based RL methods 23; 18 that use hints to guide toward correct answers, we use strategy instructions to enforce cleaner within-strategy comparisons across diverse behaviors. See Appendix A for proofs and analysis. Training-time strategy instructions do introduce a train-inference mismatch because they are removed at deployment. We study progressive instruction removal empirically in Section 6.4 as a further analysis rather than as part of ARC itself. 5.3 Entropy Regularization for Multi-Channel Generation The ARC training objective combines strategy-conditioned policy gradients with an entropy bonus: ℒ(θ)=−∑i=1MA^i(s∗)⋅logπθ(yi(s∗)|x,s∗)−β⋅H(πθ(⋅|x,s∗)).L(θ)=- _i=1^M A_i^(s^*)· _θ(y_i^(s^*)|x,s^*)-β· H\! ( _θ(·|x,s^*) ). (1) Why entropy regularization is critical for INTER3. The interleaved output format requires balancing three channels—internal reasoning, user-facing <answer> tags, and tool calls. Without the entropy bonus (β=0β=0), we observe entropy collapse 19: the policy converges to emitting redundant <answer> blocks that receive marginal format rewards but carry no meaningful content. The entropy bonus counteracts this collapse by maintaining stochasticity across all output channels, encouraging diverse exploration of valid interleaving patterns. 5.4 Idealized Sample-Efficiency View Theorem 5.3 (Idealized Gradient-Sample Scaling). Under the same stylized assumptions, if gradient-estimation error scales with the second moment of the score-function estimator, then the rollout requirement to estimate ∇J(θ)∇ J(θ) up to accuracy ε scales as nstd=O((σintra2+σinter2)log(1/δ)ε2)n_std=O\! ( (σ^2_intra+σ^2_inter) (1/δ) ^2 ) for standard GRPO and nARC=O(σintra2log(1/δ)ε2)n_ARC=O\! ( σ^2_intra (1/δ) ^2 ) for ARC. Corollary 1. Under the same assumptions, the implied sample-efficiency ratio is nstd/nARC=1+σinter2/σintra2n_std/n_ARC=1+σ^2_inter/σ^2_intra. This is best read as an idealized scaling comparison rather than a full convergence guarantee, in the same spirit as estimator-level policy-gradient analyses 16; 65. We use it as an estimator-level interpretation of why cleaner within-strategy comparison can reduce the gradient-sampling burden, rather than as a calibrated empirical speedup claim. See Appendix A for the derivation and assumptions. 5.5 Implementation ARC consists of four steps (Figure 1): 1. Strategy Instruction Assignment: each training example is paired with one strategy instruction s∗s^* from the interaction taxonomy. 2. Within-Strategy Sampling: sample M rollouts from πθ(⋅∣x,s∗) _θ(· x,s^*). 3. Advantage Computation: compute advantages within each strategy group. 4. Policy Update: update the policy using the entropy-regularized objective. At inference, no strategy instruction is provided; the model autonomously selects appropriate strategies. Section 6.4 further analyzes instruction-removal curricula as an auxiliary experiment. 6 Experiments 6.1 Experimental Setup Model and training. We use Qwen3-8B 45 trained with GRPO 42 in no-think mode, warm-started from the same INTER3 SFT checkpoint for all RL backbones. ARC’s core mechanism is strategy-conditioned rollout grouping; in the full INTER3 instantiation additionally apply entropy regularization as a stabilizer for multi-channel generation. Unless explicitly varied, all methods use the same final reward construction, selected based on ablations in Appendix C. Additional ablations on entropy regularization are provided in Appendix D. Benchmarks. We evaluate on two dimensions: (1) In-domain agentic capabilities using tau-bench 60 and tau2-bench 4 for multi-turn tool calling in airline, retail, and telecom scenarios; (2) Out-of-domain reasoning using Arena-Hard 20, AIME 2026 3, GPQA-Diamond 38, IFBench 32, and HMMT 3. Baseline methods. We benchmark ARC against two classes of baselines: (1) Qwen3-8B-noThink / Think, minimal reasoning variants to quantify ARC’s gains; (2) Standard RL backbones (PPO 41, DAPO 63, GRPO 42), representing widely-used reinforcement learning methods. 6.2 Main Results Figure 5: Training reward curves for GRPO and ARC. Table 3: Main results. ARC delivers clearest gains on in-domain tool use, especially on GRPO. Avg. is the mean over displayed metrics; red parentheses show Avg. change vs. backbone. TTFT (↓ ) averages available τ-bench latency measurements. Method Avg. TTFT ↓ Tool Calling Reasoning Instruction Following Alignment τ-airline τ-retail τ2τ^2-airline τ2τ^2-retail τ2τ^2-telecom AIME 2026 GPQA-D HMMT 2025 IFBench ArenaHard Baselines Qwen3-8B-noThink 22.85 0.05s 12.00 29.86 14.61 36.55 17.80 17.08 45.71 12.92 24.83 17.14 Qwen3-8B-Think 32.82 4.91s 28.00 36.81 29.75 38.71 23.46 47.92 53.54 20.83 19.43 29.83 RL Backbones PPO 27.49 0.45s 35.33 41.45 33.33 38.89 19.01 30.83 38.70 8.75 19.73 8.88 PPO + ARC (Ours) 28.57 (+1.08) 0.78s 39.33 46.09 41.61 44.44 20.76 19.17 41.10 10.00 14.29 8.92 DAPO 28.61 0.62s 31.33 42.32 35.33 42.11 19.37 36.67 39.02 13.33 12.24 14.36 DAPO + ARC (Ours) 29.92 (+1.31) 0.82s 34.67 41.74 34.50 40.64 19.64 41.25 43.94 12.50 17.35 12.92 GRPO 28.09 0.61s 31.33 40.29 36.67 40.64 17.84 31.67 40.91 9.17 18.71 13.66 GRPO + ARC (Ours) 33.46 (+5.37) 1.27s 44.00 50.00 48.00 45.61 21.05 40.83 41.41 12.50 15.99 15.18 In-domain performance. Table 3 shows that ARC consistently improves performance across all RL backbones, with the largest gains for GRPO (+5.37 average points). Improvements are especially pronounced in in-domain tool-calling tasks: for GRPO, τ-airline rises from 31.33 to 44.00, τ-retail from 40.29 to 50.00, and τ2τ^2-airline from 36.67 to 48.00, with smaller gains in τ2τ^2-retail and τ2τ^2-telecom. These results demonstrate ARC’s ability to enhance multi-turn tool use and in-domain capabilities, particularly with stronger RL backbones. Out-of-domain reasoning and tradeoffs. On out-of-domain reasoning, instruction following, and alignment benchmarks, ARC shows more nuanced effects. GRPO + ARC improves reasoning on AIME 2026 from 31.67 to 40.83 and provides modest gains in instruction-following and alignment metrics, while improvements for PPO and DAPO backbones are mixed, suggesting that ARC’s benefits depend on the underlying optimization dynamics. These results show that while ARC’s primary impact is on structured tool use, it can also unlock meaningful out-of-domain reasoning gains under suitable training regimes. Training dynamics and robustness. Figure 5 shows that ARC’s benefits extend beyond final performance to training dynamics. While GRPO peaks mid-training and then declines—especially in tool-call and answer rewards—ARC maintains or improves across the same budget. Since format reward saturates for both methods, this divergence reflects execution and answer quality rather than syntactic compliance. This suggests that standard group-based RL initially learns useful behaviors but then exploits reward model biases, whereas ARC’s strategy-conditioned comparison prevents degradation by preserving fair advantage signals throughout training. 6.3 Small-Model Case Study Table 4: Qwen3-4B case study. ARC remains effective at smaller scale and outperforms no-think, think, and 4B GRPO baselines on the in-domain tool-use suite. Method Avg. τ airline τ retail τ2τ^2 airline τ2τ^2 retail τ2τ^2 telecom Qwen3-4B-noThink 22.38 20.00 22.32 21.40 26.31 21.90 Qwen3-4B-Think 29.54 27.51 37.10 28.07 29.21 25.83 GRPO (4B) 22.33 29.33 10.72 36.94 15.96 18.69 GRPO+ARC (4B) 34.23 29.33 40.00 32.00 39.77 30.03 Table 4 shows that the agentic benefit of ARC is not specific to the 8B scale. Even at 4B, ARC remains stronger than no-think, think, and the corresponding 4B GRPO baseline on the tool-use suite, increasing the five-task average by about 53% over 4B no-think. This finding suggests that ARC improves performance by strengthening the training comparison signal in agentic settings, rather than simply leveraging larger model capacity. 6.4 Curriculum Learning over Training-Time Strategy Instructions ARC uses strategy instructions during training but removes them at inference, motivating a curriculum-learning question: should the conditioning signal be gradually weakened so the policy learns more autonomous strategy selection without losing ARC’s variance-control benefit. Inspired by curriculum-learning methods in RL 5; 15; 14; 28, we study this as a diagnostic analysis of the trade-off between train-inference alignment and within-strategy comparability. We compare three settings: (1) No removal, the default ARC setup that always retains the strategy instruction; (2) Linear removal, where the instruction-drop probability increases over training according to Appendix A, pd(t)=pmin+(pmax−pmin)tTp_d(t)=p_ +(p_ -p_ ) tT; and (3) Constant removal, which drops instructions with fixed probability pd=0.20p_d=0.20 throughout training. Figure 6: Training-time instruction removal underperforms the default ARC setup. Left: average number of off-strategy rollouts per group, where an rollout has a predicted strategy different from the injected strategy. Right: standard deviation of off-strategy counts across groups. Lower values are better in both panels, indicating more consistent within-strategy rollout grouping. Table 5: Curriculum learning over strategy-instruction removal does not improve ARC: no removal performs best, linear removal is intermediate, and constant removal is weakest. Training Setting Avg. τ Avg. τ2τ^2 Avg. Reasoning Avg. IFBench ArenaHard No removal (ARC default) 29.59 47.00 38.22 31.58 15.99 15.18 Linear removal 27.28 38.21 31.83 34.12 16.33 15.92 Constant removal 26.97 34.97 32.99 33.21 18.71 15.01 Table 5 and Figure 6 show that progressively removing strategy instructions does not improve ARC in our setting. Default ARC achieves the strongest τ/τ2τ/τ^2 tool-use performance and the lowest off-strategy mean and variance, indicating more stable within-strategy rollout grouping. Linear removal provides a partial trade-off by modestly improving some reasoning metrics at the cost of agentic performance, while constant removal performs worst overall. These results suggest that strategy instructions are most effective as a persistent training-time variance-control mechanism rather than a signal to be annealed away. This is consistent with ARC’s mechanism: strategy instructions are most useful as a training-time variance-control device, and annealing them away too early weakens within-strategy comparability instead of improving the final policy. 6.5 Strategy Scalability Analysis We conduct an additional ablation study of strategy scaling in Table 6, which reveals a clear performance trajectory as strategy components are incrementally integrated, moving from the highest to the lowest training data volume. The Avg. shows a consistent upward trend, achieving an impressive 36.1% total improvement. The most significant gains are concentrated in agentic benchmarks, where performance on τ-bench and τ2τ^2-bench surged by 99% and 71%, respectively, as the model transitioned from a single strategy to the full suite. Figure 7: Domain-instruction interaction analysis. Inference-time strategy hints show no uniformly superior deployment interface. These results highlight not only the effectiveness of the method in leveraging heterogeneous strategies, but also its scalability: as additional strategies are added, the model continues to improve, demonstrating that even minority strategies—such as Alignment Check, which constitutes only 2.5% of the data—contribute meaningfully to overall capabilities. This highlights the potential for accommodating larger strategy sets without performance degradation. Table 6: Strategy scaling ablation. Incremental strategy scaling consistently drives overall performance gains: the full strategy suite performs best, while reasoning tasks peak with three strategies and agentic benchmarks show the most dramatic sensitivity to strategy diversity. Strategy Segment Avg. τ Avg. τ2τ^2 Avg. Reasoning Avg. IFBench ArenaHard Progress Update 21.74 23.62 22.36 34.01 16.33 14.27 Progress Update + Direct Answer 24.91 31.67 32.46 27.79 17.69 14.96 Progress Update + Direct Answer + Clarify First 25.60 31.04 28.97 35.21 16.67 16.13 Full Suite (4) 29.59 47.00 38.22 31.58 15.99 15.18 6.6 Additional Diagnostics on Strategy Instructions We evaluate whether the final ARC policy depends on explicit strategy hints at deployment, or whether the strategy behavior has already been internalized during training. We compare matched, randomized, and removed instructions on the same ARC checkpoint across all τ-bench and τ2τ^2-bench submetrics to isolate the effect of prompt-side cues. Conclusively, Figure 7 shows that inference-time strategy hints do not improve the deployed policy. The baseline with no strategy hint removal managed to achieve the highest overall average (41.73), outperforming both Randomized Instruction (39.58) and Matched Instruction (39.85). Domain-level analysis reveals that matched instructions help on τ Airline but hurt on τ2τ^2 Airline, while all settings struggle on τ2τ^2 Telecom. This domain-dependent variability confirms that inference-time hints lack a reliable deployment benefit. Strategy instructions serve their primary function during training by structuring comparison groups, not as persistent cues that should remain at inference. 7 Conclusion Open-ended agent interaction often admits multiple valid behaviors, yet standard group-based RL compares rollouts within a shared relative-reward pool. We argue this induces a reward fairness issue: advantage estimates become confounded when heterogeneous interaction strategies are normalized together, rather than reflecting intrinsic quality differences. ARC addresses this failure by conditioning the comparison class during rollout construction, ensuring that each group contains a single strategy family. This targeted modification improves the interpretability of relative advantages in multi-strategy regimes, rather than altering the underlying policy optimization machinery. Complementarily, the INTER3 setting makes this issue observable in practice by separating user-facing communication from latent reasoning and tool execution, thereby exposing diverse valid interaction strategies within the same task family. Empirically, ARC is most effective in in-domain agentic benchmarks where multi-strategy interaction is prevalent. Training on INTER3-86K yields higher and more stable rewards, including reduced post-peak degradation under fixed compute budgets. However, improvements are not uniform across all backbones or downstream metrics, reinforcing that ARC should be viewed as a targeted correction for comparison bias rather than a universally dominant optimizer. Our theoretical analysis provides a mechanism-level explanation for these effects. The results in Section 5 are stylized estimator-level characterizations, not end-to-end convergence guarantees. They show how inter-strategy variance in reward-model evaluations can degrade sample efficiency, and how restricting the comparison class can reduce this variance under strategy-dependent reward bias. This supports the empirical findings without claiming to fully characterize RL training dynamics. Overall, the results suggest that progress in open-ended agent learning depends not only on stronger models or reward signals, but also on how learning algorithms construct fair comparison sets when multiple valid behaviors coexist. Limitations Our study has four main limitations. (1) Strategy abstraction. ARC relies on a coarse strategy taxonomy. While useful for conditioning and analysis, real interaction behaviors are more nuanced and context-dependent 25; 29. (2) Domain scope. Our strongest results are in open-ended tool-usage settings. Whether ARC generalizes to other domains remains to be tested. (3) Data and annotation dependence. The method depends on normalized interaction traces and strategy labels, which may reflect annotation bias or deployment-specific patterns. (4) Theoretical scope. Variance and sample-efficiency analyses clarify one optimization mechanism but do not constitute a full convergence theory for open-ended agent RL. References Ahmadian et al. (2024) A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, et al. Back to basics: revisiting reinforce-style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740. Cited by: Appendix B, §1, §2, §5.2. AI (2024) G. AI Glaive: function calling datasets for llms. Note: https://glaive.ai Cited by: §4.2. Balunović et al. (2025) M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev MathArena: evaluating LLMs on uncontaminated math competitions. arXiv preprint arXiv:2505.23281. Cited by: §I.2, §I.2, §6.1. Barres et al. (2025) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan τ2τ^2-Bench: evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982. Cited by: §I.2, §1, §1, §6.1. Bengio et al. (2009) Y. Bengio, J. Louradour, R. Collobert, and J. Weston Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, p. 41–48. Cited by: §6.4. Chen et al. (2024a) L. Chen, C. Zhu, D. Soselia, J. Chen, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro ODIN: disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319. Cited by: §1. Chen et al. (2025) M. Chen, C. Dai, X. Dong, C. Feng, K. Fu, J. Li, Z. Peng, Y. Tong, J. Zhang, and H. Zhu DingTalk DeepResearch: a unified multi agent framework for adaptive intelligence in enterprise environments. arXiv preprint arXiv:2510.24760. External Links: Link Cited by: Appendix B. Chen et al. (2024b) M. Chen, H. Sun, T. Li, F. Yang, H. Liang, K. Lu, B. Cui, W. Zhang, Z. Zhou, and W. Chen Facilitating multi-turn function calling for LLMs via compositional instruction tuning. arXiv preprint arXiv:2410.12952. Note: Accepted to ICLR 2025 Cited by: §4.2. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: Appendix B. Du et al. (2024a) L. Du, Y. Li, X. Wang, Z. Wang, Z. Zhang, Z. Xu, X. Chen, et al. APIGen: automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518. Cited by: Appendix B, §4.2. Du et al. (2024b) Y. Du, F. Fan, and D. Pan AnyTool: self-reflective, hierarchical agents for large-scale api calls. arXiv preprint arXiv:2402.04253. Cited by: Appendix B, §1, §2. Duan et al. (2026) Z. Duan, G. Rong, Z. Li, B. Chen, M. Zhou, and D. Guo Mitigating reward hacking in rlhf via bayesian non-negative reward modeling. arXiv preprint arXiv:2602.10623. Cited by: §1. Ethayarajh et al. (2024) K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela KTO: model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. Cited by: Appendix B, §2. Florensa et al. (2018) C. Florensa, D. Held, X. Geng, and P. Abbeel Automatic goal generation for reinforcement learning agents. In Proceedings of the 35th International Conference on Machine Learning, p. 1515–1528. Cited by: §6.4. Florensa et al. (2017) C. Florensa, D. Held, M. Wulfmeier, and P. Abbeel Reverse curriculum generation for reinforcement learning. In Proceedings of the 1st Conference on Robot Learning, p. 482–495. Cited by: §6.4. Greensmith et al. (2004) E. Greensmith, P. L. Bartlett, and J. Baxter Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research 5 (Nov), p. 1471–1530. Cited by: §5.2, §5.4. Hamilton (2024) S. Hamilton Detecting mode collapse in language models via narration. arXiv preprint arXiv:2402.04477. Cited by: Appendix B. Jia et al. (2024) H. Jia, T. Bu, Y. Wang, Z. Zhang, et al. Scaf-grpo: scaffolded group relative policy optimization for enhancing llm reasoning. arXiv preprint arXiv:2510.19807. Cited by: Appendix B, §1, §2, §5.2. Jin et al. (2025) R. Jin, P. Gao, Y. Ren, Z. Han, T. Zhang, W. Huang, W. Liu, J. Luan, and D. Xiong Revisiting entropy in reinforcement learning for large reasoning models. arXiv preprint arXiv:2511.05993. Cited by: §5.3. Li et al. (2024a) T. Li et al. From crowdsourced data to high-quality benchmarks: arena-hard and beyond. arXiv preprint arXiv:2405.20239. Cited by: Appendix B, §I.2, §6.1. Li et al. (2024b) Z. Li et al. Can we trust ai agents? an experimental study towards trustworthy llm-based multi-agent systems for ai-driven software engineering. arXiv preprint arXiv:2411.08881. Cited by: Appendix B. Liang et al. (2025) A. Liang, J. Berant, A. Fisch, A. Goyal, K. Krishna, and J. Eisenstein Plantain: plan-answer interleaved reasoning. arXiv preprint arXiv:2512.03176. Cited by: Appendix B, §1, §2. Liao et al. (2025) B. Liao, Z. Zhang, L. Frison, and C. Monz Self-hinting language models enhance reinforcement learning. arXiv preprint arXiv:2602.03143. Cited by: Appendix B, §1, §2, §5.2. Liao and Vaughan (2023) Q. V. Liao and J. W. Vaughan AI transparency in the age of llms: a human-centered research roadmap. arXiv preprint arXiv:2306.01941. Cited by: Appendix B, §1. Lin et al. (2023) Z. Lin, Z. Wang, Y. Tong, Y. Wang, Y. Guo, Y. Wang, and J. Shang ToxicChat: unveiling hidden challenges of toxicity detection in real-world user-AI conversation. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 4694–4702. External Links: Document, Link Cited by: Limitations. Liu et al. (2024) F. Liu et al. ToolACE: winning the solutions of tool learning. arXiv preprint arXiv:2409.00920. Cited by: Appendix B, §4.2. Liu et al. (2025) Z. Liu et al. Understanding r1-zero-like training: a critical perspective. arXiv preprint arXiv:2503.20783. Cited by: Appendix B, §1, §2, §5.2. Matiisen et al. (2017) T. Matiisen, A. Oliver, T. Cohen, and J. Schulman Teacher-student curriculum learning. arXiv preprint arXiv:1707.00183. Cited by: §6.4. Pan et al. (2025) L. Pan, Y. Tong, X. Zhang, X. Zhang, J. Zhou, and Z. Chu Understanding and mitigating overrefusal in LLMs from an unveiling perspective of safety decision boundary. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 21057–21075. External Links: Document, Link Cited by: Limitations. Pan et al. (2026) L. Pan, H. Yang, H. Li, Y. Lu, Y. Tong, Y. Wang, S. Wang, Z. Chu, L. Shen, Y. Lu, and H. Wang Optimal transport for LLM reward modeling from noisy preference. arXiv preprint arXiv:2605.06036. External Links: Link Cited by: Appendix B. Patil et al. (2023) S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez Gorilla: large language model connected with massive apis. In Advances in Neural Information Processing Systems, Cited by: Appendix B. Pyatkin et al. (2025) V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi Generalizing verifiable instruction following. arXiv preprint arXiv:2507.02833. Note: Accepted to NeurIPS 2025, Datasets & Benchmarks Cited by: §I.2, §6.1. Qian et al. (2025) C. Qian, E. C. Acikgoz, Q. He, H. Wang, X. Chen, D. Hakkani-Tur, G. Tur, and H. Ji ToolRL: reward is all tool learning needs. arXiv preprint arXiv:2504.13958. Cited by: §C.1. Qin et al. (2023) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. ToolLLM: facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. Cited by: Appendix B, §2. Qu et al. (2024) C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J. Wen Tool learning with large language models: a survey. arXiv preprint arXiv:2405.17935. Cited by: §2. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4.2, §4.3, §4.5.1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. arXiv preprint arXiv:2305.18290. Cited by: Appendix B, §2. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022. Cited by: §I.2, §6.1. Ross et al. (2025) H. Ross, A. S. Mahabaleshwarkar, and Y. Suhara When2Call: when (not) to call tools. arXiv preprint arXiv:2504.18851. Note: Accepted to NAACL 2025 Cited by: §4.2. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hamborg, L. Zettlemoyer, N. Cancedda, and M. Lewis Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, Vol. 36, p. 68539–68551. Cited by: Appendix B, §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06347, Cited by: Appendix B, §1, §2, §6.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, et al. DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Appendix B, §1, §2, §6.1, §6.1. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366. Cited by: Appendix B. Tao et al. (2025) L. Tao, I. Kulikov, S. Saha, T. Wang, J. Xu, Y. Li, J. E. Weston, and P. Yu Hybrid reinforcement: when reward is sparse, it’s better to be dense. arXiv preprint arXiv:2510.07242. Cited by: §C.1. Team (2025) Q. Team Qwen3 technical report. arXiv preprint. Cited by: §I.2, §I.2, §4.5.1, §6.1. TeichAI (2025) TeichAI Claude 4.5 opus high reasoning 250x dataset. Note: https://huggingface.co/datasets/TeichAI/claude-4.5-opus-high-reasoning-250xDistilled from Claude Opus 4.5 Cited by: §4.2. Tong et al. (2026a) Y. Tong, X. Feng, L. Xue, J. Li, X. Zhang, and J. Yang How to train a real-world silicon concierge? internalizing complex business workflow to only OneModel. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), p. 1303–1316. External Links: Document, Link Cited by: Appendix B. Tong et al. (2024a) Y. Tong, D. Li, S. Wang, Y. Wang, F. Teng, and J. Shang Can LLMs learn from previous mistakes? investigating LLMs’ errors to boost for reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3065–3080. External Links: Document, Link Cited by: Appendix B. Tong et al. (2024b) Y. Tong, S. Wang, D. Li, Y. Wang, S. Han, Z. Lin, C. Huang, J. Huang, and J. Shang Optimizing language model’s reasoning abilities with weak supervision. arXiv preprint arXiv:2405.04086. External Links: Link Cited by: §2. Tong et al. (2023) Y. Tong, Y. Wang, D. Li, S. Wang, Z. Lin, S. Han, and J. Shang Eliminating reasoning via inferring with planning: a new framework to guide LLMs’ non-linear thinking. arXiv preprint arXiv:2310.12342. External Links: Link Cited by: Appendix B. Tong et al. (2026b) Y. Tong, Z. Zhang, Z. Liu, Tangzeyun, Z. Peng, K. Fu, M. Song, H. Zhang, J. Zhang, H. Zhu, and J. Li Reinforcement learning with missing context to mitigate reward hacking from training only on golden answers. Note: OpenReview preprint External Links: Link Cited by: §1. Trivedi et al. (2022) H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal MuSiQue: multihop questions via single-hop question composition. In Transactions of the Association for Computational Linguistics, Vol. 10, p. 539–554. Cited by: §4.2. Wang et al. (2023) L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K. Lee, and E. Lim Plan-and-solve prompting: improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091. Cited by: Appendix B. Wang et al. (2025) S. Wang, Y. Tong, H. Zhang, D. Li, X. Zhang, and T. Chen BPO: towards balanced preference optimization between knowledge breadth and depth in alignment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 8811–8826. External Links: Document, Link Cited by: Appendix B. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 24824–24837. Cited by: Appendix B. Xie et al. (2024) C. Xie, Y. Huang, C. Zhang, D. Yu, X. Chen, B. Y. Lin, B. Li, B. Ghazi, and R. Kumar On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123. Cited by: §4.2. Xie et al. (2025) R. Xie, D. Qiu, D. Gopinath, D. Lin, Y. Sun, C. Wang, S. Potdar, and B. Dhingra Interleaved reasoning for large language models via reinforcement learning. arXiv preprint arXiv:2505.19640. Cited by: Appendix B, §1, §2. Xu et al. (2026) Z. Xu, Y. Tong, X. Zhang, J. Zhou, and X. Wang Understanding conflicts in multi-objective alignment through reward consistency. In Findings of the Association for Computational Linguistics: ACL 2026, p. 5450–5472. External Links: Document, Link Cited by: Appendix B. Yan et al. (2025) J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945. Cited by: Appendix B, §1, §2. Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ-Bench: a benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045. Cited by: §I.2, §1, §1, §4.2, §6.1. Yao et al. (2023a) S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan Tree of thoughts: deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. Cited by: Appendix B. Yao et al. (2023b) S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations, Cited by: Appendix B, §1, §1, §2. Yu et al. (2025) Q. Yu et al. DAPO: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: Appendix B, §1, §2, §6.1. Yuan et al. (2023) L. Yuan, Y. Chen, X. Wang, Y. R. Fung, H. Peng, and H. Ji CRAFT: customizing llms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309.17428. Cited by: Appendix B. Yuan and Ying (2022) R. Yuan and L. Ying Finite-time analysis of stochastic policy gradient methods. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, p. 1070–1096. Cited by: §5.4. Zhan et al. (2025) R. Zhan, Y. Li, Z. Wang, X. Qu, D. Liu, J. Shao, D. F. Wong, and Y. Cheng ExGRPO: learning to reason from experience. arXiv preprint arXiv:2510.02245. Cited by: Appendix B, §1, §2. Zhang et al. (2024a) X. Zhang et al. Controllable text generation for large language models: a survey. arXiv preprint arXiv:2408.12599. Cited by: Appendix B, §1. Zhang et al. (2024b) Y. Zhang et al. A survey on human-ai collaboration with large foundation models. arXiv preprint arXiv:2403.04931. Cited by: Appendix B, §1. Zhang et al. (2024c) Y. Zhang et al. XLAM: a large action model dataset for autonomous agents. arXiv preprint arXiv:2406.06152. Cited by: §4.2. Zhou et al. (2023) A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y. Wang Language agent tree search unifies reasoning, acting, and planning in language models. arXiv preprint arXiv:2310.04406. Cited by: Appendix B. Appendix A Proofs and Theoretical Analysis This appendix provides proofs for the stylized variance claims in Section 5. We distinguish between the target strategy s⋆s , which is assigned to a training prompt before ARC rollout generation, and the realized strategy si=f(yi)s_i=f(y_i) exhibited by a sampled response yiy_i. Pry∼πθ(⋅∣x)[f(y)=s]. _y _θ(· x)[f(y)=s]. (2) Under ARC, a target strategy s⋆s is assigned before rollout generation and appended to the prompt. Responses are then sampled as yi∼i.i.d.πθ(⋅∣x,s⋆).y_i i.i.d. _θ(· x,s ). (3) The realized strategy remains si=f(yi)s_i=f(y_i) and may differ from s⋆s when strategy adherence is imperfect. A.1 Proof of Theorem 5.1 (Variance Amplification) Proof. Decompose ri=μsi+ϵir_i= _s_i+ _i where μsi _s_i is the mean reward for realized strategy sis_i (capturing RM bias) and ϵi _i is i.i.d. zero-mean noise with variance σintra2σ^2_intra, independent of the strategy-dependent mean term. Since yi∼i.i.d.πθ(⋅∣x)y_i i.i.d. _θ(· x) and si=f(yi)s_i=f(y_i), the realized strategies s1,…,sNs_1,…,s_N are i.i.d. under the induced distribution Pθ(⋅∣x)P_θ(· x), so μs1,…,μsN _s_1,…, _s_N are i.i.d. with variance σinter2σ^2_inter. The advantage is A^i=ri−r¯=(μsi−μ¯)+(ϵi−ϵ¯) A_i=r_i- r=( _s_i- μ)+( _i- ε), where μ¯=1N∑jμsj μ= 1N _j _s_j and ϵ¯=1N∑jϵj ε= 1N _j _j. Since strategy and noise are independent: Var[A^i]=Var[μsi−μ¯]+Var[ϵi−ϵ¯] [ A_i]=Var[ _s_i- μ]+Var[ _i- ε] (4) For any i.i.d. sequence Z1,…,ZNZ_1,…,Z_N with variance σ2σ^2: Var[Zi−Z¯]=(1−1N)2σ2+N−1N2σ2=σ2(1−1N) [Z_i- Z]= (1- 1N )^2σ^2+ N-1N^2σ^2=σ^2 (1- 1N ) (5) Applying this to both terms yields Var[A^i]=(σinter2+σintra2)(1−1/N)Var[ A_i]=(σ^2_inter+σ^2_intra)(1-1/N). ∎ A.2 Proof of Theorem 5.2 (Variance Reduction via Conditioning) Proof. ARC conditions generation on the target strategy s⋆s , and under perfect compliance, f(yi)=s⋆f(y_i)=s , so μsi=μs∗ _s_i= _s^* for all i. The advantage simplifies to: A^i=ϵi−ϵ¯ A_i= _i- ε (6) Since ϵi _i are i.i.d. with variance σintra2σ^2_intra: Var[A^i|s∗]=σintra2(1−1N) [ A_i|s^*]=σ^2_intra (1- 1N ) (7) The between-strategy mean-shift term is absent from this centered-advantage variance decomposition. Var[A^i|s∗]= [ A_i|s^*]= (σintra2(x,s⋆)CLOSE (σ^2_intra(x,s ) (8) +VarS∼Pθ(⋅∣x,s⋆)[μS(x)])(1−1N). +Var_S Pθ(· x,s )[ _S(x)] ) (1- 1N ). With imperfect compliance, the ARC variance additionally contains the residual between-strategy term, so the predicted variance reduction is correspondingly attenuated. ∎ A.3 Derivation for Theorem 5.3 (Idealized Sample-Complexity Comparison) Theorem A.1 (Gradient Variance Bound). Let gARCg_ARC and gstdg_std denote policy gradients under strategy-conditioned and standard sampling with equal group size N. For analytic tractability, assume: (i) the relevant advantage-score mixed moments approximately factorize conditional on x; (i) the score-function second moments F=[‖∇i‖2]F=E[\| _i\|^2] and F′=[⟨∇i,∇j⟩]F =E[ _i, _j ] (i≠ji≠ j) are comparable up to constants across the two sampling schemes; and (i) ARC satisfies the perfect-compliance idealization of Theorem 5.2. Then: [‖gARC‖2][‖gstd‖2]≈σintra2σintra2+σinter2 E[\|g_ARC\|^2]E[\|g_std\|^2]≈ σ^2_intraσ^2_intra+σ^2_inter (9) Proof. Write g=1N∑i=1NA^i∇ig= 1N _i=1^N A_i _i where ∇i=∇θlogπθ(yi|x) _i= _θ _θ(y_i|x). Under assumption (i): [‖g‖2]=1N2[N[A^i2]F+N(N−1)[A^iA^j]F′] [\|g\|^2]= 1N^2 [NE[ A_i^2]\,F+N(N-1)\,E[ A_i A_j]\,F ] (10) For i.i.d. rewards rir_i with total variance σr2σ^2_r, the centered advantages satisfy: [A^i2] [ A_i^2] =σr2(1−1N) =σ^2_r (1- 1N ) (11) [A^iA^j] [ A_i A_j] =−σr2N(i≠j) =- σ^2_rN (i≠ j) (12) The latter follows from Cov[ri−r¯,rj−r¯]=−σr2/NCov[r_i- r,\,r_j- r]=-σ^2_r/N. Substituting: [‖g‖2]=σr2(N−1)N2(F−F′)E[\|g\|^2]= σ^2_r(N-1)N^2(F-F ) (13) Since σr2=σintra2+σinter2σ^2_r=σ^2_intra+σ^2_inter under standard sampling and σr2=σintra2σ^2_r=σ^2_intra under ARC (Theorems 5.1–5.2), assumption (i) yields the stated approximation. ∎ Derivation for Theorem 5.3. This argument should be read as a stylized variance-to-sample-efficiency translation rather than a full convergence proof. Under a sub-Gaussian gradient-estimation assumption, the number of gradient samples required to achieve ‖g^−∇J(θ)‖≤ε\| g-∇ J(θ)\|≤ with probability ≥1−δ≥ 1-δ scales as O(σg2log(1/δ)/ε2)O(σ^2_g (1/δ)/ ^2), where σg2σ^2_g is the per-sample gradient variance. From Theorem A.1, σg,ARC2/σg,std2≈σintra2/(σintra2+σinter2)σ^2_g,ARC/σ^2_g,std≈σ^2_intra/(σ^2_intra+σ^2_inter), and the same approximation carries over to this idealized sample-efficiency comparison. ∎ A.4 Effective Variance Analysis for Curriculum Learning For the curriculum learning schedule studied in Section 6.4, the expected advantage variance across the batch at training step t is a mixture of conditioned and unconditioned groups: Vareff(t)= _eff(t)= (1−pd(t))σintra2(1−1N) (1-p_d(t) )\,σ^2_intra\! (1- 1N ) (14) +pd(t)(σintra2+σinter2)(1−1N). +p_d(t)\, (σ^2_intra+σ^2_inter )\! (1- 1N ). Since the second term exceeds the first by σinter2(1−1/N)σ^2_inter(1-1/N), Vareff(t)Var_eff(t) increases monotonically from ≈σintra2(1−1/N)≈σ^2_intra(1-1/N) to ≈(σintra2+σinter2)(1−1/N)≈(σ^2_intra+σ^2_inter)(1-1/N). This monotonic increase gives the curriculum learning schedule a variance-based progression: the model trains under progressively noisier optimization landscapes as it becomes more capable. A.5 Information-Theoretic Perspective Assuming strategy s is determined by the response y (i.e., s=f(y)s=f(y) for some deterministic function), the chain rule of mutual information gives: I(r;y|x)=I(r;y|x,s)+I(r;s|x) I(r;y|x)=I(r;y|x,s)+I(r;s|x) (15) The term I(r;s|x)I(r;s|x) captures strategy-linked reward dependence that is orthogonal to within-strategy quality assessment. Under the idealized perfect-compliance setting, conditioning on the target strategy removes between-strategy variation from within-group comparisons, emphasizing the within-strategy reward signal I(r;y∣x,s)I(r;y x,s). Appendix B Extended Related Work Tool-Augmented and Interactive Agents. Tool-augmented language models have been studied through self-supervised tool learning 40, large-scale API integration 34, 31, synthetic function-calling corpora 10, 26, and increasingly sophisticated agent architectures 11, 64, 7, 47. A parallel line of work studies interaction structure, including ReAct-style reasoning-action interleaving 62, structured planning 55, 61, 53, 50, and reflective or search-based agents 43, 70, 48. Our interest is not only in adding tools or planning steps, but in exposing interaction as a first-class, user-visible channel while execution is still ongoing. Interleaved Reasoning and User Experience. Recent interleaving methods improve perceived responsiveness by alternating internal reasoning with partial textual output 22, 57. These works mainly address the sequencing of thought and answer tokens. INTER3 targets a different bottleneck: long-running external actions. When a tool call itself is the latency source, making the communication channel independent from the execution channel becomes the key design move. This connects to broader human-AI interaction work on transparency, feedback, and controllability 68, 24, 21, 67, 20, but in an agent context where the cost of opaque waiting is especially high. RLHF Estimators and Multi-Behavior Collapse. Our RL analysis sits within the broader literature on policy-gradient and RLHF optimization 41, 1, 37, 13, 54, 58, 30, especially group-relative estimators such as GRPO 42 and refinements for entropy preservation or bias correction 63, 27. These methods have proven effective for reasoning-heavy tasks 9, but open-ended multi-behavior settings raise a distinct collapse risk 17: the policy can converge to whichever behavior the reward favors. Our contribution is to isolate one concrete statistical mechanism for that collapse—cross-strategy contamination of relative advantages—and study a conditioning-based remedy. External Guidance in RL. Methods such as SAGE 23, Scaf-GRPO 18, LUFFY 59, and ExGRPO 66 show that auxiliary guidance can materially improve RL training by mitigating sparse rewards, structuring exploration, incorporating demonstrations, or reusing successful trajectories. ARC is adjacent in form but different in purpose. The strategy instruction is not introduced to reveal the answer, densify reward, or bias the policy toward a target trajectory. It is introduced to constrain which rollouts are compared to one another, so that relative advantage estimation is performed inside a behaviorally coherent comparison class. Appendix C Reward Details C.1 Final Reward Used in Main Experiments The specific reward used in our full INTER3 instantiation is not ARC’s core mechanism; it is a setting-specific design choice for stabilizing execution-grounded training. Following recent reward-design work that combines verifiable structural signals with denser model-based feedback 33, 44, we use a compact three-term reward. The format reward Rfmt∈0,1R_fmt∈\0,1\ checks for at least one well-formed <tool_call> or <answer> block with properly matched, non-nested tags. The tool reward Rtool∈−1,0,1R_tool∈\-1,0,1\ evaluates exact tool-call correctness: the tool identifier, argument key set, and argument values must all match the reference for Rtool=1R_tool=1. The answer reward Rans∈0,0.50,1R_ans∈\0,0.50,1\ is an LLM-judge score on the extracted <answer> content. To couple semantics to correct execution, we set Rans=Eval(y,y∗)R_ans=Eval(y,y^*) only when Rtool=1R_tool=1, and Rans=0R_ans=0 otherwise. The final reward is simply R=Rfmt+Rtool+RansR=R_fmt+R_tool+R_ans. If both prediction and reference contain no answer spans, we set Rans=1R_ans=1 to reflect structural agreement; if evaluation fails, we return a neutral fallback score. C.2 Detailed Reward Ablation A central challenge in training INTER3 agents is designing rewards that evaluate multi-channel outputs—internal reasoning, tool calls, and user-facing <answer> spans—without baking in strategy-dependent bias. We study two reward-side choices: the reward construction itself, and the reward model used to score semantic answer quality. Table 7: Reward-construction ablation. All variants include the format reward. Exact tool matching improves tool use, and the final gated design gives the strongest tool-centric operating point. Unless explicitly varied, all main RL baselines and ARC variants in the paper use the final row. Reward Comparisons Avg. τ Avg. τ2τ^2 Avg. Reasoning Avg. IFBench ArenaHard Partial tool + answer judge 24.90 30.92 29.16 33.19 14.97 16.24 Exact tool + answer judge 27.89 37.31 31.94 36.24 18.03 15.94 Penalized tool + gated judge 27.98 43.93 36.49 29.60 16.33 13.57 The reward constructions in Table 7 differ along two axes: (i) tool-execution strictness, and (i) coupling between execution and semantic reward. All variants include a shared format reward enforcing structural validity. Moving from partial to exact tool matching strengthens the execution-grounded signal, and the final gated design amplifies this effect, increasing τ Avg. by about 42% over the weakest variant. This comes with reduced performance on broader reasoning-style evaluations, while leaving overall average performance largely unchanged. The results reflect our objective: to prioritize rewards that favor answers grounded in correct tool execution rather than signals agnostic to whether the answer is causally supported by the executed actions. Table 8: Reward-model ablation for the same Qwen3-8B policy under the final reward construction used throughout the main experiments. Varying only the external judge shows that the 235B judge provides the strongest tool-use operating point. Judge Model Avg. τ Avg. τ2τ^2 Avg. Reasoning Avg. IFBench ArenaHard Qwen3-235B-Instruct 27.98 43.93 36.49 29.60 16.33 13.57 Qwen3.5-27B 21.67 25.69 22.55 27.97 19.39 12.77 Qwen3.5-122B 25.02 35.22 30.70 29.27 16.67 13.27 Table 8 fixes the policy at Qwen3-8B and varies only the external reward judge. Under this controlled comparison, the 235B judge gives the strongest τ/τ2τ/τ^2 performance and the best overall balance for execution-grounded training, while smaller judges recover only isolated gains on auxiliary metrics. These findings led us to select the 235B judge for our main experiments, as it provides the most consistent signal for tool-execution verification. Tool matching. Partial tool assigns credit to approximately correct tool calls, providing dense but noisy supervision. Replacing this with exact tool matching yields stricter credit assignment, improving reliability of the training signal. Negative penalties. The final variant introduces a negative penalty for incorrect tool execution, explicitly discouraging spurious or malformed calls and sharpening optimization toward valid trajectories. Semantic reward coupling. All variants use an LLM-based answer judge for user-visible responses. In the first two variants, this reward is applied unconditionally, allowing fluent but unsupported answers to receive credit. The final variant applies a gated answer judge, where semantic reward is issued only when exact tool execution is correct, coupling answer quality with execution validity. Appendix D Entropy Analysis To separate ARC’s core mechanism from setting-specific stabilizers, Table 9 fixes the reward family and varies only two ingredients: strategy-conditioned grouping and entropy regularization. The first block asks which component drives the main gain over GRPO; the second block studies entropy sensitivity once grouping is enabled. Table 9: Ablation study on ARC: Training settings and entropy hyperparameters. Strategy-conditioned grouping provides the largest single gain over base GRPO. While entropy regularization alone slightly decreases performance, combining it with strategy-conditioned grouping further improves the overall operating point. The best entropy value (0.001) yields the highest average score. Configuration Avg. τ Avg. τ2τ^2 Avg. Reasoning Avg. IFBench ArenaHard Training Ingredients GRPO 25.43 35.81 31.72 27.25 18.71 13.66 GRPO + Entropy 24.23 33.53 29.42 27.66 16.67 13.85 GRPO + Strategy-Conditioned Grouping 27.80 43.93 35.55 29.60 16.33 13.57 GRPO + Strategy-Conditioned Grouping + Entropy 29.59 47.00 38.22 31.58 15.99 15.18 Entropy Sensitivity (with Strategy-Conditioned Grouping) Entropy=0.01 (Reward Collapse) — — — — — — Entropy=0.001 29.59 47.00 38.22 31.58 15.99 15.18 Entropy=0.0001 27.48 40.95 34.11 32.32 14.29 15.73 Table 9 supports the paper’s main causal story. Holding the reward design fixed, strategy-conditioned grouping is the primary source of improvement over base GRPO, whereas entropy regularization alone does not explain the gain and can even weaken performance. Entropy becomes useful only after the comparison class has been cleaned up by grouping, where it acts as a stabilizer for multi-channel generation rather than as the main mechanism. The entropy sweep further shows that this effect is sensitive to scale: too much entropy leads to reward collapse, while a moderate value of 0.0010.001 gives the best overall operating point. Taken together, these results suggest that ARC helps mainly by changing how rollouts are compared, with entropy regularization serving as a secondary component that improves the stability of that mechanism in the full INTER3 recipe. Appendix E Annotation Guidelines for INTER3-86K This section provides the detailed annotation guidelines used by both models and human annotators for strategy assignment. Progress Update: Proactively provide progress updates to the user when simultaneously executing a tool call or multi-step tasks. Notify the user of the current progress after completing each tool call/sub-task. Let the user know the task is in progress to reduce waiting anxiety. Direct Answer: Answer directly without invoking any tools if the information is already known or contextually available. Be concise and clear; avoid verbosity. Clarify First: If the user’s request lacks sufficient information, ask for clarification first. Do not guess the user’s intent; ask directly. Execute tool calls only after the user confirms. Suitable for vague, ambiguous, or incomplete requests. Alignment Check: Restate the user’s requirements first to confirm your understanding is correct. Execute only after the user confirms. Avoid making mistakes due to misunderstanding. Suitable for easily misunderstood, important, or irreversible operations. Appendix F Mechanism-Level Analysis and Label-Noise Robustness To complement the idealized theoretical analysis, we perform rollout-level analyses of empirical reward unfairness, realized inter- and intra-strategy reward variance, strategy adherence, and sensitivity to strategy-label noise. Realized inter- and intra-strategy reward variance. On actual rollouts, we estimate the realized between-strategy and within-strategy reward variances. As shown in Table 10, the estimated inter-/intra-strategy variance ratio decreases from 0.417 under GRPO to 0.074 under ARC, corresponding to an approximately 82% reduction. This is consistent with the proposed variance-reduction mechanism. However, the confidence intervals are wide because relatively few prompt groups contain multiple realized strategies, so we interpret this result as mechanism-level evidence rather than definitive empirical validation of the stylized theoretical analysis. Table 10: Estimated realized inter- and intra-strategy reward variance on rollout data. Ratio denotes σ^inter2/σ^intra2 σ^2_inter/ σ^2_intra, with 95% confidence intervals. Method σ^inter2 σ^2_inter σ^intra2 σ^2_intra Ratio (95% CI) GRPO 0.0555 0.1331 0.417 [0.002, 3.015] ARC 0.0030 0.0401 0.074 [0.002, 0.410] Sensitivity to strategy-label noise. We additionally test the sensitivity of ARC to imperfect strategy labels by corrupting 50% of the strategy instructions during training. For each corrupted example, the original strategy instruction is replaced with a uniformly sampled alternative strategy. This experiment uses one training seed, and the reported standard deviations are computed over three independent evaluations. As shown in Table 11, ARC remains above the GRPO baseline under substantial label corruption, with a tool-use average of 35.01 compared with 33.35 for GRPO. However, performance is substantially lower than clean ARC at 41.73, indicating that ARC benefits from accurate strategy assignments and degrades under noisy training labels. Table 11: Sensitivity of ARC to strategy-label corruption. The corrupted setting replaces 50% of strategy instructions with a uniformly sampled alternative strategy during training. Results are based on a single training run (seed 1); values after ± denote standard deviations over three independent evaluation runs. Method Tool-use Avg. τ-Airline τ-Retail τ2τ^2-Airline τ2τ^2-Retail τ2τ^2-Telecom GRPO 33.35 31.33±4.6231.33± 4.62 40.29±4.9440.29± 4.94 36.67±6.4336.67± 6.43 40.64±5.9740.64± 5.97 17.84±2.8217.84± 2.82 GRPO + ARC 41.73 44.00±3.0644.00± 3.06 50.00±1.3350.00± 1.33 48.00±6.1148.00± 6.11 45.61±0.8645.61± 0.86 21.05±0.9121.05± 0.91 GRPO + ARC, 50% corrupted 35.01 34.00±3.4634.00± 3.46 42.90±1.3342.90± 1.33 37.16±0.7337.16± 0.73 42.11±3.1642.11± 3.16 18.87±1.0718.87± 1.07 Appendix G Interaction Strategy Examples G.1 Progress Update Strategy ⬇ [user] Help me analyze Acme Corp’s financial status. [assistant] The user wants a full financial analysis. I’l start by fetching the income statement. <tool_call> "name": "get_income_statement", "arguments": "company": "Acme Corp" </tool_call> <answer>I’m retrieving Acme Corp’s financial statements - step 1 of 3... </answer> [tool] get_income_statement -> "revenue": "$5.2B", "net_income": "$430M" [assistant] Income statement retrieved. Now fetching the balance sheet for ratio analysis. <tool_call> "name": "get_balance_sheet", "arguments": "company": "Acme Corp" </tool_call> <answer>Income statement retrieved. Now analyzing the balance sheet - step 2 of 3...</answer> G.2 Direct Answer Strategy ⬇ [user] What is the capital of China? [assistant] <answer>The capital of China is Beijing.</answer> G.3 Clarify First Strategy ⬇ [user] Book a meeting room for me. [assistant] Request lacks required information: time and capacity. <answer>I’d be happy to book a meeting room. Could you tell me: (1) What date and time? (2) How many people will attend?</answer> [user] Tomorrow at 10 AM, 8 people. [assistant] Details confirmed. Proceeding with booking. <tool_call>"name": "book_meeting_room", "arguments": "date": "tomorrow", "time": "10:00", "capacity": 8</tool_call> <answer>Proceeding with booking a room for tomorrow at 10 AM suitable for 8 people...</answer> [tool] book_meeting_room -> "status": "confirmed", "room": "Room 3B" [assistant] <answer>Booked! Room 3B for tomorrow at 10:00 AM, capacity 8.</answer> G.4 Alignment Check Strategy ⬇ [user] Send last quarter’s sales data to Manager Wang. [assistant] This is a send operation --- I should confirm the exact scope and recipient before executing. <answer>Just to confirm: you’d like me to send the Q1 2024 (January-March) sales data report to Manager Wang at wang@company.com. Is that correct?</answer> [user] Yes, that’s right. [assistant] Intent confirmed. Executing send operation. <tool_call>"name": "send_email", "arguments": "to": "wang@company.com", "subject": "Q1 2024 Sales Data", "attachment": "q1_sales.pdf"</tool_call> [tool] send_email -> "status": "sent" [assistant] <answer>Sent! Q1 2024 sales report delivered to Manager Wang.</answer> Appendix H System Prompt The system prompt used for training and evaluation: System Prompt Configuration System Role You are an advanced AI assistant capable of streamingly interleaved interactions, reasoning, and tool executions. Do not wait for the entire process to finish. You can provide partial updates in <answer> while continuing to reason and call tools. Output Requirements Your output must be structured into functional segments. Texts that are NOT wrapped in <answer> tags are treated as internal reasoning and are hidden from the user. Ensure all XML-style tags (<tool_call>, <answer>) are strictly paired and properly closed. Tag Definitions • Internal Reasoning: (No tag required) Use plain text outside of any tags for logical reasoning, task decomposition, or analyzing tool outputs. This is hidden from the user. • <answer>: (User Visible) This is the ONLY content displayed to the user. Use this for status updates, partial answers, or the final conclusion. If your response relies on information from the tool outputs, do not expose the raw tool output verbatim. Instead, integrate and summarize the relevant facts naturally in your own words. • <tool_call>: (Internal Only) Use this to call external functions using the provided tools, if any. Appendix I Evaluation Setup and Configuration This appendix describes the standardized evaluation configuration used across all benchmarks reported in Table 3. I.1 Inference Configuration All evaluations were conducted using the following controlled parameters: Concurrency. TTFT measurements are collected with concurrency level set to 16 parallel requests on the same GPU (H200). This concurrency level is maintained consistently across all τ-bench evaluations to ensure fair latency comparisons. Interleaved Mode. When interleaved mode is enabled, we use the same structured interface as in training: <answer> for user-visible content and <tool_call> for tool calls. This enables fine-grained measurement of time-to-first-token (TTFT) as the model can provide partial answers while continuing reasoning and tool execution. I.2 Benchmark Configurations τ-Bench. Following the official τ-Bench evaluation protocol 60, we evaluate tool-calling ability across the retail and airline domains. The user simulator is powered by the Qwen3-235B-A22B-Instruct model 45. We report the average reward score (success rate) and TTFT. Each evaluation run is repeated 3 times and we report the mean score. τ2τ^2-Bench. Following the official τ2τ^2-Bench evaluation protocol 4, we evaluate on the retail, airline, and telecom domains. The user simulator is powered by the Qwen3-235B-A22B-Instruct model 45. We repeat the evaluation 3 times and we report the mean score. AIME 2026. The American Invitational Mathematics Examination 2026 dataset 3 contains challenging math problems with integer answers in [0, 999]. We extract the model’s answer from notation. Each problem is evaluated 8 times with different random seeds, and accuracy is computed by exact match after normalization. The instruction prompt asks the model to solve the problem and format the final answer. GPQA Diamond. The Google-Proof Q&A (GPQA) Diamond subset 38 contains expert-level science questions with multiple-choice answers. We evaluate the model’s ability to select the correct option (A, B, C, or D). Each question is run 8 times, and we compute the accuracy as the fraction of correct selections. HMMT 2025. The Harvard-MIT Mathematics Tournament February 2025 dataset 3 contains competition-level math problems across multiple rounds. Answers may be integers, fractions, expressions, or other mathematical objects. Each problem is evaluated 8 times. IFBench. Instruction Following Benchmark 32 tests the model’s ability to follow complex, multi-constraint instructions. The benchmark contains test cases with specific formatting, length, and content requirements. Evaluation is performed using the official IFBench scoring script, which checks constraint satisfaction. Arena-Hard. Following the official Arena-Hard evaluation protocol 20, we use the benchmark’s GPT-4-based judge to compare model responses against a baseline (o3-mini-2025-01-31) on 580 challenging questions from diverse domains. We report the win rate percentage as the final score. I.3 TTFT Measurement Protocol Time-to-first-token (TTFT) is measured specifically for tool-calling benchmarks (τ-bench variants) where latency is most critical. The measurement protocol: • Definition: TTFT is the time from sending the request to receiving the first user-visible token of the model’s response. • Aggregation: For each row in Table 3, we report the mean TTFT over all available τ-bench domain measurements for that configuration. • Conditions: All TTFT measurements use concurrency=16 on H200 GPUs, with no other concurrent workloads. • Scope: TTFT is not reported for reasoning benchmarks (AIME, GPQA, HMMT) or instruction-following/alignment benchmarks (IFBench, Arena-Hard) as these are primarily accuracy-focused rather than latency-sensitive. I.4 Evaluation Uncertainty and Training-Seed Robustness We additionally evaluate the statistical variability of ARC under both repeated evaluation and independent training seeds. These analyses separate variability arising from stochastic evaluation from variability arising from optimization randomness during training. Table 12: Evaluation uncertainty on the tool-use benchmarks. Results are reported as mean ± standard deviation over N=3N=3 independent evaluation runs using the same trained checkpoint. Tool-use Avg. is the mean over the five displayed τ-bench and τ2τ^2-bench metrics. Method Tool-use Avg. τ-Airline τ-Retail τ2τ^2-Airline τ2τ^2-Retail τ2τ^2-Telecom Qwen3-8B-NoThink 22.16±2.3722.16± 2.37 12.00±0.0012.00± 0.00 29.86±5.3129.86± 5.31 14.61±7.7914.61± 7.79 36.55±1.0136.55± 1.01 17.80±2.5317.80± 2.53 Qwen3-8B-Think 31.35±0.1931.35± 0.19 28.00±6.0028.00± 6.00 36.81±4.2936.81± 4.29 29.75±3.2729.75± 3.27 38.71±1.0638.71± 1.06 23.46±4.1923.46± 4.19 PPO 33.60±1.3433.60± 1.34 35.33±3.0635.33± 3.06 41.45±3.9241.45± 3.92 33.33±3.0633.33± 3.06 38.89±2.2138.89± 2.21 19.01±1.3419.01± 1.34 PPO + ARC 38.45±0.4038.45± 0.40 39.33±2.3139.33± 2.31 46.09±1.7446.09± 1.74 41.61±0.6841.61± 0.68 44.44±3.6544.44± 3.65 20.76±2.2120.76± 2.21 DAPO 34.09±3.2634.09± 3.26 31.33±4.1631.33± 4.16 42.32±5.5242.32± 5.52 35.33±7.5735.33± 7.57 42.11±4.8842.11± 4.88 19.37±3.4019.37± 3.40 DAPO + ARC 34.24±1.8934.24± 1.89 34.67±2.3134.67± 2.31 41.74±3.9841.74± 3.98 34.50±3.9734.50± 3.97 40.64±7.0440.64± 7.04 19.64±2.7919.64± 2.79 GRPO 33.35±3.0433.35± 3.04 31.33±4.6231.33± 4.62 40.29±4.9440.29± 4.94 36.67±6.4336.67± 6.43 40.64±5.9740.64± 5.97 17.84±2.8217.84± 2.82 GRPO + ARC 41.73±2.1141.73± 2.11 44.00±3.0644.00± 3.06 50.00±1.3350.00± 1.33 48.00±6.1148.00± 6.11 45.61±0.8645.61± 0.86 21.05±0.9121.05± 0.91 Table 13: Robustness to training randomness. Results are reported as mean ± standard deviation over three independently trained models with different training seeds. Each seed-level score is obtained by averaging repeated evaluation runs for that checkpoint. Method Tool-use Avg. τ-Airline τ-Retail τ2τ^2-Airline τ2τ^2-Retail τ2τ^2-Telecom PPO 31.81±1.8731.81± 1.87 33.78±3.2933.78± 3.29 36.13±5.0936.13± 5.09 32.89±3.3632.89± 3.36 37.14±1.9237.14± 1.92 19.10±0.1719.10± 0.17 PPO + ARC 36.76±1.5636.76± 1.56 37.78±1.3937.78± 1.39 44.83±2.9644.83± 2.96 37.28±3.7837.28± 3.78 43.84±0.5943.84± 0.59 20.08±1.1820.08± 1.18 DAPO 34.69±0.5234.69± 0.52 35.55±4.0235.55± 4.02 40.97±1.3140.97± 1.31 35.56±1.0235.56± 1.02 41.91±0.3441.91± 0.34 19.44±0.4619.44± 0.46 DAPO + ARC 34.77±0.7034.77± 0.70 32.89±1.6832.89± 1.68 42.22±3.3642.22± 3.36 37.50±2.6237.50± 2.62 41.13±0.8541.13± 0.85 20.11±0.5820.11± 0.58 GRPO 34.74±1.3534.74± 1.35 32.67±1.3432.67± 1.34 42.61±2.0342.61± 2.03 38.20±1.3738.20± 1.37 41.80±3.3641.80± 3.36 18.44±0.8018.44± 0.80 GRPO + ARC 37.53±1.5337.53± 1.53 39.11±1.6839.11± 1.68 43.87±4.3643.87± 4.36 40.19±1.6540.19± 1.65 43.84±1.6043.84± 1.60 20.67±1.5320.67± 1.53 Evaluation uncertainty. Table 12 reports variability across repeated evaluations of the same trained checkpoint. For the main τ-bench and τ2τ^2-bench results, we perform three independent evaluation runs for each model checkpoint and report the mean and standard deviation across runs. In particular, ARC increases the tool-use average from 33.60 to 38.45 for PPO and from 33.35 to 41.73 for GRPO. The corresponding DAPO results are substantially closer, indicating that the effect of ARC depends on the underlying RL backbone. Robustness to training randomness. Table 13 evaluates sensitivity to optimization randomness across three independent training seeds. Each resulting checkpoint is evaluated using the same repeated-evaluation protocol. We first average the repeated evaluations within each seed and then report the mean and standard deviation across the three seed-level scores. The improvements are consistent across training seeds for PPO and GRPO. PPO improves from 31.81 to 36.76 on the tool-use average, while GRPO improves from 34.74 to 37.53. In contrast, DAPO remains essentially unchanged (34.69 versus 34.77), and we therefore do not interpret the DAPO result as evidence of a meaningful tool-use improvement. Appendix J Training Details J.1 Hyperparameters Table 14 details RL training hyperparameters. Table 14: RL hyperparameters for the main experiments. Parameter Value Base Model Qwen3-8B Learning Rate 1e-6 Batch Size 64 PPO Mini Batch Size 8 PPO Micro Batch Size (per GPU) 4 Max Prompt Length 6000 Max Response Length 1024 Training Epochs 1 KL Loss Coefficient 0.001 KL Loss Type low_var_kl Use KL in Reward False GRPO Groups (n) 8 Entropy Coefficient 0.001 J.2 Compute Resources All experiments were conducted on NVIDIA H200 GPUs, using NVIDIA driver version 570.158.01 and CUDA 13.0.