Paper deep dive

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo, Chenyi Zhuang, Leilei Gan

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 57

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/14/2026, 6:21:31 AM

Summary

The paper introduces CrEST, a hierarchical credit assignment framework for training multi-turn multi-step LLM agents. CrEST combines verifier-bounded reinforcement learning (RL) with dense token-level signals from a self-teacher. It addresses credit assignment at two levels: turn-segmented verified advantages for inter-turn credit and entropy-gated self-teacher modulation for intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show CrEST outperforms RL and distillation baselines, particularly in long-trajectory and strict session-level metrics.

Entities (12)

CrEST · method · 98%BFCL v3 · dataset · 95%WildToolBench · dataset · 95%Entropy-gated self-teacher modulation · concept · 90%On-Policy Distillation · method · 90%Qwen3-4B-Instruct · model · 90%Qwen3-8b · model · 90%RLVR · method · 90%Turn-segmented verified advantages · concept · 90%GRPO · method · 85%MT-GRPO · method · 85%OPSD · method · 85%

Relation Signals (10)

CrEST → evaluatedon → BFCL v3

confidence 95% · Experiments on BFCL V3 and WildToolBench show that CrEST consistently outperforms both RL and distillation baselines

CrEST → evaluatedon → WildToolBench

confidence 95% · Experiments on BFCL V3 and WildToolBench show that CrEST consistently outperforms both RL and distillation baselines

CrEST → uses → Entropy-gated self-teacher modulation

confidence 95% · while entropy-gated self-teacher modulation refines intra-turn token contributions.

CrEST → uses → Turn-segmented verified advantages

confidence 95% · CrEST resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution

CrEST → outperforms → RL

confidence 90% · CrEST consistently outperforms both RL and distillation baselines across two model scales

CrEST → outperforms → distillation baselines

confidence 90% · CrEST consistently outperforms both RL and distillation baselines across two model scales

RLVR → provides → verifier-bounded performance ceiling

confidence 90% · Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents

Qwen3-8b → usedwith →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce $\textbf{CrEST}$, a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher. $\textbf{CrEST}$ resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that $\textbf{CrEST}$ consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

55,310 characters extracted from source content.

Expand or collapse full text

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Zechuan Wang 1,3,∗ , Siyuan Lu 1,2,3,4,∗ , Hongxuan Zhang 3,5 , Linjian Mo 3 , Chenyi Zhuang 3 , Leilei Gan 1,† 1 Zhejiang University 2 Shanghai Innovation Institute 3 AWorld Team, Inclusion AI 4 Westlake University 5 Nanjing University Abstract Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded perfor- mance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce CREST, a hierarchical credit assignment framework that retains RL’s verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher. CREST resolves credit at two levels: turn-segmented veri- fied advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that CREST consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher ’s role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling. 1 Introduction Large language models (Hurst et al., 2024; Comanici et al., 2025; Qwen et al., 2025; Liu et al., 2024) are evolving from pure question-answering systems into autonomous agents capable of reasoning, acting, and interacting with diverse environments (Merrill et al., 2026; Jimenez et al., 2024; Xie et al., 2024; Liu et al., 2026; Froger et al., 2025; Towers et al., 2026). These scenarios share a common structure: agents must handle multi-turn sessions (sequences of user requests) where each turn requires multi-step execution (chains of actions and observations) (Shridhar et al., 2021; Wei et al., 2025a; Xie et al., 2024; OpenClaw, 2026). Reinforcement learning (Schulman et al., 2017; Ahmadian et al., 2024), particularly RL with verifiable rewards (RLVR) (Shao et al., 2024; Qian et al., 2026), is now the dominant post-training paradigm for tool-use agents (Liu et al., 2025; Zhang et al., 2025a; Yu et al., 2026b), offering a verifier-bounded performance ceiling determined solely by reward quality. Yet standard objectives broadcast a single trajectory-level reward across all tokens. Within a single turn, where all actions serve one query, this is merely noisy (Lu et al., 2026a). In multi-turn sessions (Li et al., 2023; Patil et al., 2025; Yu et al., 2026a), however, turns carry independent outcomes yet share one reward signal, making credit assignment ill-posed: on WildToolBench, no model exceeds 15% session accuracy, and performance degrades sharply with turn count (Laban et al., 2025). On-policy distillation (OPD) (Lu & Thinking Machines, 2025; Li et al., 2026; Song & Zheng, 2026) provides a natural credit assignment mechanism, where the teacher inherently scores each token in the student’s trajectory and distinguishes which tokens are preferable from the teacher’s perspective. However, this requires a same-family, tokenizer-matched teacher (Fu et al., 2026), which is a rare constraint satisfied in practice. On-policy self-distillation (OPSD) (Zhao et al., 2026; Hübotter et al., 2026; Shenfeld et al., 2026a; Penaloza et al., 2026) removes this dependency by using the student itself as self-teacher, conditioned on privileged information such as ground truth or successful demonstrations from rollout group peers. Yet ∗ Equal contributions. Work was done during Zechuan and Siyuan’s internship in Ant Group. † Corresponding Authors. 1 arXiv:2608.13179v1 [cs.AI] 13 Aug 2026 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Figure 1: Comparison of credit assignment strategies on a 2-turn trajectory (Turn 1 succeeds, Turn 2 fails). GRPO broadcasts a single trajectory-level reward uniformly, erroneously reinforcing failed-turn tokens. OPSD provides dense token-level signal but concentrates gradient on low-entropy format tokens, with a teacher-bounded ceiling. CREST (Ours) combines turn-level advantages (inter-turn credit) with entropy- gated self-teacher modulation (intra-turn credit), correctly assigning negative advantage to Turn 2 while focusing gradient mass on high-uncertainty content tokens in Turn 1. OPSD is brittle and restricts exploration. First, Zhao et al. (2026); Brown (2026) found that without per-token KL clipping, OPSD collapses within∼100 training steps due to gradient concentration on a few tokens. Second, OPSD’s performance ceiling remains teacher-bounded (Kim et al., 2026), which means that the student cannot surpass the teacher’s privileged distribution. This raises a central question: Can we design a training approach that maintains the verifier-bounded performance ceiling of RL while providing token-level signal to address the credit assignment challenges of multi-turn multi-step LLM agents training? We propose CREST (Hierarchical Credit Assignment via Entropy-Gated Self-Teacher), which addresses both levels through a unified policy gradient with structured per-token advantages (Figure 1). At the turn level, CREST uses turn-segmented verifier rewards to prevent rewards from being diluted across long multi-turn trajectories. At the token level, to prevent gradient concentration on a few tokens, CREST uses an entropy-gated self-teacher to modulate token-level contributions within each turn. Additionally, CREST employs the verified reward to determine all gradient directions, ensuring the performance ceiling remains verifier-bounded without requiring an external same-family teacher. The two level credit assignment are complementary—neither alone recovers the full performance. Our main contributions are: 1. We identify and formalize the two-level credit assignment problem in multi-turn multi-step agentic RL, and provide a gradient-geometric analysis explaining why existing dense-signal methods fail in this setting (Section A). 2.We propose CREST, a hierarchical credit assignment framework combining turn-segmented advantage with entropy-gated self-teacher token reweighting. CREST is verifier-bounded by construction and introduces only one hyperparameter (λ) beyond standard GRPO (Section 3). 3.We demonstrate that CREST achieves 52.0% average accuracy on BFCL V3 (Qwen3-4B) and 9.38% session accuracy on WildToolBench (Qwen3-8B), consistently outperforming all baselines across both benchmarks and model scales (Section 4). 2 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents 2 Related Work Tool use for LLM-based agents. LLM tool-use research spans a complexity spectrum, from reasoning before function-calling (Zhang et al., 2025a; Qian et al., 2026) to tool-integrated reasoning (TIR) for fulfilling user’s query (Li et al., 2025; Jin et al., 2025; Feng et al., 2025; Zheng et al., 2025). Most relevant to our work, multi-turn multi-step benchmarks (Zhang et al., 2025b; Yu et al., 2026a; Patil et al., 2025) evaluate agents across sessions of multiple queries, each requiring its own execution chain. These settings expose two challenges absent in single-turn work: cross-turn state consistency and error propagation across turns. Empirically, agent performance degrades substantially with turn count; Yu et al. (2026a) report session accuracy below 15% across 57 LLMs, motivating training-time methods that respect the turn structure of multi-turn sessions. Agentic Reinforcement Learning.RL-based post-training has become the standard approach for aligning LLM agents with task objectives (Shao et al., 2024; Yu et al., 2026b; Wang et al., 2025). Some methods focus on reward design: Lu et al. (2026a) decompose trajectory rewards into fine-grained per-step process scores combining state correctness and execution accuracy, while Qian et al. (2026) and Zhang et al. (2025a) ground rewards directly in environment feedback from tool execution. Despite these advances, most existing methods compute a single advantage per trajectory (or per-step within a single turn), leaving the inter-turn credit assignment problem unaddressed—when turns within the same session have heterogeneous outcomes, trajectory-level or step-level averaging still mislabels tokens in failed turns. MT-GRPO (Wei et al., 2025b) attempts per-agent-step advantage computation, yet still lacks intra-step token-level credit assignment, treating all tokens within each step identically. On-policy distillation.On-policy distillation (OPD) (Lu & Thinking Machines, 2025) replaces RL’s sparse reward with dense per-token reverse KL against a same-family teacher, providing token-level credit as- signment throughout the trajectory (Li et al., 2026). OPD converges faster than RL at moderate compute budgets, but its ceiling is teacher-bounded and requires tokenizer-matched, recipe-matched teachers (Brown, 2026). When such teachers are unavailable, self-distillation methods use the student itself conditioned on privileged information like expert demonstrations (Shenfeld et al., 2026b), feedback (Hübotter et al., 2026) and ground-truth answers (Zhao et al., 2026). More recently, hybrid methods attempt to combine RL’s verifier-bounded direction with distillation’s dense signal: SDAR (Lu et al., 2026b) gates self-distillation as an auxiliary loss alongside RL, and RLSD (Yang et al., 2026) uses self-distillation for update magnitudes while relying on verifiable rewards for directions. However, these hybrids either lack principled control over gradient concentration (Zhao et al., 2026; Brown, 2026) or do not address the inter-turn credit assignment structure unique to multi-turn settings. 3 Method Multi-turn multi-step training faces a hierarchical credit assignment problem: at the coarse level, trajectories mix successes and failures across turns; at the fine level, individual turns conflate high-value decisions with low-entropy formatting. CREST addresses both levels through a unified policy gradient framework with structured per-token advantages (Equation (1)): • Inter-turn (coarse): Turn-segmented verified-reward advantages (Section 3.2) isolate per-turn outcomes, preventing dilution from trajectory-level averaging. •Intra-turn (fine): Entropy-gated self-teacher signals (Section 3.3) refine per-token contributions, concen- trating gradient mass on high-uncertainty content tokens. The two levels are complementary: without inter-turn segmentation, no amount of intra-turn refinement can rescue tokens in failed turns; without intra-turn reweighting, pivot decisions in successful turns remain indistinguishable from formatting tokens. Critically, verified reward determines all gradient directions while the self-teacher only modulates magnitudes, ensuring the performance ceiling stays verifier-bounded (Section A). 3.1 Unified Objective CREST is a standard on-policy policy gradient with a structured per-token advantage. Given a promptxand an on-policy rollout y∼ π θ (·| x), the objective is: J(θ) =E x∼D, y∼π θ (·|x) " ∑ t A t · log π θ (y t | y <t , x) # (1) 3 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Figure 2: Overview of CREST. Left (Inter-turn): A group ofGrollouts is scored by the environment verifier per turn; group-relative advantagesA turn are computed independently for each turn, providing the gradient direction. Right (Intra-turn): Within each turn, a privileged self-teacher (conditioned on ground truth) provides per-token divergence∆ t ; an entropy gate concentrates modulation on high-uncertainty content tokens (e.g.,get_flight,"JFK") while suppressing format tokens, yielding the magnitude factorφ t . Bottom: The final per-token advantageA t = A turn [t] × φ t combines verifier-bounded direction with teacher-modulated magnitude. where the per-token advantageA t exposes the hierarchical structure: A t =A turn [t] | z inter-turn: which turn? ·φ t |z intra-turn: which token? (2) A turn [t] is a turn-segmented verified-reward advantage (Section 3.2) that resolves which turn contributed to success or failure.φ t ∈ [1, 1+λε]is a magnitude modulation factor (Section 3.3) that refines which tokens within that turn deserve stronger updates, derived from a privileged self-teacher gated by student uncertainty. Table 1: Design space of hierarchical credit assignment. Methods differ in how they allocate credit across turns (A turn [t] ) and tokens (φ t ). MT-GRPO computes per-turn advantages but propagates discounted future-outcome credit to earlier turns (cumulative), whereas CREST computes each turn’s advantage from that turn’s re- ward only (independent). OPD and OPSD share the same slot (φ t ∝∆ t with no verified-reward base, hence teacher-bounded); they differ only in teacher source (external same-family vs. self with privileged context). MethodA turn [t] φ t GRPOtrajectory-level1 MT-GRPOper-turn (cumulative)1 OPD / OPSD—∝∆ t CREST (Ours)per-turn (independent)1 + λ eff t (w t − 1) The factorization in Equation (2) reveals why both levels are necessary:A turn [t] provides the correct sign (reinforce or suppress) for each turn’s tokens, but treats all tokens within a turn uniformly;φ t refines the magnitude of each token’s contribu- tion, but cannot change sign (proof in Section A). Together, they ensure gradient direction is de- termined entirely by verified rewards (verifier- bounded), while token-level allocation benefits from dense teacher signals. Table 1 shows how existing methods occupy restricted subspaces of this design. 3.2 Turn-Level Credit Assignment Instead of computing a single trajectory-level re- ward averaged across turns, we compute group- relative advantages within each turn independently (Figure 2, left). For a trajectory containing turns 1,. . .,K, each turnkreceives its own verified rewardR k from the environment. Within a GRPO group ofG rollouts, the advantage for turn k in rollout i is: A (i) k = R (i) k − mean(R (j) k G j=1 ) std(R (j) k G j=1 ) + ε (3) All tokens within turnkof rolloutishare the advantageA (i) k . This ensures that a failed turn receives negative advantage regardless of whether other turns in the same trajectory succeeded. 4 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents 3.3 Token-Level Credit: Self-Teacher Signal The modulation factorφ t uses a privileged self-teacher (the same model conditioned on ground-truth context) to refine per-token gradient magnitude within each turn (Figure 2, right): φ t = 1 + λ eff t · (w t − 1)(4) Whenλ eff t =0,φ t =1 (no teacher influence); whenλ eff t >0 andw t >1,φ t >1 (teacher amplifies the gradient). We now define the two components: the raw teacher signal (∆ t → w t ) and the gating mechanisms that control it (Section 3.4). Teacher–student divergence. For each response tokent, letπ T (·| h T t )denote the self-teacher—the same model conditioned on ground truth as privileged context, following Zhao et al. (2026)—andπ θ (·| h t )the student: ∆ t = log π T (y t | h T t )− log π θ (y t | h t ) τ (5) where τ is a temperature that smooths the divergence signal. Token weight. The raw teacher preference is converted into a bounded multiplicative weight: w t = clip exp sign(A turn [t] )·∆ t , 1−ε, 1+ε (6) Thesign(A turn [t] )·∆ t construction aligns teacher preference with the verifier ’s direction: for positive-advantage tokens, teacher agreement increasesw t ; for negative-advantage tokens, teacher agreement with the “this was bad” signal also increasesw t . This guarantees direction preservation (P1):sign(A t ) = sign(A turn [t] )for allt, i.e., the gradient direction is determined entirely by verified reward (proof in Section A). 3.4 Stabilization via Selective Gating The self-teacher provides dense per-token signals, but unconstrained use leads to concentration collapse (Brown, 2026): gradient mass flows to a small set of tokens where teacher–student divergence is largest, often low- entropy formatting rather than high-value decisions. We stabilize the intra-turn reweighting through a composed gating functionλ eff t (Equation (10)) that controls where (agreement with verifier) and how much (student uncertainty) the teacher signal is applied. Direction gate. g dir t = 1 h sign(A turn [t] )·∆ t > 0 i (7) When teacher and verifier disagree on the direction of update,g dir t =0 and the teacher signal is fully suppressed. This is the key mechanism ensuring the performance ceiling remains verifier-bounded rather than teacher-bounded. Entropy gate.To prevent gradient concentration on low-entropy format tokens, we modulate the reweight- ing strength by the student’s token-level uncertainty, using surprisalu t =− log π θ (y t | h t )as a lightweight proxy: g ent t = σ u t −E[u] std(u) + ε (8) m ent t = 1 + ρ (2 g ent t − 1)(9) whereσis the sigmoid function and statistics are computed over response tokens. The Z-score normalization centers surprisal around the mean and scales by standard deviation, mapping to[0, 1]via sigmoid. For low-uncertainty format tokens,m ent t <1 reduces modulation strength; for high-uncertainty content tokens, m ent t > 1 increases it. The range [1−ρ, 1+ρ] redistributes gradient budget without zeroing out any token. Composed gating. The direction and entropy gates compose into a single effective gating function: λ eff t = clip    λ· g dir t · m ent t | z safety & focus , 0, λ    (10) 5 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents This yieldsφ t =1+ λ eff t (w t −1) ∈ [1, 1+λε], ensuring three formal properties (Section A): (P1) direction preservation (sign(A t ) = sign(A turn [t] )always), (P2) bounded bias (∥Bias∥ ≤ λε·∥∇ J GRPO ∥ ≈8.4% with λ=0.3, ε=0.28), and (P3) sign-consistent amplification (φ t ≥ 1 when active). λis the only tunable hyperparameter controlling teacher influence strength. The remaining values— τ=2.0 (divergence temperature),ε=0.28 (clip bound, inherited from OPSD stabilization),ρ=0.5 (entropy redistribution range)—are fixed constants requiring no tuning (ablation in Section 4). The complete training procedure, consolidating all equations into executable pseudocode, is provided in Section B. 4 Experiments 4.1 Experimental Setup Benchmarks. We conduct experiments on two multi-turn multi-step tool-use benchmarks: (1) Berkeley Function-Calling Leaderboard (BFCL) V3 (Patil et al., 2025). Under our processed protocol, we use 100 fixed IDs from the multi-turn Base split for training and 400 non-overlapping examples for evaluation: 100 examples each from Base, Missing Functions, Missing Parameters, and Long-Context. (2) WildToolBench (Yu et al., 2026a) contains 256 multi-turn sessions designed to capture real-world interaction challenges, including compositional tasks, implicit intent through pronoun reference or ellipsis, and task switching between goal-directed queries and casual chat. The benchmark evaluates both action correctness and parameter accuracy. We allocate 128 sessions for training and reserve the remaining ones for evaluation. The turn index is semantically aligned within each rollout group: all rollouts use the same fixed session and the same ordered list of user turns, so turnkdenotes the same benchmark request across the group. This assumption is specific to these benchmarks and does not directly extend to variable turn structures, terminal-only rewards, or strongly coupled turns without local verification. BFCL rewards compare post- execution environment states with annotated per-turn target executions, including result-coverage checks for read-only tools; WildToolBench rewards path-match calls against human-verified valid paths and score action correctness, parameter correctness, and path completion. Base models. We evaluate on two model scales from the Qwen3 family (Yang et al., 2025): Qwen3-4B- Instruct, 1 an instruction-tuned model, and Qwen3-8B, a thinking model. Both have native tool-calling support and represent practical deployment sizes where post-training efficiency matters most. Baselines.We compare against four training methods spanning RL-based and distillation-based paradigms: GRPO (Shao et al., 2024) applies standard Group Relative Policy Optimization with binary trajectory-level reward (1 if all turns succeed, 0 otherwise). Advantages are averaged across all tokens in the trajectory regardless of per-turn outcomes. MT-GRPO (Wei et al., 2025b) computes advantages at the per-agent-step granularity, providing finer-grained credit assignment than trajectory-level methods. However, within each step, all tokens still share the same advantage, lacking token-level differentiation. EnvTuning (Lu et al., 2026a) uses fine-grained process rewards that combine per-turn state correctness and execution accuracy, aggregating per-turn scores into a trajectory-level reward via weighted averaging. Despite richer reward signals, it still computes a single trajectory-level advantage. OPD (Lu & Thinking Machines, 2025) performs on-policy distillation with a same-family, same-variant teacher, providing dense per-token supervision via reverse KL. To satisfy the tokenizer-matched constraint, we pair each student with a teacher of the same model variant: Qwen3-4B-Instruct uses Qwen3-235B-A22B- Instruct 2 as teacher, while Qwen3-8B uses Qwen3-32B. OPD’s performance ceiling is teacher-bounded. OPSD (Zhao et al., 2026) uses on-policy self-distillation where the student itself, conditioned on ground-truth context, serves as the privileged teacher. This removes the external teacher dependency but risks training instability from gradient concentration. Implementation details. All methods are trained with the same on-policy rollout infrastructure, group sizeG=16, and learning rate schedule. For CREST, we setλ=0.3,ε=0.28,τ=2.0, andρ=0.5; onlyλis tuned, 1 https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 2 https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507 6 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Table 2: Main results on BFCL V3 multi-turn and WildToolBench benchmarks. Best results per model in bold, second-bestunderlined. Methods are grouped by training paradigm: RL-based (top), distillation-based (middle), and our method (bottom). Method BFCL V3 Multi-Turn (%)WildToolBench (%) AverageBaseMiss FuncMiss ParamLong ContextTask Acc.Session Acc. Qwen3-4B-Instruct22.1226.521.015.525.537.893.13 + GRPO43.6353.542.531.547.040.234.69 + MT-GRPO49.2563.046.035.053.043.366.25 + EnvTuning47.2560.047.032.050.044.924.69 + OPD44.5052.043.038.045.042.586.25 + OPSD38.7546.041.026.042.038.884.69 + CREST52.00 (+29.88)67.0 (+40.5)48.0 (+27.0)38.0 (+22.5)60.0 (+34.5)48.44 (+10.55)7.03 (+3.90) Qwen3-8B33.3841.538.527.026.543.754.69 + GRPO43.2553.046.036.038.045.435.47 + MT-GRPO44.0057.045.036.038.049.617.03 + EnvTuning46.0054.045.040.045.049.027.81 + OPD44.7550.052.039.038.045.253.91 + OPSD41.7548.043.039.037.043.953.91 + CREST50.00 (+16.62)60.0 (+18.5)51.0(+12.5)42.0 (+15.0)47.0 (+20.5)52.34 (+8.59)9.38 (+4.69) while the other values are fixed design choices across all experiments. Each method uses a single training seed. We evaluate each trained policy with three decodes at temperature 10 −6 ; these runs measure decoding consistency rather than training variance, and should not be interpreted as independent training replicates. Full training configuration and OPSD stabilization details are provided in Section C. 4.2 Main Results Table 2 reports BFCL V3 and WildToolBench performance for both model scales across all baselines. CREST outperforms every baseline on average and achieves the best result on nearly all individual splits across both benchmarks and both model scales. RL outperforms distillationTwo patterns stand out in Table 2. First, RL-based methods (GRPO, MT-GRPO, EnvTuning) systematically outperform distillation-based methods (OPD, OPSD) at both model scales. The clearest signal is OPSD: on Qwen3-4B BFCL Avg it scores 38.75%, falling below GRPO (43.63%) despite using a privileged ground-truth-conditioned teacher, giving direct empirical evidence for the teacher-bounded ceiling argued in Section 3. Second, within the RL family, finer-grained credit assignment helps: on Qwen3-4B BFCL Avg, MT-GRPO (49.25%) and EnvTuning (47.25%) both improve clearly over trajectory-level GRPO (43.63%), yet every token within a step still shares a single advantage. The remaining headroom for dense per-token supervision, on top of verifier-determined directions, is exactly the room CREST is designed to fill. CREST closes both gaps simultaneously.CREST reaches 52.00% on Qwen3-4B-Instruct and 50.00% on Qwen3-8B, surpassing the strongest RL baseline at each scale and achieving the best or tied-best result in 13 of 14 cells. OPD is best on Qwen3-8B Missing Functions, while CrEST ties OPD on Qwen3-4B Missing Parameters. The margins are largest on the splits that stress hierarchical credit assignment: on BFCL Long Context (the longest-trajectory split), CREST improves over the strongest baseline by+7.0 on 4B and+2.0 on 8B; on WildToolBench Session Accuracy (the strictest end-to-end multi-turn metric), by+0.78 on 4B and +1.57 on 8B. These are exactly the regimes where credit dilution is worst, validating that turn-segmented advantages and entropy-gated modulation address complementary failure modes. 4.3 Training Dynamics Analysis Figure 3(a) compares training rollout accuracy on BFCL V3 (Qwen3-4B). CREST reaches 0.60 accuracy within ∼20 steps and converges to∼0.70, while GRPO rises slowly to∼0.57 by step 160. OPSD plateaus at∼0.49 and declines thereafter, empirically confirming the teacher-bounded ceiling discussed by Lu & Thinking Machines (2025) and Brown (2026). CREST surpasses this ceiling by step 20 and continues improving, validating that our direction gate successfully decouples gradient magnitude (teacher-informed) from gradient direction (verifier-determined). Figure 3(b) provides a mechanistic explanation for the performance gap. We measure the fraction of total gradient magnitude captured by the top-p% of tokens (ranked by per-token advantage magnitude) for each method. OPSD exhibits extreme concentration: the top-1% of tokens account for∼42% of the total gradient 7 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents ((a)) Training rollout accuracy.((b)) Top-p% token gradient share. Figure 3: Training dynamics on BFCL V3 (Qwen3-4B). (a) CREST achieves both faster convergence and a higher ceiling than GRPO, surpassing the OPSD ceiling (dashed) by step 20, while OPSD plateaus and declines. (b) Top-p% token gradient share: OPSD concentrates>77% of gradient on the top-5% tokens, so its signal is dominated by a few tokens; CREST’s entropy gate redistributes gradient budget to high-uncertainty content tokens. signal, and the top-5% account for∼77%. This matches the gradient-geometric analysis of Brown (2026): a handful of pivot tokens dominate the update, causing instability. GRPO, by contrast, is far more diffuse (top-10%≈31%), but at the cost of treating all tokens equally. CREST occupies the desirable middle ground: more concentrated than GRPO (top-10%≈57%), confirming that the self-teacher signal provides meaningful token differentiation, but far less extreme than OPSD, confirming that the entropy gate prevents concentration collapse. 4.4 Ablation Studies Table 3: Hierarchical ablation on BFCL V3 (Qwen3-4B- Instruct). Neither inter-turn nor intra-turn credit as- signment alone recovers the full CREST performance. MethodAvgBaseM.FuncM.ParamL.Ctxt GRPO (baseline)43.6353.542.531.547.0 + Inter-turn only47.8862.044.035.550.0 + Intra-turn only48.7561.050.032.052.0 + Both (CREST)52.0067.048.038.060.0 Effect of hierarchical credit decomposition.Ta- ble 3 isolates the contribution of each credit as- signment level. The Intra-turn only variant retains the trajectory-level GRPO group-relative advan- tageA traj i for every token and applies only the teacher modulation,A t = A traj i φ t ; it uses neither turn segmentation nor process rewards. Inter-turn segmentation alone improves average accuracy over GRPO (43.63→47.88), with the largest gain on Base (+8.5) where cross-turn reward dilution is most severe. Intra-turn modulation alone yields+5.12 points, with particular strength on Miss Func (+7.5) where distinguishing content tokens from format tokens matters most. Combining both levels achieves+8.37 points (43.63→52.00), with Long Context showing the strongest synergy. Neither component alone approaches the full model, confirming that the two levels address complementary failure modes: turn segmentation provides correct per-turn reward direction, while entropy gating allocates gradient budget to high-value tokens within each turn. Table 4: Gating ablation on BFCL V3 (Qwen3-4B- Instruct). Removing the direction gate caps perfor- mance at the teacher-bounded ceiling; removing the entropy gate causes gradient concentration on format tokens; removing both leads to training instability. VariantAvgBaseM.FuncM.ParamL.Ctxt CREST52.0067.048.038.060.0 + w/o Direction gate46.7560.047.028.052.0 + w/o Entropy gate46.2559.050.027.049.0 + w/o Both gates43.5053.043.027.051.0 Both gates are necessary. Table 4 ablates the di- rection gate and entropy gate individually and jointly. Removing the direction gate drops average accuracy from 52.00% to 46.75% (−5.25), with Miss Param suffering the largest decline (38.0→28.0), consistent with the teacher overriding verifier sig- nals on ambiguous parameter tokens. Remov- ing the entropy gate yields a similar degradation (52.00→46.25), with Long Context dropping from 60.0 to 49.0, where long sequences amplify gra- dient concentration on format tokens. Removing 8 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents both gates (43.50%) falls to GRPO-level performance, confirming that uncontrolled self-distillation provides no benefit over standard RL. A sensitivity analysis of λ is provided in Section C.3. 5 Conclusion We presented CREST, a hierarchical credit assignment framework for multi-turn multi-step agent training. By decomposing per-token advantages into turn-segmented verified rewards and entropy-gated self-teacher modulation, CREST resolves the two-level credit assignment problem that afflicts standard RL in multi- turn settings: inter-turn reward dilution and intra-turn token uniformity. Experiments on BFCL V3 and WildToolBench demonstrate that CREST consistently outperforms both RL and distillation baselines across two model scales, with ablations confirming that the two levels address complementary failure modes. Our central finding is that a self-teacher need not determine gradient directions to be useful—restricting it to magnitude modulation, gated by student uncertainty, unlocks dense credit assignment while preserving the verifier-bounded ceiling that makes RL effective. Several directions for future work are worth exploring. The magnitude-only modulation principle demon- strated here could be applied to other structured generation tasks beyond tool use, such as multi-hop retrieval or collaborative dialogue, where hierarchical outcome structure is present. Additionally, the design space between our fixed self-teacher and fully online co-training remains largely unexplored; adaptive scheduling of teacher influence strengthλacross training stages may yield further gains. Finally, combining CREST with complementary advances in reward design (e.g., progress reward from Lu et al. (2026a)) could compound the benefits of both finer reward signals and better credit allocation. 6 Limitations Our evaluation is conducted on two multi-turn tool-use benchmarks at the 4B and 8B scale. While these represent distinct interaction paradigms (structured function-calling and naturalistic agentic sessions), broader validation across additional benchmarks and larger model scales would further strengthen the generality claims. Additionally, the entropy gate uses per-token surprisal as a proxy for token importance, which is effective for separating format tokens from content tokens in tool-use trajectories but remains a heuristic that may not optimally capture token informativeness in all generation contexts. 9 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents References Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12248–12267, 2024. Will Brown. On SFT, RL, and on-policy distillation.https://x.com/willccbb/status/2050038277454143918, 2026. Blog post. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536, 2025. Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, et al. Are: Scaling up agent environments and evaluations. arXiv preprint arXiv:2509.17158, 2025. Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562, 2026. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint, 2026. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, p. 54107–54157, 2024. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms? arXiv preprint arXiv:2603.24472, 2026. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Llms get lost in multi-turn conversation. arXiv preprint arXiv:2505.06120, 2025. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 3102–3116, 2023. Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025. Yaxuan Li et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025. 10 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Simon Yu, Xiangxin Zhou, Haotian Xu, Shaopan Xiong, Bo Liu, Chenmien Tan, Weixun Wang, Hao Zhu, Weiyan Shi, Diyi Yang, Michael Qizhe Shieh, Yee Whye Teh, Wee Sun Lee, and Min Lin. GEM: A gym for generalist LLMs. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=vsqQ1lG52a. Kevin Lu and Thinking Machines. On-policy distillation of language models.https://thinkingmachines. ai/blog/on-policy-distillation/, 2025. Blog post. Siyuan Lu, Zechuan Wang, Hongxuan Zhang, Qintong Wu, Leilei Gan, Chenyi Zhuang, Jinjie Gu, and Tao Lin. Don’t just fine-tune the agent, tune the environment. In The Fourteenth International Conference on Learning Representations, 2026a. URL https://openreview.net/forum?id=nzodtGccEM. Zhengxi Lu et al. Self-distilled agentic reinforcement learning. arXiv preprint arXiv:2605.15155, 2026b. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. OpenClaw. Openclaw.https://github.com/openclaw/openclaw, 2026. Open-source personal AI assistant, version 2026.3.8, accessed 2026-03-09. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tur, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. Advances in Neural Information Processing Systems, 38: 105523–105553, 2026. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026a. Idan Shenfeld et al. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026b. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0IOX0YcCdTn. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626, 2026. Mark Towers, Ariel Kwiatkowski, John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Kallinteris Andreas, Markus Krimmel, Arjun Kg, Rodrigo Perez-Vicente, et al. Gymnasium: A standard interface for reinforcement learning environments. Advances in Neural Information Processing Systems, 38, 2026. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025. 11 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025a. Quan Wei, Siliang Zeng, Chenliang Li, William Brown, Oana Frunza, Wei Deng, Yuriy Nevmyvaka, Yang Katie Zhao, Alfredo Garcia, and Mingyi Hong. Reinforcing multi-turn reasoning in LLM agents via turn-level reward design and credit assignment. In First Workshop on Multi-Turn Interactions in Large Language Models, 2025b. URL https://openreview.net/forum?id=drP7qVUnUt. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Chenxu Yang et al. Self-distilled RLVR. arXiv preprint arXiv:2604.03128, 2026. Peijie Yu, Wei Liu, Yifan Yang, Jinjian Li, Zelong Zhang, Xiao Feng, and feng zhang. Benchmarking LLM tool-use in the wild. In The Fourteenth International Conference on Learning Representations, 2026a. URL https://openreview.net/forum?id=yz7fL5vfpn. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026b. Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemotron-research-tool-n1: Tool-using language models with reinforced reasoning. arXiv preprint arXiv:2505.00024, 2(10), 2025a. Yiran Zhang, Mo Wang, Xiaoyang Li, Kaixuan Ren, Chencheng Zhu, and Usman Naseem. Turnbench-ms: A benchmark for evaluating multi-turn, multi-step reasoning in large language models. Findings of the Association for Computational Linguistics: EMNLP, p. 19892–19924, 2025b. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deep- researcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 414–431, 2025. 12 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents A Gradient Analysis We prove the three properties stated in §3. Recall the structured per-token advantage: A t = A turn [t] · φ t , φ t = 1 + λ eff t · (w t − 1) (11) and the resulting policy gradient: ∇ θ J(θ) =E " ∑ t A t ∇ θ log π θ (y t ) # (12) A.1 (P1) Direction Preservation Proposition A.1 (Direction Preservation). For all tokens t, sign(A t ) = sign(A turn [t] ). Proof. We show φ t > 0 always holds, so multiplication by φ t preserves the sign of A turn [t] . Case 1: g dir t = 0 (direction gate off). Then λ eff t = 0 by Eq. 10, so φ t = 1> 0. Case 2:g dir t =1 (direction gate on). This requiressign(A turn [t] )·∆ t > 0 by Eq. 7, which impliesexp(sign(A turn [t] )· ∆ t )> 1. After clipping, w t ∈ [1, 1 + ε], so w t − 1≥ 0. Since λ eff t ≥ 0: φ t = 1 + λ eff t |z ≥ 0 (w t − 1) | z ≥ 0 ≥ 1> 0(13) In both cases φ t > 0, so sign(A t ) = sign(A turn [t] · φ t ) = sign(A turn [t] ). Interpretation. The scalar coefficient of each token’s policy-gradient contribution has the same sign as the inter-turn-only coefficient (i.e., CREST withφ t =1). The teacher signal modulates the magnitude of each token’s contribution but cannot flip its scalar sign. This is a local, token-wise anchoring property: reweighting can still change how token gradients cancel in the aggregate, so it does not imply that aggregate gradient directions, stationary points, or a global performance ceiling are determined solely by the verifier. A.2 (P2) Absolute Perturbation Bound Proposition A.2 (Absolute Perturbation Bound). For a sampled trajectory, letg t = A turn [t] ∇ θ log π θ (y t | h t ) denote the per-token inter-turn gradient contribution. Let d ∇ θ Jand d ∇ θ J TG denote the reweighted and inter-turn-only single-trajectory gradients, respectively. The teacher-induced perturbation satisfies: ∑ t (φ t − 1)g t ≤ λε ∑ t ∥g t ∥.(14) The bound is absolute; it is not relative to ∥ ∑ t g t ∥ . Proof. The difference between the reweighted and inter-turn-only gradients on one sampled trajectory is: d ∇ θ J− d ∇ θ J TG = ∑ t (φ t − 1)g t .(15) Ifg dir t = 0, thenφ t =1. Ifg dir t = 1, thenφ t −1= λ eff t (w t − 1), withλ eff t ≤ λ andw t −1≤ εby Eqs. 10 and 6. Thus, for every token,|φ t − 1|≤ λε. Applying the triangle inequality gives: ∑ t (φ t − 1)g t ≤ ∑ t |φ t − 1|∥g t ∥≤ λε ∑ t ∥g t ∥.(16) 13 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Interpretation. The teacher-induced perturbation is controlled in absolute magnitude by the sum of per-token inter-turn gradient norms. With the default settings (λ =0.3,ε =0.28), each token’s advantage coefficient can be amplified by at most 8.4%; this is not a relative bound on the aggregate gradient, which may be small because token contributions cancel. Reducingλstill makes the per-token modulation arbitrarily weak, smoothly approaching the inter-turn-only ablation. A.3 (P3) Sign-Consistent Amplification Proposition A.3 (Sign-Consistent Amplification). Wheng dir t =1,φ t ≥1. That is, the teacher can only amplify the gradient magnitude along the verifier-approved direction, never suppress it. Proof. This was established in the proof of (P1), Case 2:g dir t =1=⇒ w t ≥1=⇒ φ t =1+ λ eff t (w t −1)≥ 1. Interpretation. Combined with (P1), this means: for every token, either the teacher has no effect (φ t =1, when the direction gate is off) or the teacher increases the magnitude of the update in the direction already approved by the verified reward (φ t >1). The teacher signal is therefore a selective token-level amplifier. This does not imply that the aggregate parameter update or its stationary points are unchanged, because token gradients can cancel differently after reweighting. A.4 Discussion: Relationship to Adaptive Preconditioning The modulation factorφ t can be interpreted as a form of token-level adaptive preconditioning on the policy gradient. Standard GRPO applies a uniform scale to all tokens within a trajectory (or turn). Our method replaces this with a per-token scale that is: • Larger on tokens where the privileged teacher agrees with the verifier direction and the student is uncertain (high surprisal). • Equal to 1 on tokens where the teacher disagrees, or the student is already confident. This is analogous to how natural gradient methods or Adam apply per-parameter adaptive learning rates, except that our formal guarantee is token-wise scalar-sign preservation rather than preservation of the aggregate parameter-gradient direction. In our case, the “curvature proxy” is the teacher–student divergence gated by student uncertainty, and the adaptation is at the token level rather than the parameter level. The key difference from OPSD is that OPSD uses the teacher signal as the advantage itself (both direction and magnitude), whereas our method uses it only as a magnitude modulator on top of verified reward. Thus, OPSD’s token-level optimization target is teacher-defined, while CrEST keeps each token’s scalar update sign anchored to the verifier; this distinction is not a guarantee about the aggregate optimum or a global performance ceiling. B Complete Algorithm Algorithm 1 provides the complete pseudocode for a single CREST training step, consolidating all equations from Section 3 into an executable procedure. C Experimental Details C.1 Training Configuration All experiments use the following shared configuration: •Hardware: Single-node 8×H200 GPUs. For methods requiring a teacher (OPSD, CREST), 4 GPUs serve the teacher via SGLang and 4 GPUs handle student training and rollout generation, with tensor parallelism =2 and sequence parallelism enabled. For OPD with Qwen3-4B-Instruct, the external teacher (Qwen3- 235B-A22B-Instruct) requires an additional 8 GPUs for deployment, resulting in a 16-GPU setup (8 for teacher, 8 for student). •Rollout: On-policy sampling with group sizeG=16, temperature 1.0, max generation length 10,000 tokens per rollout. • Batch size: 32 prompts per step× 16 samples per prompt = 512 rollouts per step. • Optimizer: Adam with β 1 =0.9, β 2 =0.999, weight decay 0.01, gradient clipping 1.0. 14 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents Algorithm 1 CREST Training Step Require: Prompt batchD, policy π θ , group size G, hyperparams λ, τ, ε, ρ 1: for each prompt x ∈D do 2:Sample G rolloutsy (i) G i=1 ∼ π θ (·|x) 3:Obtain per-turn verified rewardsR (i) k 4:// Inter-turn: turn-segmented advantage 5:for each turn k do 6:A (i) k ← (R (i) k − μ k )/(σ k + ε)(Equation (3)) 7:end for 8:// Intra-turn: entropy-gated self-teacher 9:Compute privileged teacher logprobs log π T (y t | h T t ) 10:for each token t in rollout i do 11:∆ t ← (log π T − log π θ )/τ(Equation (5)) 12:w t ← clip(exp(sign(A turn [t] )·∆ t ), 1−ε, 1+ε)(Equation (6)) 13:g dir t ← 1[sign(A turn [t] )·∆ t > 0](Equation (7)) 14:g ent t ← σ((u t −E[u])/(std(u) + ε))(Equation (8)) 15:λ eff t ← clip(λ· g dir t · (1 + ρ(2g ent t − 1)), 0, λ)(Equation (10)) 16:A t ← A turn [t] · (1 + λ eff t · (w t − 1)) 17:end for 18: end for 19: Update θ via policy gradient with per-token advantagesA t • Learning rate: 1×10 −6 , constant schedule. •Training steps: The main Qwen3-4B BFCL V3 runs use method-specific schedules: 145 steps for GRPO, 150 for MT-GRPO, 160 for EnvTuning, 100 for OPD, 89 for OPSD, and 160 for CREST. We do not assume a single universal step count across methods. • PPO clipping: ε low =0.2, ε high =0.28. • KL penalty: β KL =0.0 (no explicit KL regularization; clipping provides implicit constraint). For CREST-specific hyperparameters:λ=0.3 (modulation strength),ε=0.28 (clip bound),τ=2.0 (temperature for∆ t ), andρ=0.5 (entropy gate range). Onlyλis tuned; the remaining values are fixed design choices across all benchmarks and model scales. Each method is trained with a single training seed. Evaluation uses three decodes at temperature 10 −6 ; these repeated decodes measure decoding consistency rather than training variance and are not independent training replicates. The self-teacher is constructed by conditioning the same model on ground-truth tool-call results for preceding turns, providing privileged context without requiring an external teacher model or tokenizer alignment. C.2 OPSD Stabilization and Teacher Update Strategies Directly applying OPSD (Zhao et al., 2026) to multi-turn tool-use trajectories results in training collapse within∼20 steps, consistent with findings reported by the original authors. To establish a fair OPSD baseline, we investigated three teacher update strategies (Figure 4(a)): • Online teacher (updated with policy): Catastrophic collapse by step 20—training accuracy drops from ∼0.45 to near zero, caused by pivot-token concentration shifting direction each update. •Fixed teacher (frozen at initialization): The most stable strategy, maintaining training accuracy around 0.44–0.48 throughout. However, the static teacher distribution defines the token-level optimization target and produces an observed plateau. •EMA teacher (exponential moving average,α=0.995): Comparable stability to the fixed teacher but no improvement—the teacher converges to a lagged copy of the student, erasing the privileged information advantage. Based on these results, our OPSD baseline adopts the fixed teacher with per-token KL clipping (clip threshold 0.2) as recommended by Zhao et al. (2026). This prevents collapse but leaves the student at the observed 15 Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents ((a)) OPSD teacher update strategies.((b)) Modulation strength λ sensitivity. Figure 4: Analysis of OPSD teacher update strategies and modulation strength on BFCL V3 (Qwen3-4B). (a) Comparison of OPSD teacher update strategies: the online teacher collapses catastrophically within∼20 steps, while fixed and EMA teachers are both stable, with the fixed teacher achieving marginally higher and more consistent accuracy; we adopt the fixed teacher with per-token KL clipping (threshold 0.2) for the OPSD baseline in all experiments. (b) Effect of the modulation strengthλon training accuracy:λ=0.3 provides the best trade-off; higher values over-concentrate gradients. plateau associated with a teacher-defined optimization target shown in Figure 4(a). This empirical observation directly motivates CREST’s design: rather than using the teacher signal as the optimization target, we use it solely as a gradient magnitude modulator anchored to the verified token sign. C.3 Modulation Strength Sensitivity Figure 4(b) varies the modulation strengthλ, which controls how aggressively the entropy gate selectively amplifies token updates. Atλ=0 (inter-turn-only CREST,φ t =1), final accuracy is∼0.63. Atλ=0.3, entropy- gated modulation raises final accuracy to∼0.69 (+6%) by emphasizing high-uncertainty content tokens and reducing amplification on low-entropy format tokens. Atλ=0.5, over-concentration causes early gains but stagnation belowλ=0.1 by step 80, ending at∼0.61. The sweet spot atλ=0.3 confirms that calibrated selective amplification, not maximal concentration, best resolves intra-turn credit ambiguity. 16

Instant research discovery