Paper deep dive
AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL
Xiaolong Jin, Dingmin Wang, Vijay Lingam, Varun Kumar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/26/2026, 5:42:10 AM
Summary
The paper introduces AHEAD (Adaptive Hindsight with Environment-Augmented Distillation), a step-aware framework for training multi-turn LLM agents using reinforcement learning. AHEAD addresses the limitation of trajectory-level rewards by distinguishing between routine steps and critical error steps. It uses environment feedback as a base signal for all steps and injects LLM-generated corrective hints specifically for error steps identified in failed trajectories. This adaptive privileged information is used in a self-distillation process to reweight the GRPO advantage, improving sample efficiency and task success rates across benchmarks like ALFWorld, WebShop, and Search-based QA.
Entities (10)
Relation Signals (10)
AHEAD → buildsupon → GRPO
confidence 95% · The method introduces minimal changes to the standard GRPO algorithm.
AHEAD → evaluatedon → ALFWorld
confidence 95% · Across ALFWorld, WebShop, and Search-based QA... AHEAD raises task success
AHEAD → evaluatedon → WebShop
confidence 95% · Across ALFWorld, WebShop, and Search-based QA... AHEAD raises task success
AHEAD → improves → task success
confidence 95% · AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO)
AHEAD → uses → Environment Feedback
confidence 95% · The teacher receives environment feedback on all steps as a grounded dense signal
AHEAD → uses → Corrective Hint
confidence 95% · additionally receives LLM-generated corrective hints on error steps
AHEAD → evaluatedon → Search-based QA
confidence 90% · Across ALFWorld, WebShop, and Search-based QA... AHEAD raises task success
AHEAD → improves → Sample Efficiency
confidence 90% · reaches a given success rate in fewer training steps
AHEAD → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
Tags
Links
- Source: https://arxiv.org/abs/2608.24114v1
- Canonical: https://arxiv.org/abs/2608.24114v1
Trouble viewing inline? Open PDF directly →
Full Text
79,520 characters extracted from source content.
Expand or collapse full text
AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL Xiaolong Jin 1,2∗ , Dingmin Wang 1† , Vijay Lingam 1 , Varun Kumar 1 1 AWS AI Labs 2 Purdue University Project Page Model Card Abstract Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3points on ALFWorld and+11.0on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines. 1 Introduction Reinforcement learning has become a central approach for post-training multi-turn LLM agents (Shao et al., 2024; Dong et al., 2025; Feng et al., 2025). Unlike in single-turn reasoning, multi-turn agents interact with environments over extended horizons, where each action changes future observations and shapes subsequent decisions. Methods such as GRPO (Shao et al., 2024) train policies from trajectory-level environment rewards without requiring a critic, but assign a uniform advantage to every token in the trajectory. However, not all steps contribute equally to the outcome. Most are routine: the agent acts reasonably and the environment progresses normally. A small number of steps, however, largely determine the outcome; while a wrong object selection, an invalid action, or a misguided search query can compromise the entire episode. Assigning the same gradient signal to both types provides no mechanism to correct the decisions that actually matter. On-policy self-distillation offers denser supervision by augmenting a teacher branch with privileged information (PI) and comparing its token-level predictions against the unaugmented student (Zhao et al., 2026; Yang et al., 2026; Lu et al., 2026b), producing a log-probability gap that serves as a token-level credit-assignment signal. The informativeness of this signal, however, depends entirely ∗ Work done during an internship at Amazon. † Corresponding author. Preprint. arXiv:2608.24114v1 [cs.AI] 25 Aug 2026 Existing Uniform PI Task-level PI same signal at every step Trajectory t₁ Normal t₂ Normal t₃ Error t₄ Normal t₅ Error t₆ Normal Failure signal only no correction Failure signal only no correction Ours Step-Aware PI Env Feedback what happened Corrective Hint what should happen errors only Trajectory t₁ Normal t₂ Normal t₃ Error t₄ Normal t₅ Error t₆ Normal Hint: go to t Hint: use the knife Normal step Error step Figure 1: Existing self-distillation methods apply a single source of privileged information (PI) uniformly across all steps (top): on error steps such ast 3 andt 5 , environment feedback (e.g., “Nothing happens”) only signals failure and provides no corrective direction. AHEAD (bottom) uses environment feedback as base PI on all steps, and adaptively provides an LLM-generated corrective hint on error steps, telling the agent what it should have done. on what PI the teacher receives. Existing methods use task-level PI, such as retrieved skills (Lu et al., 2026b; Wang et al., 2026a) or reference answers (Zhao et al., 2026), that provides the same guidance to every step in the trajectory. This ignores a key asymmetry: routine steps need only confirmation that the action was appropriate, while error steps need specific corrective direction telling the agent what it should have done instead. Task-level PI, which summarizes general workflows rather than diagnosing individual decisions, cannot provide such step-specific corrective information. We observe that multi-turn environments naturally produce a step-specific signal that current methods overlook: environment feedback. After each action, the environment returns an observation that is unavailable to the student when generating its response, making it a natural source of PI. For routine steps, this feedback is sufficient: signals such as “You arrived at shelf 2” confirm the action’s outcome, and the teacher’s assessment shifts only mildly. For error steps, environment feedback is necessary but insufficient: signals such as “Nothing happens” indicate failure but reveal neither the cause nor the corrective action. LLM-generated corrective hints (e.g., “Go to diningtable 1 to find the target object”) supply the missing piece: what the agent should have done at this specific step. Together, the two sources produce naturally adaptive supervision: weak confirmatory signals on routine steps and strong corrective signals on error steps, without any explicit gating or per-step coefficient. Based on this observation, we propose AHEAD (Adaptive Hindsight with Environment-Augmented Distillation) for multi-turn agentic RL. For each failed trajectory, AHEAD injects environment feedback into the teacher context at every step, and additionally provides LLM-generated corrective hints at error steps identified by an LLM analyzer. Successful trajectories, where the GRPO advantage already provides the correct gradient direction, bypass the PI pipeline and retain the vanilla advantage. We use the token-level distillation signal to reweight the GRPO advantage following Yang et al. (2026), which requires minimal changes to the standard algorithm. At inference time, no PI, LLM calls, or environment feedback injection are needed. We validate AHEAD across the Qwen2.5 (Yang et al., 2024) and Qwen3 (Yang et al., 2025) model fam- ilies on three benchmarks for LLM-based agents: ALFWorld (Shridhar et al., 2020), WebShop (Yao et al., 2022), and Search-based QA (Jin et al., 2025). AHEAD achieves substantial improvements over GRPO (+13.3points on ALFWorld and+11.0on WebShop-Succ at 7B) and consistently outperforms self-distillation baselines such as SDAR (Lu et al., 2026b), Skill-SD (Wang et al., 2026a), and RLSD (Yang et al., 2026). 2 Agentic RL Rollouts Task prompt K-turn interaction Observation 烙 Response ⋮ Observation Group of trajectories Per-trajectory episode advantage broadcast to all tokens in Success trajectories Successful trajectory ✓ Token advantage Success Failure Failed trajectories: AHEAD Stage 1 Failed-Trajectory Analysis Failed trajectory ★ ✕ 烙 LLM Analyzer locate errors & generate hints • Error steps • Corrective hints Stage 2 Step-Aware PI Construction ★ ★ ... Env. feedback Corrective hint Teacher PI ENV − ENV ENV − ENV ENV HINT ENV +HINT ENV − ENV ENV HINT ENV +HINT ENV − ENV normal step → ENV only ★ error step → ENV + HINT Stage 3 Step-Aware Self-Distillation Contexts for the same response Original context(student) PI-augmented context(teacher) Token-level distillation signal +0.12−0.12+0.15−0.05 ... ◀▶ more0more Reweighting decays during training Final failed-trajectory token advantage Final Token Advantage Success Failed GRPO Objective normal step ★ error step ENV environment feedback HINT corrective hint ENV+HINT combined information ✓ Success ✕ Failure : stop gradient Figure 2: Overview of AHEAD . Stage 1 identifies error steps in failed trajectories via an LLM analyzer. Stage 2 constructs step-aware privileged information: environment feedback for routine steps, environment feedback combined with an LLM corrective hint for error steps. Stage 3 computes a token-level self-distillation signalδ t,ℓ by comparing the policy’s log-probabilities under original and PI-augmented contexts, and converts it into a bounded reweight of the GRPO advantage. Successful trajectories bypass the PI pipeline entirely and retain the vanilla GRPO advantage. Our contributions are: • We identify that routine and error steps in multi-turn trajectories require fundamentally different supervision, and that environment feedback combined with LLM corrective hints naturally provides on-policy, step-aware PI. •We propose AHEAD, which constructs step-appropriate PI and selectively applies it to failed trajectories, integrating into GRPO with minimal changes. • We validate AHEAD on three agentic benchmarks across three model scales, showing consistent improvements over GRPO and self-distillation baselines. 2 Method We present AHEAD (Adaptive Hindsight with Environment-Augmented Distillation), a framework that constructs step-aware privileged information for multi-turn agent self-distillation and integrates the resulting token-level signal into the GRPO objective. Figure 2 illustrates the overall pipeline. 2.1 Preliminaries Problem Setting.We consider a multi-turn setting, in which an agent interacts with an environment over a finite horizon. At stept, the agent receives an observationo t and maintains an interaction historyh t = (o 0 ,y 0 ,o 1 ,y 1 ,...,o t ), wherey i denotes the response generated at stepi. The policy π θ generates the next response asy t ∼ π θ (·| h t ). A completed trajectory isτ =(o t ,y t ) T−1 t=0 , with terminal outcome reward R(τ ). GRPO. For each task promptq, GRPO samplesNtrajectoriesG q = τ (1) ,...,τ (N) and com- putes a group-relative advantage: A ep = R(τ )− μ q σ q ,(1) whereμ q andσ q are the group mean and standard deviation. This scalar is broadcast to every token in the trajectory. The policy is optimized with the clipped surrogate: L GRPO (θ) =−E τ,t,ℓ [min(ρ t,ℓ A ep , ˆρ t,ℓ A ep )],(2) 3 whereρ t,ℓ = π θ (y t,ℓ | h t ,y t,<ℓ )/π θ old (y t,ℓ | h t ,y t,<ℓ )is the token-level importance ratio and ˆρ t,ℓ = clip(ρ t,ℓ , 1−ε, 1+ε)is its clipped counterpart. SinceA ep is identical for every token, GRPO provides no mechanism to distinguish which steps or tokens were responsible for the outcome. On-Policy Self-Distillation. On-policy self-distillation (Yang et al., 2026) uses the same model as both teacher and student: the student scores each token under the original contexth t , while the teacher receives privileged information (PI) unavailable at decision time. The log-probability gap δ t,ℓ = logπ θ old (y t,ℓ | ̃ h t ,y t,<ℓ )− logπ θ old (y t,ℓ | h t ,y t,<ℓ )(3) measures how the PI revises the model’s assessment of each sampled token. Ifδ t,ℓ > 0, the PI- augmented teacher assigns higher probability to the token than the student, endorsing the student’s choice; ifδ t,ℓ < 0, the teacher disfavors it, suggesting the token should be suppressed. The informativeness ofδdepends entirely on what PI is injected into ̃ h t , which is the focus of our method. 2.2 Step-Aware Privileged Information AHEAD constructs different PI for routine and error steps. We first describe how error steps are identified, then how the two PI sources are combined into a step-aware teacher context. Error Step Identification. After a trajectory completes with a failure outcome, the full trajectory record (observations, actions, environment feedback, and terminal outcome) is passed to an LLM- based analyzer. The analyzer identifies steps whose actions were critical errors (e.g., picking up the wrong object, navigating to an irrelevant location). We denote the set of identified error steps asE τ . Environment Feedback as PI.After the agent generate an action at stept, the environment returns an observationo t+1 . This feedback is not available to the student when generatingy t , making it a natural source of PI. For routine steps, it confirms that the action was appropriate (e.g., “You pick up the mug from shelf 2”). For error steps, it signals that something went wrong (e.g., “Nothing happens”). Environment feedback is grounded and local, reflecting the actual consequence of the agent’s specific action. However, it only describes what happened, not what should have happened. LLM Corrective Hints as PI.For each error stept∈E τ , the LLM analyzer generates a corrective hint describing what the agent should have done instead. For example, if the agent attempted to pick up an object from a wrong location, the hint might state: “You should first go to diningtable 1 to find the target object.” While environment feedback can only signal that an action failed, the corrective hint provides the missing direction: why the action was wrong and what the alternative should be. PI Construction. We construct the PI-augmented context ̃ h t by injecting the appropriate PI: ̃ h t = ( H(h t , Φ env t , Φ llm t ) t∈E τ H(h t , Φ env t )t /∈E τ (4) whereH(·)appends PI to the history,Φ env t is the environment feedback, andΦ llm t is the corrective hint. Error steps receive both sources simultaneously: the feedback identifies the failure, and the hint supplies the correction. Because the teacher sees richer PI on error steps, the resulting|δ t,ℓ |is naturally larger than on routine steps, producing stronger token-level signals on error steps without any explicit gating or per-step coefficient. Note that the LLM analyzer is invoked only on failed trajectories, while environment feedback requires no additional computation and covers all steps. 2.3 Selective Trajectory Filtering Not all trajectories benefit equally from PI-based supervision. Failed trajectories carry a uniform negative advantage but lack information about which steps were responsible. This is precisely the gap that the self-distillation signalδcan fill. Successful trajectories already carry positive advantages that provide the correct gradient direction; injecting PI risks introducing noise when the feedback is not aligned with the tokens that led to success. 4 AHEAD therefore applies PI-based reweighting only to failed trajectories. LetM τ denote the set of steps that participate in reweighting: M τ = ( t : Φ t ̸=∅ τ failed ∅τ succeeded (5) whereΦ t denotes the PI available at stept. For steps outsideM τ , the reweighted advantage in Sec. 2.4 reduces to A ep . 2.4 Advantage Reweighting Following RLSD (Yang et al., 2026), we convert the token-level self-distillation signal into a mul- tiplicative weight on the GRPO advantage, rather than using it as a separate distillation loss. This ensures that the environment reward determines whether the policy is reinforced or penalized, while the distillation signal only adjusts how strongly each token is updated. Specifically, we exponentiate the gap, gated by the sign of the episode advantage: w t,ℓ = clip exp sgn(A ep ) sg(δ t,ℓ ) , 1−ε, 1+ε (6) ̃ A t,ℓ = A ep · (1− λ k ) + λ k · w t,ℓ (7) wheresgdenotes stop-gradient andε = 0.2. Since AHEAD applies reweighting only to failed trajectories (Sec. 2.3), whereA ep < 0, the effect is straightforward: tokens disfavored by the teacher (δ < 0) receive larger penalties, while tokens endorsed by the teacher (δ > 0) receive smaller penalties. Becauseexp(·) > 0and clipping preserves positivity, the reweighted advantage always preserves sign( ̃ A) = sign(A ep ), so the environment reward always controls the update direction. The mixing coefficientλ k decays linearly fromλ 0 = 0.5to0overDtraining steps. Early training benefits from dense PI-based credit assignment, while later training returns to vanilla GRPO to avoid over-reliance on privileged information. Objective. The final objective replaces A ep with ̃ A t,ℓ in the GRPO clipped surrogate: L(θ) =−E τ,t,ℓ h min ρ t,ℓ ̃ A t,ℓ , ˆρ t,ℓ ̃ A t,ℓ i .(8) Training–Inference Boundary. The LLM analyzer, corrective hints, and PI-augmented scoring are used only during training to construct the advantage. At inference time, the policy acts fromh t alone, without any PI, LLM calls, or environment feedback injection. 3 Experiments 3.1 Experimental Setting Benchmarks.We conduct experiments on three agentic benchmarks that cover embodied reasoning, web navigation, and search-augmented question answering. ALFWorld (Shridhar et al., 2020) is a text-based household environment where an agent must complete language-specified goals via sequential textual actions. It includes six task categories: Pick, Look, Clean, Heat, Cool, and Pick2. We use the training split from GiGPO (Feng et al., 2025). WebShop (Yao et al., 2022) simulates an e-commerce website where an agent searches for and pur- chases products matching natural-language specifications. We evaluate on 128 fixed tasks following prior work (Feng et al., 2025), reporting both task-completion score and binary success rate. Search-based QA (Jin et al., 2025) requires an agent to answer questions by issuing search queries and reading returned documents. The benchmark spans single-hop (NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023)) and multi-hop (HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), Bamboogle (Press et al., 2023)) datasets. Following Search-R1 (Jin et al., 2025), we train on NQ and HotpotQA; the remaining datasets serve as out-of-domain evaluation. Full dataset and metric details are given in Appendix B. 5 ALFWorldSearch-based QAWebShop MethodPickLookCleanHeatCoolPick2AvgNQTrivPopHotp2WkMuSBamAvgScoreSucc. Qwen2.5-3B-Instruct Vanilla44.411.16.215.428.612.521.924.648.131.026.325.37.259.731.76.70.8 Skill-Prompt*51.766.748.40.04.310.028.923.746.230.624.422.17.512.523.90.20.8 OPSD48.841.716.70.015.816.728.10.10.10.10.00.00.00.00.011.33.1 GRPO91.262.596.261.965.047.475.039.360.641.137.434.615.426.436.479.863.3 Skill-GRPO88.971.458.870.640.729.260.243.558.843.036.832.211.712.534.177.360.9 Skill-GRPO*94.357.1100.066.773.157.180.544.359.644.339.036.114.514.936.176.366.4 GRPO+OPSD100.082.485.775.070.060.081.244.961.245.240.438.516.066.144.677.866.4 Skill-SD88.250.096.252.465.057.973.444.460.444.039.540.415.464.944.175.964.0 RLSD87.975.090.975.073.168.479.741.558.642.340.440.216.866.943.884.466.4 SDAR97.162.5100.061.975.084.284.444.858.144.338.636.215.766.143.485.068.0 AHEAD97.154.596.790.972.789.587.546.161.249.240.341.712.759.244.388.573.4 Qwen2.5-7B-Instruct Vanilla36.122.23.10.00.00.012.525.250.829.529.029.010.463.733.95.91.6 Skill-Prompt*51.750.032.35.34.30.023.430.952.132.732.727.912.766.136.41.70.8 OPSD50.060.022.721.417.69.532.88.88.617.52.54.20.51.26.24.52.3 GRPO91.287.596.281.065.057.981.245.163.744.043.643.216.837.642.080.972.6 Skill-GRPO88.566.765.261.157.773.169.545.263.745.743.143.319.621.440.380.471.9 Skill-GRPO*100.083.396.483.375.078.988.344.863.045.143.743.720.571.447.587.081.2 GRPO+OPSD91.461.5100.087.576.552.280.447.364.546.943.839.318.069.447.086.876.5 Skill-SD93.993.890.9100.069.268.485.147.164.547.844.242.120.269.047.886.176.5 RLSD100.087.592.358.880.065.282.046.863.044.445.548.921.573.049.087.477.3 SDAR94.775.0100.086.768.278.985.946.363.548.243.848.419.673.049.089.482.8 AHEAD96.781.8100.0100.089.395.094.547.164.748.345.544.719.369.848.589.983.6 Qwen3-1.7B-Instruct Vanilla25.022.23.10.021.44.212.529.446.937.023.519.66.410.524.846.54.7 Skill-Prompt*10.350.016.10.00.05.09.429.446.536.222.920.84.310.124.323.02.3 OPSD26.333.39.10.04.55.314.14.28.34.66.615.30.71.25.847.49.3 GRPO71.141.736.440.031.831.646.140.058.943.535.430.312.065.740.867.338.3 Skill-GRPO27.654.522.727.30.019.221.139.258.643.935.228.211.566.140.473.446.1 Skill-GRPO*31.442.951.98.311.57.128.138.058.443.936.329.012.566.940.780.450.0 GRPO+OPSD38.250.030.828.630.021.132.040.758.945.037.034.613.365.742.270.738.3 Skill-SD52.937.569.242.960.036.852.339.157.545.434.834.110.764.140.881.853.9 RLSD50.037.561.519.050.021.142.238.657.343.034.534.111.565.340.674.050.8 SDAR73.525.076.933.340.036.853.939.758.945.335.935.512.665.341.976.858.6 AHEAD73.363.663.652.971.470.067.243.359.048.239.036.810.956.041.979.364.8 Table 1: Performance Comparison on long-horizon benchmarks. We report the success rate (%) on ALFWorld, accuracy on search-based QA, and task-completion score/success rate on WebShop. An asterisk (*) denotes validation with skills. Within each backbone block, the best result in a column is in bold and the second-bestis underlined. Baseline numbers are taken from Lu et al. (2026b). Baselines. We compare against three categories of methods. (1) Training-free: Vanilla uses the base instruction-tuned model without post-training. Skill-Prompt* retrieves a task-relevant skill and prepends it to the prompt at inference time. (2) RL methods: GRPO (Shao et al., 2024) optimizes the policy with group-relative trajectory-level advantages. Skill-GRPO augments GRPO by injecting retrieved skills into training prompts; Skill-GRPO* additionally retains skills at validation time. (3) Hybrid methods combine RL with self-distillation or skill-conditioned supervision. OPSD (Zhao et al., 2026) distills token-level knowledge from a frozen reference policy. GRPO+OPSD adds OPSD as an auxiliary loss on top of GRPO. Skill-SD (Wang et al., 2026a) uses importance-weighted distillation with retrieved skills as privileged context. RLSD (Yang et al., 2026) re-weights GRPO advantages using the teacher-student log-probability gap. SDAR (Lu et al., 2026b) applies a sigmoid-gated auxiliary distillation loss that selectively distills teacher-endorsed tokens. Implementation Details. We use Qwen2.5-3B/7B-Instruct (Yang et al., 2024) and Qwen3-1.7B- Instruct (Yang et al., 2025) as backbone models. All models are trained for 150 steps on 8 H100 GPUs. For ALFWorld and WebShop, each batch samples 16 tasks with 8 rollouts per prompt. For Search-based QA, the batch size is 128 tasks. The maximum prompt length is 2,048 for ALFWorld and 4,096 for WebShop and Search-based QA. We use a GRPO clipping range ofε = 0.2and a reweight bound ofε = 0.2; the distillation coefficient decays linearly fromλ 0 = 0.5to0over D = 50training steps (Sec. 2.4). Error detection and corrective hint generation use Claude Opus 4.7 as the LLM analyzer. Full hyperparameters are provided in Appendix F. Evaluation Protocol.We report ALFWorld results on Val-128: a fixed 128-task set sampled from the seen split, following Lu et al. (2026b). We evaluate generalization to unseen room layouts on the Unseen-134 split. Results on the full Seen-140 and Unseen-134 splits are provided in Appendix E. 6 050100150 Training Step 0 25 50 75 100 Success Rate (%) Qwen3-1.7B 43% fewer steps 050100150 Training Step Qwen2.5-7B 57% fewer steps GRPO AHEAD Figure 3: Sample efficiency on ALFWorld, mea- sured on the seen split during training, for Qwen3- 1.7B (left) and Qwen2.5-7B (right). AHEAD con- verges faster and reaches a higher final success rate than GRPO at both scales; arrows mark the training-step saving to reach GRPO’s final score. 01020304050 Step Budget N 0 20 40 60 80 100 Solved (%) Qwen3-1.7B 01020304050 Step Budget N Qwen2.5-7B GRPO AHEAD Seen Unseen Figure 4: Fraction of ALFWorld tasks solved within N steps for Qwen3-1.7B (left) and Qwen2.5-7B (right). Solid and dashed lines de- note seen and unseen splits, respectively. Beyond the first few steps (N ≥ 5), AHEAD consistently outperforms GRPO across both splits and scales. 3.2 Main Results Overall Performance.AHEAD achieves the best or second-best result in most columns in Table 1. Compared to GRPO, it delivers consistent gains on ALFWorld (+12.5points on 3B,+13.3on 7B, +21.1on 1.7B) and WebShop-Succ (+10.1,+11.0and+26.5points respectively), and raises Search- Avg at every scale (+7.9,+6.5and+1.1). On the two interactive benchmarks the improvements are most pronounced on the smaller Qwen3-1.7B. This is expected: smaller models make more errors per trajectory, so there are more error steps where corrective hints can provide useful supervision. Comparison with Self-Distillation Baselines. On ALFWorld, AHEAD matches or outperforms all self-distillation baselines across every model scale. At 7B, it reaches 94.5%, surpassing SDAR (85.9%), Skill-SD (85.1%), and RLSD (82.0%) by large margins. At 1.7B, where smaller models struggle to utilize retrieved skills effectively, AHEAD achieves 67.2% while SDAR reaches 53.9% and RLSD only 42.2%. On WebShop, AHEAD leads across all three model scales, reaching 83.6% Succ at 7B against SDAR’s 82.8%. Stability. Standalone OPSD collapses catastrophically (near-zero on Search-QA), and the naive GRPO+OPSD combination degrades severely on Qwen3-1.7B (32.0% vs. 46.1% for GRPO on ALFWorld) due to unbounded distillation gradients overwhelming the RL signal. AHEAD avoids these instabilities entirely, because the reweighting (Sec. 2.4) can only adjust how strongly each token’s advantage is applied, never its sign. 3.3 Analysis Sample Efficiency. As shown in Figure 3, AHEAD converges substantially faster than GRPO. It reaches GRPO’s final performance level early in training and continues to improve to94.5against GRPO’s81.2. The same pattern holds at 1.7B, where AHEAD ends21.1points above GRPO (67.2 vs.46.1). Step-aware PI therefore does not merely raise the final score; it reaches any given score sooner, because the corrective hints tell the policy what it should have done at each error step rather than leaving it to infer this from trajectory-level rewards alone. Step Efficiency at Test Time. The efficiency gains transfer to test time (Figure 4). Beyond the first few steps (N ≥ 5), AHEAD’s curve lies above GRPO’s on both the seen and unseen splits of ALFWorld, so the improvement is not an artifact of a generous interaction limit. The gap is largest at practical step budgets: on the unseen split, AHEAD solves 74.6% of tasks within 20 steps where GRPO solves 53.0%, and AHEAD needs only 15 steps to match the success rate GRPO attains with its full 54-step budget. The margin is wider at 1.7B, where AHEAD solves 47.8% of tasks within 20 steps against GRPO’s 23.9% and needs only 12 steps to reach GRPO’s full-budget coverage. The gains therefore come from eliminating unnecessary steps, rather than needing more room to explore. In deployment, this usually means lower latency and fewer API calls, not merely a higher score. Ablation Study. Table 2 removes one component at a time; the full method leads nearly every column. Without the linear decay ofλ k , performance drops by7.0–16.4points on ALFWorld and 7 ComponentsALFWorldWebShop ConfigurationEnv. fb. Hints Multi Fail-only Decay 1.7B3B7B1.7B3B7B AHEAD✓67.287.594.564.873.483.6 w/o decay✓ ×60.271.1 85.254.766.4 83.6 w/o failure-only✓ ×✓48.482.0 93.064.172.7 75.8 w/o multi-step✓ ×✓61.781.2 86.757.047.7 75.0 w/o hints✓ ×–✓53.173.4 87.563.368.0 74.2 w/o env. feedback ×✓59.481.2 89.867.272.7 72.7 Env. feedback only✓ ×– ×✓56.272.7 88.360.271.9 78.9 Table 2: Component ablation. Success rate (%) on ALFWorld and WebShop. Ticks give the configuration of each run, so rows differing in a single tick are one-component comparisons. Env. fb.: environment feedback as base PI. Hints: the LLM-generated corrective hintΦ llm on error steps. Multi: the analyzer may mark several error steps per trajectory rather than one. Fail-only: PI is applied to failed trajectories only (Sec. 2.3). Decay: the linear schedule onλ k . The last row removes both the hints and the failure-only filter, leaving environment feedback applied uniformly. up to10.1on WebShop, confirming that PI-based reweighting must fade as training progresses. Applying PI to all trajectories instead of only failed ones hurts most at 1.7B (−18.8on ALFWorld), where PI on a successful trajectory pulls against an already correct gradient. Restricting the analyzer to a single error step per trajectory costs up to25.7points (WebShop-3B), since failed episodes rarely contain just one mistake. Both PI sources contribute: removing corrective hints costs7.0–14.1 points on ALFWorld, confirming that a failure signal alone is not enough without corrective direction; removing environment feedback incurs smaller but consistent losses on ALFWorld (4.7–7.8points), though the relative importance of the two sources varies across benchmarks. Analyzer3B1.7B Opus 4.7 (default)87.567.2 Sonnet 589.168.0 Kimi83.668.0 GLM-581.267.2 GRPO (no analyzer)75.046.1 Table 3: ALFWorld success rate (%) when the error-step analyzer is swapped, holding everything else fixed. GRPO, which uses no analyzer, is repeated from Table 1 as a floor. Choice of LLM Analyzer.AHEAD depends on an external LLM to select error steps and write correc- tive hints, so a natural question is how much of the gain is attributed to the analyzer. Table 3 swaps it for three alternatives while holding the rest of the pipeline fixed. Two observations follow. First, every analyzer beats GRPO by a wide margin (the weakest, GLM-5, still scores 81.2 at 3B against GRPO’s 75.0, and 67.2 at 1.7B against 46.1), so the gain is not con- tingent on one particular model. Second, the spread across analyzers is scale-dependent: at 3B the four span 7.9 points (81.2–89.1), while at 1.7B they fall within 0.8 points of each other (67.2–68.0). At the smaller scale the policy appears unable to exploit the difference between a better and a worse hint, which puts a ceiling on what a stronger analyzer can buy: the bottleneck there is the policy’s capacity to act on corrective information, not the quality of that information. A cheaper analyzer therefore suffices for smaller backbones. Qualitative Trajectory Comparison. Figure 5 contrasts a GRPO-trained policy with a AHEAD- trained policy on the same ALFWorld task, put a hot plate in cabinet. Both start from identical observations, but the two trajectories diverge immediately at the first plate-bearing countertop. The GRPO agent repeatedly overlooks the plates in front of it, cycles through the same countertops and empty cabinets, and never issues a single take, heat, or move action before hitting the 50-step limit, so the episode fails. The AHEAD agent instead grounds each decision in the current observation: it picks up the plate it sees, heats it in the microwave, and places it in the cabinet, completing the full find →pick→heat→place sequence in nine steps. This is the failure mode that step-aware corrective hints target during training, an early perception error that GRPO compounds into an unrecoverable loop, and it is consistent with the shorter, more successful trajectories AHEAD produces (Sec. E). The reasoning and observation text shown is lightly condensed for readability; “...” marks omitted spans, and the retained excerpts are verbatim. 8 panel boundary Trajectory of model trained with GRPO Task: put a hot plate in cabinet. Reasoning excerpts are verbatim; “...” marks omissions. panel boundary Trajectory of model trained by AHEAD Task: put a hot plate in cabinet. Reasoning excerpts are verbatim; “...” marks omissions. [Steps 0–1] Observation: Countertop 1 contains no plate. Reasoning: To find a hot plate to put in a cabinet, I should first look for a hot plate in the most likely places. The most likely places ... are usually the countertops, the stove, and the cabinets. I will start by checking the countertops ... Action: go to countertop 1 → go to countertop 2Search prior [Step 2] Observation: On countertop 2, the agent sees bread 1, cup 1, plate 2, spatula 1, tomato 2, ... Reasoning: I need to find a hot plate to put in a cabinet. Based on the current observation, I am at countertop 2, which does not contain any plates. ... I should check other countertops or cabinets to see if a plate is located there. Action: go to countertop 3 Plate 2 overlooked [Step 3] Observation: On countertop 3, the agent sees plate 3, plate 1, spatula 2, spoon 2, spoon 1, ... Reasoning: I need to find a hot plate to put in a cabinet. Based on the current observation, I am at countertop 3, which does not contain any plates. ... I should check the remaining countertops to see if a plate is located there. Action: go to countertop 4 Plates overlooked [Steps 4–6] Observation: Each attempt to visit countertop 4, 5, or 6 returns “Nothing happens.” Reasoning: Based on the current observation, nothing is happening ... The admissible actions include examining countertop 3, which might contain a hot plate ... The most logical next step is to examine countertop 3. Action: countertop 5 → countertop 6 → examine countertop 3 Invalid locations [Steps 7–49] The trajectory repeatedly revisits the same plate-bearing countertops, then cycles through empty cabinets. No take, heat, or move action occurs before the 50-step limit. Failed trajectory [Steps 0–1] Observation: Countertop 1 contains no plate. Reasoning: To complete the task of putting a hot plate in the cabinet, I need to first find a plate. The most likely place to find a plate is the countertop, as it is a common place to put food and dishes. ... [Step 1:] The most logical place to find a plate is the countertop 2 ... Action: go to countertop 1 → go to countertop 2 Search prior [Steps 2–3] Observation: On countertop 2, the agent sees bread 1, cup 1, plate 2, spatula 1, tomato 2, ... Reasoning: I have already gone to the countertop 2 and seen a plate 2. The task is to put a hot plate in the cabinet, so I should take this plate and go to the cabinet to heat it up. The most logical choice is to take plate 2 ... and go to microwave 1 to heat it up. Action: take plate 2 from countertop 2 → go to microwave 1 Grounded pickup [Steps 4–5] Observation: Microwave 1 is closed; after opening it, the microwave is empty. Reasoning: The microwave is closed, so I cannot heat the plate directly. The most logical action is to open the microwave ... [After opening:] The next logical step is to heat the plate using the microwave. Action: open microwave 1 → heat plate 2 with microwave 1 Heat plate [Steps 6–8] Observation: The plate is hot; cabinet 1 is closed, then opened and found empty. Reasoning: I have already heated plate 2 using the microwave 1. The next logical step is to put it in the cabinet. ... [Step 8:] I have already heated plate 2 and am now at cabinet 1. The cabinet is open and empty, so I can place the plate inside. Action: go to cabinet 1 → open cabinet 1 → move plate 2 to cabinet 1 Complete placement [Outcome] The agent completes the full find → pick → heat → place sequence in 9 steps. Successful trajectory Figure 5: Trajectory comparison on an ALFWorld task put a hot plate in cabinet. Left: a GRPO- trained policy overlooks the visible plates, revisits the same countertops and empty cabinets, and never takes, heats, or moves an object before the 50-step limit (failed trajectory). Right: a AHEAD-trained policy grounds each action in the current observation and completes the find→pick→heat→place sequence in nine steps. Reasoning and observation excerpts are lightly condensed for space; “...” marks omissions and retained text is verbatim. 4 Related Work Reinforcement Learning for LLM Agents.Reinforcement learning is now widely used for post- training language model agents in interactive environments (Shao et al., 2024; Dong et al., 2025; Feng et al., 2025). Agents trained with RL interact with environments over many steps, making sequential decisions in settings such as embodied reasoning (Shridhar et al., 2020), web navigation (Yao et al., 2022), and GUI automation (Lu et al., 2025, 2026a). A central difficulty in these settings is credit assignment: outcome rewards indicate whether an episode succeeded but provide no information about which intermediate decisions were responsible (Deng et al., 2025). On-Policy Self-Distillation for Agents.On-policy self-distillation provides an alternative source of dense supervision by letting the same model serve as both student and teacher under different contexts (Agarwal et al., 2024; Zhao et al., 2026; He et al., 2026). Several recent methods combine this idea with RL for multi-turn agents. SDAR (Lu et al., 2026b) uses a sigmoid gate on the teacher-student log-probability gap to selectively distill teacher-endorsed tokens while attenuating noisy signals. RLSD (Yang et al., 2026) re-weights GRPO advantages using the same gap without a separate distillation loss. Skill-SD (Wang et al., 2026a) augments the teacher with retrieved natural-language skills and applies importance-weighted distillation. These methods apply privileged information uniformly across steps or select among different granularities of the same PI source. Our work differs by using two heterogeneous PI sources and allocating them based on step type. Privileged Information in Agent Training.Using PI during training while removing it at inference has roots in the learning-by-cheating paradigm (Chen et al., 2019). In LLM agent training, this idea appears as skill-conditioned learning, where natural-language skills are provided during training but removed at test time (Lu et al., 2026c; Xia et al., 2026; Shi et al., 2026). These approaches typically use a single type of privileged information applied uniformly across steps. Our work extends this line by combining multiple PI sources with different costs and informativeness, and allocating them based on step-level supervision needs. More related work are discussed in Appendix A. 5 Conclusion We presented AHEAD, which constructs step-aware privileged information for multi-turn agent self- distillation. By combining environment feedback on all steps with LLM-generated corrective hints on identified error steps, AHEAD produces adaptive token-level supervision: weak confirmatory signals on routine steps and strong corrective signals where they matter most. Across three benchmarks and three model scales, AHEAD improves on both pure RL and self-distillation baselines. 9 Limitations Cost of the LLM analyzer.AHEAD calls an external LLM on failed trajectories to identify error steps and write corrective hints. This adds an inference cost to training that GRPO does not pay, and it makes the method’s behaviour depend on the analyzer’s quality. Table 3 shows that the gain survives swapping the analyzer for three weaker models, but all four are large proprietary or frontier-scale systems; whether a small open model can play the role is untested. The cost is confined to training. Inference uses no privileged information, no LLM calls, and no environment-feedback injection. Dependence on error step identifiability. The effectiveness of AHEAD’s corrective hints relies on the LLM analyzer’s ability to correctly identify which steps were critical errors. Our evaluation focuses on environments with relatively discrete, identifiable mistakes such as picking up the wrong object, navigating to an irrelevant location, or issuing an unproductive search query. In environments where errors are subtle, cumulative, or arise from omissions rather than overt wrong actions, the analyzer may fail to pinpoint the true decision points, reducing the quality of step-aware PI. How well the approach transfers to such settings remains an open question. Scope of evaluation.Our experiments cover three benchmarks spanning embodied reasoning, web navigation, and search-augmented QA. While these represent diverse agent interaction patterns, they share relatively short trajectories with clear binary success/failure outcomes. Environments with longer horizons, continuous action spaces, partial observability, or soft reward signals may pose additional challenges for both error detection and PI construction. References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations (ICLR), 2024. URLhttps: //arxiv.org/abs/2306.13649. Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by Cheating. In Conference on Robot Learning (CoRL), 2019. URL https://arxiv.org/abs/1912.12294. Zeyun Deng, Jasorsi Ghosh, Fiona Xie, Yuzhe Lu, Katia Sycara, and Joseph Campbell. Energy-based transfer for reinforcement learning, 2025. URL https://arxiv.org/abs/2506.16590. Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic Reinforced Policy Optimization. arXiv preprint arXiv:2507.19849, 2025. URL https://arxiv.org/abs/2507.19849. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. arXiv preprint arXiv:2505.10978, 2025. URLhttps://arxiv.org/abs/2505. 10978. Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision. arXiv preprint arXiv:2604.12002, 2026. URL https://arxiv.org/abs/2604.12002. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a Multi- hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Proceedings of the 28th International Conference on Computational Linguistics (COLING), p. 6609–6625, 2020. doi: 10.18653/v1/2020.coling-main.580. URL https://arxiv.org/abs/2011.01060. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv preprint arXiv:2503.09516, 2025. URLhttps://arxiv.org/ abs/2503.09516. 10 Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), p. 1601–1611, 2017. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147/. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics (TACL), 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026/. Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, and Yueting Zhuang. UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning. arXiv preprint arXiv:2509.11543, 2025. URLhttps://arxiv.org/ abs/2509.11543. Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2026a. URL https://arxiv.org/abs/2503.21620. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Self-Distilled Agentic Reinforcement Learning. arXiv preprint arXiv:2605.15155, 2026b. URLhttps://arxiv.org/abs/2605.15155. Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Skill0: In-Context Agentic Reinforcement Learning for Skill Internalization. arXiv preprint arXiv:2604.02268, 2026c. URLhttps://arxiv.org/ abs/2604.02268. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi.When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories.In Proceedings of the 61st Annual Meeting of the Asso- ciation for Computational Linguistics (ACL), Volume 1: Long Papers, p. 9802–9822. As- sociation for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.546. URL https://aclanthology.org/2023.acl-long.546/. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and Narrowing the Compositionality Gap in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 5687–5711. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.378. URLhttps://aclanthology. org/2023.findings-emnlp.378/. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300. Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi Gu, Xunliang Cai, Xiang Wang, and An Zhang. Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning. arXiv preprint arXiv:2605.06130, 2026. URL https://arxiv.org/abs/2605.06130. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In International Conference on Learning Representations (ICLR), 2020. URLhttps://arxiv.org/ abs/2010.03768. Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah, and Dimitris Papailiopoulos. ECHO: Terminal Agents Learn World Models for Free. arXiv preprint arXiv:2605.24517, 2026. URL https://arxiv.org/abs/2605.24517. 11 Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Mul- tihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics (TACL), 10:539–554, 2022. doi: 10.1162/tacl_a_00475. URL https://aclanthology.org/2022.tacl-1.31/. Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, and Honggang Qi. Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. arXiv preprint arXiv:2604.10674, 2026a. URLhttps://arxiv.org/ abs/2604.10674. Zhitong Wang, Songze Li, Hao Peng, Shuzheng Si, Yi Wang, Maosong Sun, and Juanzi Li. En- vRL: Learn from Environment Dynamics in Agentic Reinforcement Learning. arXiv preprint arXiv:2606.17680, 2026b. URL https://arxiv.org/abs/2606.17680. Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. arXiv preprint arXiv:2602.08234, 2026. URL https://arxiv.org/abs/2602.08234. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115, 2024. URL https://arxiv.org/abs/2412.15115. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-Distilled RLVR. arXiv preprint arXiv:2604.03128, 2026. URL https://arxiv.org/abs/2604.03128. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. doi: 10.18653/v1/D18-1259. URLhttps://arxiv.org/abs/1809. 09600. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems 35 (NeurIPS), 2022. URL https://arxiv.org/abs/2207.01206. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv preprint arXiv:2601.18734, 2026. URL https://arxiv.org/abs/2601.18734. 12 Appendix A Related Work Reinforcement Learning for LLM Agents.Reinforcement learning is now widely used for post- training language model agents in interactive environments (Shao et al., 2024; Dong et al., 2025; Feng et al., 2025). Agents trained with RL interact with environments over many steps, making sequential decisions in settings such as embodied reasoning (Shridhar et al., 2020), web navigation (Yao et al., 2022), search-augmented QA (Jin et al., 2025), and GUI automation (Lu et al., 2025, 2026a). A central difficulty in these settings is credit assignment: outcome rewards indicate whether an episode succeeded but provide no information about which intermediate decisions were responsible. Our work addresses this by introducing step-aware dense supervision that complements the coarse trajectory-level signal. Environment Feedback as Supervision. Standard agent RL discards environment observations during optimization, using them only as context for future actions. Recent work has recognized that these observations contain useful supervision signals. ECHO (Shrivastava et al., 2026) trains the policy to predict environment observations as an auxiliary task, learning an implicit world model at no additional rollout cost. EnvRL (Wang et al., 2026b) extends this with separate state prediction and inverse dynamics objectives. These methods treat environment feedback as prediction targets for representation learning. Our work takes a different perspective: we use environment feedback as privileged information for the teacher branch, enabling grounded token-level guidance. We further combine it with LLM-generated corrective hints on error steps, providing corrective direction that environment feedback alone cannot supply. Privileged Information in Agent Training. Using privileged information during training while removing it at inference has roots in the learning-by-cheating paradigm (Chen et al., 2019). In LLM agent training, this idea appears as skill-conditioned learning, where natural-language skills are provided during training but removed at test time (Lu et al., 2026c; Xia et al., 2026; Shi et al., 2026). These approaches typically use a single type of privileged information applied uniformly across steps. Our work extends this line by combining multiple PI sources with different costs and informativeness, and allocating them based on step-level supervision needs. B Dataset and Metric Details Table 4 summarizes the datasets used in our experiments. Below we provide additional details on each benchmark and the corresponding evaluation metrics. Benchmark#Train#Test ALFWorld2,400128 (Val-128) / 140 (Seen-140) / 134 (Unseen-134) WebShop2,400128 NQ, TriviaQA, PopQA, HotpotQA,19,20051,713 2WikiMultiHopQA, MuSiQue, Bamboogle Table 4: Datasets used in our experiments: ALFWorld (embodied reasoning), WebShop (web navigation), and Search-based QA. For Search-based QA we train only on NQ and HotpotQA; the remaining five datasets are held out and never seen during training. ALFWorld. ALFWorld (Shridhar et al., 2020) connects text-based interaction with the ALFRED household environment. The agent receives a natural-language goal and textual observations, and must issue a sequence of valid actions to complete the task. The benchmark covers six task categories: Pick, Look, Clean, Heat, Cool, and Pick2. We sample 2,400 training examples from the GiGPO split (Feng et al., 2025). Evaluation uses three sets: Val-128, a fixed 128-task subset of the seen split that all baselines report on and that we use by default; Seen-140, the full seen split; and Unseen-134, the unseen split, which measures generalisation to room layouts not encountered during training. On 13 each set we report the micro-averaged success rate, in which every task counts equally: ALFWorld-Avg = P 6 c=1 n c SR c P 6 c=1 n c ,(9) wheren c is the number of tasks in categoryc. Because the categories are not equally sized, this differs from the unweighted mean of the six per-category rates. WebShop. WebShop (Yao et al., 2022) is a simulated e-commerce environment where an agent searches for products, navigates product pages, selects attributes, and makes a purchase to satisfy a natural-language request. We use 2,400 training examples and evaluate on 128 fixed tasks following Feng et al. (2025). The environment returns two metrics: a normalized task-completion score that gives partial credit for matching requested attributes, and a binary success indicator for exact task completion. We report Score (the mean normalized score multiplied by 100) and Succ. (the percentage of exactly successful tasks). Search-based QA.Following Search-R1 (Jin et al., 2025), the agent interacts with a search engine to retrieve relevant documents before producing a final answer. The evaluation spans seven datasets: three single-hop (NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023)) and four multi-hop (HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), Bamboogle (Press et al., 2023)). We train on 19,200 examples drawn from NQ and HotpotQA; the remaining five datasets serve as out-of-domain evaluation. We compute answer accuracy on each dataset and report the unweighted macro-average: Search-Avg = 1 7 7 X d=1 Acc d .(10) Unlike ALFWorld, where the six categories partition a single task suite and are therefore pooled, the seven QA datasets are independently constructed benchmarks with test sets of very different sizes; averaging them without weights keeps a large dataset such as TriviaQA from dominating the score. C Baseline Descriptions Following Lu et al. (2026b), we adopt the same set of baselines for a fair comparison. All methods share the same backbone model, environment configuration, rollout budget, and evaluation setup, as well as the number of optimization steps, group size, and learning rate schedule. An asterisk (*) marks methods that retain the retrieved skill in the prompt at validation/test time. C.1 Prompting-Only Methods Vanilla. The base instruction-tuned model is evaluated as-is, with no post-training applied. It receives the environment prompt and interaction history as its only input. Skill-Prompt*. No post-training is applied. Instead, a retrieved task-relevant skill is added to the prompt at validation/test time. Any performance gain therefore comes from the model’s ability to leverage the skill in context. C.2 Outcome-Based Reinforcement Learning GRPO. GRPO (Shao et al., 2024) is a critic-free policy-gradient method. It generates multiple trajectories per task and computes advantages by normalizing their outcome rewards relative to the group. All tokens in a trajectory share the same sequence-level advantage. The policy is optimized with a clipped importance-ratio objective. Skill-GRPO. This method trains with the same GRPO objective but additionally includes a task- relevant skill in the prompt during rollouts. At validation/test time the skill is removed. This evaluates whether the policy has learned from skill-guided exploration well enough to perform without the skill. 14 Skill-GRPO*. Training is the same as Skill-GRPO. The difference is that the skill remains in the prompt during validation/test time, so the model has consistent access to the skill in both training and evaluation. C.3 Self-Distillation and Hybrid Methods OPSD. OPSD (Zhao et al., 2026) is a self-distillation method where the student and teacher are initialized from the same model. The two branches differ in their conditioning: the student sees only the task prompt, while the teacher has access to privileged information (e.g., a ground-truth solution). The teacher provides token-level distributional targets along the student’s own trajectory, and only the student receives gradient updates. No privileged context is used at inference time. Skill-SD. Skill-SD (Wang et al., 2026a) applies self-distillation to multi-turn agent settings. It first distills successful trajectories into natural-language skills that describe effective strategies and common failure modes. During training, these skills serve as privileged context for the teacher, while the student operates with the standard task prompt only. Since the skills are not available at test time, the student must learn to reproduce the teacher’s behavior from its parameters alone. GRPO+OPSD. This baseline adds the OPSD distillation loss as an auxiliary term to the GRPO objective. The outcome-based component operates at the trajectory level, while the distillation com- ponent provides token-level supervision from a frozen reference. This tests whether a straightforward combination of the two signals is effective. RLSD. RLSD (Yang et al., 2026) modifies GRPO by using a privileged self-teacher to assign token-level importance weights. The log-probability gap between teacher and student at each token is mapped to a bounded scalar that adjusts the magnitude of that token’s gradient update, while the sign still follows the outcome-level advantage. The teacher’s contribution is scheduled to decay over training, so that later stages reduce to standard GRPO. SDAR. SDAR (Lu et al., 2026b) augments GRPO with a gated distillation loss as an auxiliary objective. A teacher conditioned on privileged context (e.g., a retrieved skill) evaluates the student’s on-policy tokens and produces token-level signals. These signals pass through a bounded gate that upweights reliable positive guidance and downweights noisy negative ones before entering the loss. The GRPO advantage is not modified; the gating operates solely on the distillation term. D Algorithm The full procedure of AHEAD is presented in Algorithm 1. Compared to standard GRPO (which corresponds to removing lines 9–28 and setting ̃ A t,ℓ = A ep everywhere), AHEAD adds three components: (1) an LLM analyzer call on failed trajectories, (2) a PI-augmented forward pass to computeδ t,ℓ , and (3) a bounded reweight of the advantage. All three are confined to training; at inference time the policy acts from h t alone. E Additional Results The main text reports the analysis of Sec. 3.3 on Qwen2.5-7B. This appendix gives the per-category numbers behind it and repeats each experiment on Qwen2.5-3B and Qwen3-1.7B. The trends observed at 7B carry over to 1.7B, where the gains are larger. At 3B the average gains are smaller, because GRPO already scores 81.4 on Seen-140, and the per-category changes are mixed. E.1 Per-Category Breakdown Table 5 and Table 6 report per-category success rates on Seen-140 and Unseen-134. Figure 6 plots the 7B rows of Table 6 and Figure 7 the 1.7B rows. Unlike Table 1, whose baseline numbers are taken from Lu et al. (2026b) on Val-128, the GRPO rows here come from our own reproduced GRPO checkpoints, since Lu et al. (2026b) does not report per-category or full-split results. GRPO and AHEAD are therefore trained and evaluated in the same codebase, so the two are directly comparable. 15 Algorithm 1 AHEAD Require:Policyπ θ , task setS, group sizeN, clip boundε, reweight boundε, initial mixing coefficientλ 0 , decay horizonD, LLM analyzerA 1: for each training iteration k do 2: λ k ← max(0, λ 0 · (1− k/D))▷ Linear decay 3:Sample a batch of tasksq fromS 4:for each task q do 5:// Stage 0: On-policy rollout 6:Sample N trajectoriesτ (1) ,...,τ (N) ∼ π θ (·| q) 7:Compute A ep i = (R(τ (i) )− μ q )/σ q ▷ Group-relative advantage 8:for i = 1,...,N do 9:if R(τ (i) ) indicates failure then 10:// Stage 1: Error Step Detection 11:E τ , Φ llm t t∈E τ ←A(τ (i) )▷ LLM analyzer 12:// Stage 2: Step-Aware PI Construction 13:for each step t in τ (i) do 14:Φ env t ← o t+1 ▷ Environment feedback 15:if t∈E τ then 16: ̃ h t ← H(h t , Φ env t , Φ llm t )▷ Env + Hint 17:else 18: ̃ h t ← H(h t , Φ env t )▷ Env only 19:end if 20:end for 21:// Stage 3: Token-Level Self-Distillation 22:for each step t, token ℓ do 23:δ t,ℓ ← logπ θ old (y t,ℓ | ̃ h t ,y t,<ℓ )− logπ θ old (y t,ℓ | h t ,y t,<ℓ ) 24:w t,ℓ ← clip exp sgn(A ep )· sg(δ t,ℓ ) , 1−ε, 1+ε 25: ̃ A t,ℓ ← A ep · (1− λ k ) + λ k · w t,ℓ 26:end for 27:else 28: ̃ A t,ℓ ← A ep for all t,ℓ▷ Vanilla advantage 29:end if 30:end for 31:// Policy update 32:Update θ by minimizingL(θ) =−E h min ρ t,ℓ ̃ A t,ℓ , ˆρ t,ℓ ̃ A t,ℓ i 33:end for 34: end for The improvement is not uniform across task categories. On Unseen-134 at 7B, AHEAD leaves Clean and Cool untouched, categories that GRPO already solves at83.9%and85.7%, while lifting Look by44.4points and Pick2 by35.3points. These two categories require the longest action sequences and are the ones where a single misstep most often derails the episode, which is where step-aware corrective hints are most valuable. The same ordering holds at 1.7B: the largest gains fall on Pick2 (+47.1) and Look (+44.4, from a GRPO baseline that solves none of the 18 tasks). The gain does not shrink on the unseen split. On Qwen2.5-7B, AHEAD improves the average by +16.4points on Seen-140 (93.6vs.77.1) and by exactly+16.4points on Unseen-134 (83.6vs.67.2). On Qwen3-1.7B the gains are+24.3(66.4vs.42.1) and+28.4(59.0vs.30.6), larger on the unseen split than on the seen one. Qwen2.5-3B shows the same direction more sharply: the average gain is only+1.5on Seen-140 (82.9vs.81.4) but+10.5on Unseen-134 (82.1vs.71.6). If AHEAD were simply overfitting to the training room layouts, its advantage would shrink on unseen rooms. It does not. Since the corrective hints are used only during training and removed at inference time, the improvement comes from what the policy has learned, not from access to privileged information. E.2 Training Dynamics Figure 8 tracks training at 7B and Figure 9 at 1.7B. AHEAD not only achieves higher final scores but also produces different training behavior. Success rate and train reward are consistently higher throughout optimization, with AHEAD reaching a final train reward of 4.40 compared to 3.47 for GRPO. The trajectory length is particularly informative: both methods start at an average of 46.8 steps, but by the end of training AHEAD has driven it down to 23.8 while GRPO only reaches 31.2. The agent thus solves more tasks and does so in fewer 16 MethodPickLookCleanHeatCoolPick2Avg. Qwen2.5-3B-Instruct GRPO91.476.981.562.588.075.081.4 AHEAD97.153.888.987.572.079.282.9 ∆+5.7−23.1+7.4+25.0−16.0+4.2+1.5 Qwen2.5-7B-Instruct GRPO82.961.596.393.880.041.777.1 AHEAD100.084.6100.0100.080.091.793.6 ∆+17.1+23.1+3.7+6.3+0.0+50.0+16.4 Qwen3-1.7B-Instruct GRPO68.615.455.650.020.020.842.1 AHEAD82.953.870.468.868.041.766.4 ∆+14.3+38.5+14.8+18.8+48.0+20.8+24.3 Table 5: Success rate (%) per task category on the ALFWorld Seen-140 split.∆is the absolute gain of AHEAD over GRPO in percentage points, and Avg. is the micro-average over all 140 tasks. MethodPickLookCleanHeatCoolPick2Avg. Qwen2.5-3B-Instruct GRPO70.888.977.465.281.041.271.6 AHEAD83.377.877.487.076.294.182.1 ∆+12.5−11.1+0.0+21.8−4.8+52.9+10.5 Qwen2.5-7B-Instruct GRPO62.538.983.965.285.752.967.2 AHEAD79.283.383.982.685.788.283.6 ∆+16.7+44.4+0.0+17.4+0.0+35.3+16.4 Qwen3-1.7B-Instruct GRPO50.00.029.047.833.311.830.6 AHEAD66.744.451.673.957.158.859.0 ∆+16.7+44.4+22.6+26.1+23.8+47.1+28.4 Table 6: Success rate (%) per task category on the ALFWorld Unseen-134 split.∆is the absolute gain of AHEAD over GRPO in percentage points, and Avg. is the micro-average over all 134 tasks. steps. A policy that merely explored more aggressively would improve reward at the cost of longer episodes; AHEAD improves reward and shortens trajectories, indicating that the corrective hints help the agent avoid the wasted actions that typically follow an error step. This pattern is consistent with the mechanism of step-aware PI: by providing corrective direction at error steps during training, the policy learns to avoid the common failure mode of repeating or escalating an incorrect action, which in standard GRPO often extends trajectories without progress. The same picture holds at 1.7B. Both methods start from an average episode length of47.8steps; by step150AHEAD has driven it down to29.8while GRPO stalls at40.2, and it does so at a higher train reward (3.62vs.1.36). Success rises as trajectory length falls, the same pattern observed at 7B but a sharper one: the 10.4-step separation between the two methods at 1.7B exceeds the 7.4 steps at 7B. PickLookCleanHeatCoolPick2Avg 0 20 40 60 80 100 Success Rate (%) 62 39 84 65 86 53 67 79 83 84 83 86 88 84 GRPOAHEAD Figure 6: Per-task success rate on the ALF- World Unseen-134 split (Qwen2.5-7B-Instruct). AHEAD raises the average from67.2to83.6. The gains concentrate on the categories where GRPO is weakest, Look (38.9→ 83.3) and Pick2 (52.9 → 88.2), while categories GRPO already solves (Clean, Cool) are left unchanged. PickLookCleanHeatCoolPick2Avg 0 20 40 60 80 100 Success Rate (%) 50 0 29 48 33 12 31 67 44 52 74 57 59 59 GRPOAHEAD Figure 7: Per-task success rate on the ALFWorld Unseen-134 split (Qwen3-1.7B-Instruct). Com- panion to Figure 6, at the smaller scale. E.3 Where the Reweighting Signal Concentrates Setup. AHEAD assumes the self-distillation gapδ t,ℓ is larger on error steps than on routine steps, so that reweighting the advantage puts more credit on the tokens that caused the failure. We check this directly. We freeze a mid-training checkpoint (step 25) on ALFWorld and collectδ t,ℓ on the reweighted stepsM τ (Eq. 5): steps of failed trajectories with non-zero advantage, i.e. the steps that actually affect the loss. A step is an error step if it is in the LLM-identified setE τ , and routine otherwise. This gives 2,525 error steps and 26,877 routine steps. The signal is larger on error steps. Figure 10a shows the per-step|δ t,ℓ |for both groups. Error steps have a larger gap (mean0.37vs.0.17), and the two distributions separate clearly: ranking steps 17 050100150 Training Step 20 40 60 80 Success Rate (%) GRPO AHEAD 050100150 Training Step 20 30 40 50 Avg. Traj. Length 050100150 Training Step 0 2 4 6 8 Train Reward Figure 8: Training dynamics on ALFWorld (7B); the success-rate panel is measured on the seen split. AHEAD attains a higher success rate and train reward throughout training, while driving average trajectory length down faster. The agent solves more tasks and does so in fewer steps. 050100150 Training Step 20 40 60 Success Rate (%) GRPO AHEAD 050100150 Training Step 20 30 40 50 Avg. Traj. Length 050100150 Training Step 0 2 4 6 Train Reward Figure 9: Training dynamics on ALFWorld (Qwen3-1.7B-Instruct); the success-rate panel is measured on the seen split. Companion to Figure 8. by|δ t,ℓ |recovers the LLM error labels at AUROC0.87. So the richer PI on error steps (Eq. 4) makes the signal stronger there on its own. The reweight in the loss is smaller but still targets error steps.We do not apply the raw gap. The clip (ε = 0.2) and the mixing coefficientλ k bound and scale it (Eq. 6–7). Figure 10b shows what the advantage is actually multiplied by,| ̃ A t,ℓ /A ep −1|. Error steps are still favored, but the gap shrinks (AUROC 0.70), leaving the reward in control of the update direction. Token-level view. Figure 11 shows one failed trajectory (task: find two statues and put them in the diningtable). At the error step (t∈E τ ), the tokens for the wrong decision—re-examining the diningtable instead of searchingsidetable 2—have the largest|δ t,ℓ |. SinceA ep < 0on failed trajectories, these are exactly the tokens amplified byw t,ℓ > 1(Eq. 6). A routine step in the same trajectory stays near uniform. This shows the reweighting localizes credit through δ alone. 18 10 −2 10 −1 10 0 self-distillation signal: mean |δ t, ℓ | per step 0 1 2 3 4 5 density (per step) AUROC = 0.87 routine t∉ τ (n= 26877) error t∈ τ (n= 2525) (a) Self-distillation signal|δ t,ℓ |. 0.000.010.020.030.040.050.06 reweight on advantage: mean | ̃ A t, ℓ /A ep −1| per step 0 10 20 30 40 50 density (per step) AUROC = 0.70 routine t∉ τ (n= 26877) error t∈ τ (n= 2525) (b) Reweight applied to the advantage. Figure 10: The reweighting signal targets error steps. Per-step distributions on the reweighted stepsM τ (ALFWorld, frozen step-25 checkpoint). (a) The gap|δ t,ℓ |is larger on error stepst∈E τ than on routine steps (AUROC0.87). (b) After the clip and mixingλ k , the reweight that actually scales the advantage,| ̃ A t,ℓ /A ep −1|, still favors error steps but with a smaller gap (AUROC0.70), leaving the reward in control of the update direction. <think >↵I have already taken statue·1 from the dining table ·1. The next step is to find another statue and put both statues on the dining table . Since the statues are not explicitly located in the current observation , I should examine the dining table again to confirm and also check other locations like the sid et ables , coff e etable , and drawers , which are commonly places to find statues.↵</ think >↵< action >ex amine dining table ·1</ action > <think >↵ Given the current task is to find two statues and put them in the dining table , I have already picked up statue·1 and moved it to the dining table . The current observation indicates there 's nothing special about statue·1, so it 's likely that I need to find the second statue. Since statue·1 is already at the dining table , the next logical step is to examine the dining table to see if there 's a second statue there . If there isn 't, I might need to look for it elsewhere , but the most efficient way is to check the dining table first .↵</ think >↵< action >ex amine dining table ·1</ action > Task: find two statues and put them in the diningtable error step 27 (t∈ τ ) · agent should not stay at the diningtable, but search sidetable 2 for statue 2 routine step 9 (t∉ τ ) δ t, ℓ (red: teacher disfavors, δ< 0 ↔ blue: endorses, δ> 0) <think >↵I have already taken statue·1 from the dining table ·1. The next step is to find another statue and put both statues on the dining table . Since the statues are not explicitly located in the current observation , I should examine the dining table again to confirm and also check other locations like the sid et ables , coff e etable , and drawers , which are commonly places to find statues.↵</ think >↵< action >ex amine dining table ·1</ action > <think >↵ Given the current task is to find two statues and put them in the dining table , I have already picked up statue·1 and moved it to the dining table . The current observation indicates there 's nothing special about statue·1, so it 's likely that I need to find the second statue. Since statue·1 is already at the dining table , the next logical step is to examine the dining table to see if there 's a second statue there . If there isn 't, I might need to look for it elsewhere , but the most efficient way is to check the dining table first .↵</ think >↵< action >ex amine dining table ·1</ action > Task: find two statues and put them in the diningtable error step 27 (t∈ τ ) · agent should not stay at the diningtable, but search sidetable 2 for statue 2 routine step 9 (t∉ τ ) 0.811.2 w t, ℓ = clip(exp(sgn(A ep )δ t, ℓ ), 1−ε, 1+ε) (blue: ↓ weaker ↔ red: ↑ stronger update) Figure 11: Token-level credit on one failed trajectory (ALFWorld). Top: signed gapδ t,ℓ ; red marks tokens the teacher disfavors (δ < 0). At the error step (t∈E τ ) the tokens for the wrong decision have the largest|δ|; the routine step (t /∈E τ ) stays near zero. Bottom: the weightw t,ℓ = clip(exp(sgn(A ep )δ t,ℓ ), 1−ε, 1+ε). SinceA ep < 0on failed trajectories, these tokens get the largest amplification (w> 1). 19 HyperparameterValue Training steps150 Training batch size16 for ALFWorld and WebShop; 128 for Search-based QA Rollout group size N8 Learning rate1× 10 −6 Hardware8× H100 GRPO clipping range ε0.2 Reweight bound ε0.2 Initial distillation coefficient λ 0 0.5 Decay horizon D50 LLM analyzerClaude Opus 4.7 Maximum prompt length2,048 for ALFWorld; 4,096 for WebShop and Search-based QA Maximum interaction steps50 for ALFWorld; 15 for WebShop; 4 for Search-based QA Table 7: Training configuration. The block in the middle lists the parameters introduced by AHEAD (Sec. 2.4); everything else is inherited unchanged from the GRPO baseline, so the comparison in Table 1 isolates the effect of step-aware distillation. Prompt for ALFWorld You are an expert agent operating in the ALFRED Embodied Environment.Your task is to: task_description Prior to this step, you have already taken step_count step(s).Below are the most recent history_length observations and the corresponding actions you took: action_history You are now at step current_step and your current observation is: current_observation Your admissible actions of the current situation are: [admissible_actions]. Now it’s your turn to take an action. You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags. Figure 12: Prompt template for the ALFWorld environment. F Hyperparameters Table 7 lists the full configuration. All backbones share these settings. Note that AHEAD adds no KL regularisation term and no per-step coefficient: the only method-specific hyperparameters are the reweight bound ε and the decay schedule of λ k . G Prompts G.1 Environment Interaction Prompts The ALFWorld and WebShop templates follow Feng et al. (2025), and the Search-based QA template follows Jin et al. (2025). Every method (AHEAD and all baselines alike) rolls out with the same template in a given environment, so the comparison in Table 1 is not confounded by prompt wording. Braces mark slots filled at run time by the environment. Note that the agent is shown only a recent window of the interaction history rather than the full trajectory, making the step-level historyh t in Sec. 2.2 a bounded context. G.2 Error-Step Analyzer Prompt Figure 15 gives the prompt behind Stage 1 and Stage 2 of AHEAD (Sec. 2.2). A single call does both jobs: it selects the error stepsE τ and, for each one, writes the corrective hintΦ llm t that is injected into the teacher context. It is invoked only on failed trajectories. 20 Prompt for WebShop You are an expert autonomous agent operating in the WebShop e-commerce environment. Your task is to: task_description. Prior to this step, you have already taken step_count step(s).Below are the most recent history_length observations and the corresponding actions you took: action_history You are now at step current_step and your current observation is: current_observation. Your admissible actions of the current situation are: [ available_actions ]. Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within<think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags. Figure 13: Prompt template for the WebShop environment. Prompt for Search-based QA You are an expert agent tasked with answering the given question step-by-step. Your question: task_description Prior to this step, you have already taken step_count step(s). Below is the interaction history where <search> </search>wrapped your past search queries and<information> </information>wrapped the corresponding search results returned by the external search engine. History: memory_context Now it’s your turn to respond for the current step. You should first conduct reasoning process. This process MUST be enclosed within<think> </think> tags. After completing your reasoning, choose only one of the following actions (do not perform both): (1) If you find you lack some knowledge, you can call a search engine to get more external information using format: <search> your query </search>. (2) If you have enough knowledge to answer the question confidently, provide your final answer within <answer> </answer> tags, without detailed illustrations. For example, <answer>Beijing</answer>. Figure 14: Prompt template for the Search-based QA environment. Two constraints in the prompt matter for the method. First, the analyzer is asked for at most max_steps key decision steps rather than a label for every step, which keepsE τ sparse so that most steps fall back to environment feedback alone. Second, the hint is required to be an imperative naming the correct object or query, and is explicitly forbidden from describing what went wrong. A hint that merely restated the failure would duplicate what the environment feedback already provides; the corrective prescription is what environment feedback cannot provide. 21 Prompt for the Error-Step Analyzer System: You are an expert agent trajectory analyzer. User: ## Task task ## Failed Trajectory failed success_block ## Instructions You are given a FAILED trajectory. Each step is labeled[step N] action: ... | env: ...; N is that step’s index. Indices are 0-based and contiguous (first step is[step 0]); valid indices are EXACTLY the N values printed above. noref_note Read the ENTIRE trajectory (and the reference Successful Trajectory, if provided) and judge each step against the## Taskgoal. Select AT MOST max_steps KEY decision steps: the steps where the choice of action most determined whether the task would succeed. Prefer the steps that need correcting; do NOT list every step, only the few that mattered most.first_fatal_error= the earliest step that made failure unavoidable. Rules: eachstep_indexmust be copied verbatim from a printed[step N]label (do NOT add/subtract 1). Respond ONLY with JSON (no prose, no markdown fences): "error_steps": ["step_index": <int>, "diagnosis": "<one sentence: why this step is a key decision>", "hint": "<provide one imperative, concise sentence telling the agent the correct action to take at this step; do NOT describe what went wrong or use the word ’agent’>", ...], "first_fatal_error": <int> Figure 15: Prompt template for the LLM error-step analyzer. 22