Paper deep dive
Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation
Hangxi Guo, Fengyuan Liu, Yue Wang, Yuhua Qi, Haoyi Xiong, Fei Sun, Mengnan Du
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/10/2026, 1:25:53 AM
Summary
The paper introduces SELF (agentic SElf-distilLation with environmental Feedback modeling), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. Unlike previous methods that only condition a teacher on feedback, SELF explicitly models how the environment responds to agent actions, creating a mutually reinforcing mechanism that improves policy learning without verifiable rewards. SELF outperforms baselines like SDPO and GRPO on benchmarks such as tau-bench and AppWorld.
Entities (8)
Relation Signals (7)
SELF → evaluatedon → tau-bench
confidence 95% · SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in $ au$-bench success rate
SELF → evaluatedon → AppWorld
confidence 95% · and by 10.71 and 3.57 percentage points in AppWorld task goal completion
SELF → outperforms → SDPO
confidence 95% · SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in $ au$-bench success rate
SELF → outperforms → GRPO
confidence 95% · SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in $ au$-bench success rate
SELF → comprises → Hindsight Self-Distillation
confidence 90% · SELF, a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation.
SELF → comprises → Environmental Feedback Modeling
confidence 90% · SELF, a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation.
SELF → uses → Qwen3-8b
confidence 90% · With Qwen3-8B, SELF outperforms SDPO and GRPO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable. Recent methods use environmental feedback as privileged context for hindsight self-distillation, but our analysis suggests that simply conditioning the teacher on feedback is insufficient, motivating us to rethink how environmental feedback is used in agentic self-distillation. Given that environmental feedback contains rich supervision for modeling how the environment responds to agent actions, we introduce \textit{agentic SElf-distilLation with environmental Feedback modeling} (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. SELF learns to predict environmental responses while distilling guidance from a feedback-conditioned self-teacher into the policy. Our analysis reveals a mutually reinforcing mechanism: environmental feedback modeling strengthens hindsight supervision and policy learning, while self-distillation enhances the model's ability to model environmental feedback. With Qwen3-8B, SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in $\tau$-bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively. These results show that SELF uses environmental feedback more effectively within agentic self-distillation, improving agent capabilities.
Tags
Links
- Source: https://arxiv.org/abs/2610.11384v1
- Canonical: https://arxiv.org/abs/2610.11384v1
Trouble viewing inline? Open PDF directly →
Full Text
91,302 characters extracted from source content.
Expand or collapse full text
Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation Hangxi Guo Affiliation: The Chinese University of Hong Kong, Shenzhen Fengyuan Liu Affiliation: The Chinese University of Hong Kong, Shenzhen Yue Wang Affiliation: Shanghai AI Laboratory Affiliation: University of Science and Technology of China Yuhua Qi Affiliation: The Chinese University of Hong Kong, Shenzhen Haoyi Xiong Affiliation: Independent Fei Sun Affiliation: Institute of Computing Technology, CAS Mengnan Du Email: keepstudying931@gmail.com, mengnandu@cuhk.edu.cn*Equal contribution, †Corresponding author Affiliation: The Chinese University of Hong Kong, Shenzhen Abstract Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable. Recent methods use environmental feedback as privileged context for hindsight self-distillation, but our analysis suggests that simply conditioning the teacher on feedback is insufficient, motivating us to rethink how environmental feedback is used in agentic self-distillation. Given that environmental feedback contains rich supervision for modeling how the environment responds to agent actions, we introduce agentic SElf-distilLation with environmental Feedback modeling (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. SELF learns to predict environmental responses while distilling guidance from a feedback-conditioned self-teacher into the policy. Our analysis reveals a mutually reinforcing mechanism: environmental feedback modeling strengthens hindsight supervision and policy learning, while self-distillation enhances the model’s ability to model environmental feedback. With Qwen3-8B, SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in τ-bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively. These results show that SELF uses environmental feedback more effectively within agentic self-distillation, improving agent capabilities. 1 Introduction In recent years, training language agents to solve complex tasks in interactive environments involving tool use and code execution has become increasingly important (Yao et al., 2023; Trivedi et al., 2024; Choudhury and Sodhi, 2025). A common training approach is Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2025), where algorithms such as Group Relative Policy Optimization (GRPO) (Shao et al., 2024) use trajectory-level rewards to optimize the policy. In some settings, however, reliable rewards are unavailable (Tian et al., 2026; Liu et al., 2026a), and environmental feedback becomes the primary source of external information for agent training. This feedback arises naturally from interactions and includes tool outputs, code execution results, and environmental state changes caused by the agent’s actions. Since such feedback typically describes action consequences rather than providing direct supervision for policy learning, using it effectively in agent training remains a challenge. Recent methods tackle this challenge through hindsight self-distillation. SDPO (Hübotter et al., 2026) conditions a self-teacher on environmental feedback, while OEL (Ye et al., 2026) further extracts transferable knowledge from interaction experience. Our analysis shows that, despite containing sufficient information for policy improvement, such feedback yields only modest gains when used in these ways, suggesting that the key limitation lies in how it is exploited during training. Although environmental feedback does not directly supervise action generation, it provides direct targets for learning how the environment responds to agent actions. We therefore ask: Can learning to predict environmental feedback enable more effective use of it in agentic self-distillation? Our analysis reveals a mutually reinforcing relationship: environmental feedback modeling improves hindsight self-distillation, while self-distillation in turn strengthens environmental feedback prediction. Building on this finding, we introduce agentic SElf-distilLation with environmental Feedback (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. For each interaction transition, SELF uses the realized feedback in two complementary ways: as a prediction target for learning how the environment responds to the agent’s action, and as privileged context for a self-teacher whose action distribution is distilled into the student policy. By jointly learning from these two signals within a shared model, SELF enables more effective use of environmental feedback for policy improvement. We evaluate SELF on τ-bench Retail (Yao et al., 2025) for multi-turn customer-service interactions with domain-specific tools, and AppWorld (Trivedi et al., 2024) for code-based task completion across applications. Across Qwen3-4B-Instruct and Qwen3-8B (non-thinking) (Yang et al., 2025), SELF outperforms feedback-learning baselines and reward-trained GRPO (Shao et al., 2024) in task success and goal completion. With Qwen3-8B, SELF surpasses SDPO (Hübotter et al., 2026) and GRPO by 6.4 and 4.1 percentage points in τ-bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively, which demonstrates SELF’s effectiveness in turning environmental feedback into useful supervision for policy improvement. Further analyses show that SELF also retains strong generalization capabilities beyond the training environments. Our key contributions can be summarized as follows: • We identify a mutually reinforcing mechanism: environmental feedback modeling improves hindsight self-distillation, while self-distillation strengthens environmental feedback prediction. • We propose SELF, an agent training framework that learns from environmental feedback by jointly optimizing environmental feedback modeling and hindsight self-distillation. • We empirically show that SELF improves performance on τ-bench Retail and AppWorld, outperforming both existing self-distillation approaches and reward-trained GRPO, which demonstrates its effectiveness in enabling agentic self-distillation to make better use of environmental feedback. 2 Related Work Agentic Self-Distillation. Agentic self-distillation provides dense policy supervision without requiring a separate stronger teacher. OPSD (Zhao et al., 2026) constructs student and teacher policies from the same model, conditioning the teacher on privileged reference information. SDPO (Hübotter et al., 2026) instead conditions the self-teacher on textual feedback and distills its feedback-informed predictions into the policy. SDFT (Shenfeld et al., 2026) uses demonstration-conditioned self-distillation to acquire new knowledge and skills while mitigating forgetting. RLSD (Yang et al., 2026a) further combines self-distillation with RLVR, using privileged teachers for token-level credit while retaining verifier rewards to determine the policy-update direction. Learning from Environmental Feedback. Environmental feedback contains rich information about the realized consequences of agent actions, and recent methods exploit this information in different forms. OCSD (Yang et al., 2026b) contrasts replay contexts with and without future observations to calibrate policy updates, while HERO (Liu et al., 2026b) converts subsequent observations into turn-level hindsight diagnoses. OEL (Ye et al., 2026) extracts transferable experiential knowledge from interaction trajectories before distilling it into the policy. Kleine Buening et al. (Kleine Buening et al., 2026) use raw user follow-up messages as hindsight information for self-distillation. AHEAD (Jin et al., 2026) conditions the teacher on environment feedback and supplements critical error steps with generated corrective hints, whereas OpenClaw-RL (Wang et al., 2026) converts next-state signals into process rewards or hindsight guidance for online agent learning. These approaches use realized feedback to construct or refine supervision for action generation. Environmental Feedback Modeling for Policy Learning. A complementary line of work directly learns the predictive structure contained in interaction feedback. ECHO (Shrivastava et al., 2026) and PaW (Lu et al., 2026) incorporate environment prediction into policy optimization, while AAWM (Cai et al., 2026) studies decision-oriented modeling targets rather than directly reconstructing future observations. Early Experience (Zhang et al., 2026) uses next states produced by agent actions as supervision for implicit world modeling and self-reflection. Most closely related, RLTF (Song et al., 2026) studies both feedback-conditioned self-distillation and feedback modeling, where textual critiques are predicted as an auxiliary objective. In contrast, SELF models environmental responses directly from interactive trajectories and uses the same realized feedback both as a prediction target and as privileged context for hindsight self-distillation. Our analysis further shows that these objectives mutually reinforce each other. 3 Methodology We present SELF, a framework that improves agentic self-distillation by explicitly modeling environmental feedback. We first formalize agent–environment interaction and the problem of learning from environmental feedback. We then describe on-policy hindsight self-distillation, which uses realized feedback as privileged information to construct policy supervision. Finally, we introduce the environmental feedback modeling objective and show how it is jointly optimized with hindsight self-distillation within the framework. Figure 1: Overview of SELF. The current policy interacts with the environment to collect transitions (hi,ai,fi)(h_i,a_i,f_i). Each transition is used for two complementary objectives: hindsight self-distillation, where a feedback-conditioned EMA teacher supervises the student policy, and environmental feedback modeling, where the model predicts the realized feedback from the interaction history and action. The two objectives are jointly optimized to obtain the next-round policy. 3.1 Problem Formulation Agent–Environment Interaction. We consider an agent that interacts with an environment through actions and observable feedback, including tool outputs, code execution results, and state changes. Let q denote a task instruction, o0o_0 the initial observation, and fif_i the feedback returned after action aia_i. Before taking aia_i, the agent has access to the interaction history hi=(q,o0,a1,f1,…,ai−1,fi−1).h_i=(q,o_0,a_1,f_1,…,a_i-1,f_i-1). (1) At training round t, the current policy and environment generate ai∼πθt(⋅∣hi,promptstu),fi∼env(⋅∣hi,ai),a_i _ _t(· h_i,prompt_stu),\;f_i _env(· h_i,a_i), (2) where πθt _ _t is the policy with parameters θt _t, promptstuprompt_stu specifies the action-generation interface, and envP_env is the conditional distribution of environmental feedback. Each transition (hi,ai,fi)(h_i,a_i,f_i) associates an action with its observed consequence, without requiring access to latent environment states. A trajectory is τ=(q,o0,a1,f1,…,anτ,fnτ)τ=(q,o_0,a_1,f_1,…,a_n_τ,f_n_τ), where nτn_τ is the number of interaction turns, and tT_t denotes the trajectories collected at round t. Training uses only the resulting actions and environmental feedback, without task rewards, outcome labels, or external teachers. For feedback modeling, the observed feedback fif_i is represented by the point-mass target δenv,i(f)=f=fi _env,i(f)=1\f=f_i\. 3.2 Hindsight Self-Distillation Hindsight self-distillation provides a general approach for converting environmental feedback into supervision for policy learning. We consider its on-policy form, where trajectories are collected by the current policy and distillation is performed on the action prefixes encountered along these trajectories (Agarwal et al., 2024). For each transition (hi,ai,fi)(h_i,a_i,f_i), the student predicts the action using only the information available before the environmental feedback is observed, while a self-teacher additionally conditions on the realized feedback fif_i as privileged hindsight information. The student and teacher use distinct prompting interfaces, promptstuprompt_stu and promptteaprompt_tea, respectively. For an action aia_i containing LiL_i tokens, we define the student and teacher next-token distributions as pstui,j(⋅)=πθ(⋅∣hi,ai,<j,promptstu),pteai,j(⋅)=πθ¯(⋅∣hi,fi,ai,<j,prompttea), p^stu_i,j(·)= _θ (· h_i,a_i,<j,prompt_stu ),\;p^tea_i,j(·)= _ θ (· h_i,f_i,a_i,<j,prompt_tea ), (3) where both distributions are evaluated on the same on-policy action prefix ai,<ja_i,<j, but only the teacher has access to the realized environmental feedback. The teacher parameters θ¯ θ are maintained as an exponential moving average (EMA) (Tarvainen and Valpola, 2017) of the student parameters, θ¯←βθ¯+(1−β)θ θ←β θ+(1-β)θ, where β is the EMA decay coefficient. The feedback-conditioned teacher is distilled into the student through the reverse-KL objective (Gu et al., 2024) ℓsd(hi,ai,fi,θ,θ¯)=j∼Unif([Li])[DKL(pi,jstu∥sg[pi,jtea])], _sd(h_i,a_i,f_i;θ, θ)=E_j ([L_i]) [D_KL\! (p^stu_i,j\, \|\,sg[p^tea_i,j] ) ], (4) where [Li]=1,…,Li[L_i]=\1,…,L_i\ and sgsg stops gradients through the teacher. The collected action therefore determines the contexts at which the two distributions are compared, rather than serving as a ground-truth action target. This formulation allows information available only after an action is executed to supervise a policy that must act without access to that information at inference time (Choudhury and Sodhi, 2025). In our analysis, however, hindsight self-distillation in this form yields only modest policy improvements, suggesting that conditioning the teacher on feedback alone does not fully exploit the information contained in environmental feedback. Why Environmental Feedback Modeling? Hindsight self-distillation uses environmental feedback as privileged context for constructing policy supervision, but does not explicitly learn the information contained in the feedback itself. Our analysis suggests that this limits how effectively the self-teacher can exploit environmental feedback. Environmental feedback, however, provides rich supervision about how the environment responds to an agent’s actions. Each interaction therefore naturally provides a supervised action–feedback relation, (hi,ai)⟶fi.(h_i,a_i) f_i. This motivates explicitly modeling environmental feedback alongside hindsight self-distillation, so that the model learns how the environment responds while using the same feedback to improve its policy. 3.3 Environmental Feedback Modeling Motivated by the observation in the previous section, we treat each transition (hi,ai,fi)(h_i,a_i,f_i) not only as privileged context for hindsight self-distillation, but also as direct supervision for modeling how the environment responds to an action. We therefore jointly train environmental feedback modeling and agentic self-distillation. Given (hi,ai,fi)(h_i,a_i,f_i), we approximate the conditional feedback distribution env(⋅∣hi,ai)P_env(· h_i,a_i) with penv,θ(⋅∣hi,ai,promptenv)p_env,θ(· h_i,a_i,prompt_env), where promptenvprompt_env is a dedicated prompt for environmental feedback prediction. The conditioning context contains the complete interaction history up to and including aia_i, while the realized feedback fif_i serves as the prediction target. This prediction interface shares parameters θ with the student policy, rather than introducing a separate environment model. For a feedback sequence fif_i containing MiM_i tokens, let δenv,i,k _env,i,k denote the point-mass target on the observed token fi,kf_i,k. We define the autoregressive next-token distribution as pi,kenv(⋅)=penv,θ(⋅∣hi,ai,fi,<k,promptenv),p^env_i,k(·)=p_env,θ (· h_i,a_i,f_i,<k,prompt_env ), (5) where fi,<kf_i,<k denotes the observed feedback prefix. The environmental feedback modeling objective is ℓenv(hi,ai,fi,θ)=k∼Unif([Mi])[DKL(δenv,i,k∥pi,kenv)]. _env(h_i,a_i,f_i;θ)=E_k ([M_i]) [D_KL\! ( _env,i,k\, \|\,p^env_i,k ) ]. (6) Because each target is a point mass, this forward-KL objective is equivalent to the token-normalized negative log-likelihood of the realized feedback. It therefore trains the shared model to capture observable action–feedback relationships directly from interaction, without requiring latent environment states or treating the sampled action as a correct action target. Table 1: Comparison of learning paradigms under the configurations considered in this work. Supervision describes the granularity of the learning signal. Teacher denotes the source of teacher supervision. Env. Modeling indicates whether an explicit objective predicts environmental responses from interaction histories and actions. Dashes indicate inapplicable attributes. Method Sampling Supervision Feedback Source Teacher Env. Modeling SFT Off-policy Dense Teacher External No GRPO On-policy Sparse Verifier None No Mem0 – – Environment None No SDPO On-policy Dense Environment Self No SELF (Ours) On-policy Dense Environment Self Yes 3.4 Overall SELF Training Objective The proposed SELF framework jointly optimizes environmental feedback modeling and hindsight self-distillation within a shared model. The same realized feedback fif_i serves two complementary roles: it provides a prediction target for learning how the environment responds to an action, while also acting as privileged hindsight information for constructing policy supervision. By optimizing both objectives through the same underlying model, SELF jointly improves environmental feedback modeling and agent action generation. For each transition (hi,ai,fi)(h_i,a_i,f_i), the SELF objective is ℓSELF=λoℓenv+λaℓsd, _SELF= _o _env+ _a _sd, (7) where λo _o and λa _a control the contributions of environmental feedback modeling and hindsight self-distillation, respectively. At training round t, the current policy collects a set of trajectories tT_t. Averaging the token-normalized objectives over trajectories and interaction turns yields ℒSELF,t(θ)=τ∼Unif(t)i∼Unif([nτ])[ _SELF,t(θ)=E_τ (T_t)E_i ([n_τ]) [ λok∼Unif([Mi])DKL(δenv,i,k∥pi,kenv) _oE_k ([M_i])D_KL\! ( _env,i,k\, \|\,p^env_i,k ) (8) +λaj∼Unif([Li])DKL(pstui,j∥sg[pteai,j])], + _aE_j ([L_i])D_KL\! (p^stu_i,j\, \|\,sg[p^tea_i,j] ) ], where nτn_τ denotes the number of interaction turns in trajectory τ. Optimizing ℒSELF,tL_SELF,t updates the shared parameters from θt _t to θt+1 _t+1. Environmental feedback modeling therefore updates the same model from which subsequent hindsight supervision is derived, while self-distillation directly improves the deployable policy and simultaneously updates the representation used for feedback prediction. The updated policy then collects new interactions for the next training round. At deployment, only the student policy πθ(⋅∣hi,promptstu) _θ(· h_i,prompt_stu) is required. The complete training procedure is summarized in Algorithm 1 in the appendix, and illustrative examples of promptstuprompt_stu, promptteaprompt_tea, and promptenvprompt_env are provided in the appendix. 4 Experiments In this section, we evaluate SELF on τ-bench Retail and AppWorld and address the following three research questions (RQs). RQ1: Can SELF improve interactive agent performance using environmental feedback without verifiable rewards? RQ2: Do environmental feedback modeling and hindsight self-distillation mutually improve each other? RQ3: Does SELF retain general language-model capabilities after agent training? Table 2: Performance on τ-bench and AppWorld. SR denotes the pass@4pass@4 success rate on τ-bench, and Turns denotes the average number of interaction turns. TGC and SGC denote task goal completion and scenario goal completion on AppWorld, respectively. Observation and Full denote baseline feedback settings, while Reward denotes reward-based training. SELF learns from agent-generated actions and environmental observations, and its rows are shaded light blue. Higher SR, TGC, and SGC and lower Turns are better. Within each backbone, excluding external model baselines, the best means are bold with pale gold shading, and the second-best means are underlined with pale pink shading. Missing results and inapplicable settings are denoted by –. Method Setting τ-bench AppWorld SR ↑ Turns ↓ TGC ↑ SGC ↑ External Model Baselines Qwen3-30B-A3B-Instruct – 33.6 11.9 31.55 12.50 Qwen3-4B-Instruct Base – 27.2 ± 1.4 10.5 ± 0.3 16.67 ± 1.19 5.36 ± 0.62 OEL Observation 30.6 ± 1.5 10.9 ± 0.4 19.05 ± 1.43 6.25 ± 0.74 SDPO Observation 30.4 ± 1.7 11.5 ± 0.5 20.24 ± 1.56 7.14 ± 0.83 Mem0 Observation 19.2 ± 1.8 12.6 ± 0.6 11.90 ± 1.31 3.57 ± 0.58 SDPO Full 22.5 ± 1.6 12.4 ± 0.5 14.29 ± 1.47 4.46 ± 0.69 Mem0 Full 27.4 ± 1.5 11.7 ± 0.4 17.86 ± 1.38 5.95 ± 0.77 GRPO Reward 33.3 ± 1.9 13.4 ± 0.7 28.57 ± 1.72 10.71 ± 1.04 SELF (Ours) Observation 38.7 ± 1.5 10.4 ± 0.4 32.14 ± 1.61 12.50 ± 0.96 Qwen3-8B (Non-Thinking) Base – 35.7 ± 1.3 10.0 ± 0.3 22.62 ± 1.28 7.14 ± 0.71 OEL Observation 37.9 ± 1.4 10.4 ± 0.4 26.19 ± 1.46 8.04 ± 0.82 SDPO Observation 36.8 ± 1.6 11.0 ± 0.4 25.00 ± 1.53 8.93 ± 0.89 Mem0 Observation 29.5 ± 1.7 12.0 ± 0.6 16.67 ± 1.35 4.36 ± 0.63 SDPO Full 30.6 ± 1.5 11.8 ± 0.5 19.05 ± 1.42 6.25 ± 0.76 Mem0 Full 35.8 ± 1.4 11.1 ± 0.4 22.62 ± 1.39 7.14 ± 0.84 GRPO Reward 39.1 ± 1.8 12.7 ± 0.6 32.14 ± 1.68 11.61 ± 1.02 SELF (Ours) Observation 43.2 ± 1.4 10.8 ± 0.4 35.71 ± 1.57 13.39 ± 0.93 4.1 Experimental Setup Benchmark Datasets. We evaluate SELF on τ-bench Retail (Yao et al., 2025) and AppWorld (Trivedi et al., 2024), covering multi-turn customer-service interactions with domain-specific tools and code-based task completion across applications, respectively. We use the official training splits for learning and held-out test splits for evaluation. On τ-bench Retail, we report success rate (SR; pass@4pass@4) and the average number of interaction turns (Turns) to assess task performance and interaction efficiency. On AppWorld, we report Task Goal Completion (TGC) and Scenario Goal Completion (SGC), measuring task-level success and consistency across task variants, respectively. We additionally evaluate MMLU (Hendrycks et al., 2021), MMLU-Pro (Wang et al., 2024), and IFEval (Zhou et al., 2023) to assess the retention of general capabilities after agent training. Comparing Baselines. We compare SELF with hindsight-distillation methods OEL (Ye et al., 2026) and SDPO (Hübotter et al., 2026), the memory-based method Mem0 (Chhikara et al., 2025), and the reinforcement learning method GRPO (Shao et al., 2024). We distinguish three supervision settings: Observation supplies only environmental observations, without ground-truth task outcomes or success/failure labels; Full additionally supplies ground-truth information and explicit success/failure feedback; and Reward provides verifiable task rewards for reinforcement learning. OEL and SELF use Observation, SDPO and Mem0 are evaluated under both Observation and Full, and GRPO uses Reward. Table 1 summarizes the methods’ sampling strategies, supervision sources, and use of environment modeling. We also report unadapted base models and a strong external model to contextualize trained-agent performance: Qwen3-30B-A3B-Instruct (Yang et al., 2025). Implementation Details. Our main experiments use Qwen3-4B-Instruct and Qwen3-8B (non-thinking) as backbone models (Yang et al., 2025). All ablation studies, mechanism analyses, and general-capability evaluations use Qwen3-4B-Instruct. Models evaluated in our framework operate under the same ReAct-style tool-use interface (Yao et al., 2023). For SELF, the self-distillation teacher is maintained as an exponential moving average (EMA) (Tarvainen and Valpola, 2017) of the student, with decay coefficient β=0.99β=0.99. Unless otherwise specified, we use λo=0.2 _o=0.2 and λa=1.0 _a=1.0 as the default loss weights. We repeat each experiment over three independent runs and report the mean together with the standard deviation. Complete training hyperparameters and implementation details are provided in the appendix. Figure 2: Performance comparison on τ-bench Retail and AppWorld. (a) Success rate (SR, pass@4) gains of SELF over selected comparators on τ-bench Retail. Open and filled circles denote SELF and comparator scores, respectively; annotations and line colors indicate gains in percentage points. The superscript ‡ denotes Qwen3-8B; unmarked methods use Qwen3-4B-Instruct unless otherwise specified. (b) Task goal completion (TGC) and scenario goal completion (SGC) on AppWorld. Colors distinguish methods, and marker shapes distinguish model backbones. 4.2 Main Results (RQ1) We first evaluate whether SELF improves interactive agent performance using environmental feedback alone, comparing it with self-distillation, memory-based, and reward-trained baselines under different supervision settings. Table 2 and Figure 2 report task performance and interaction efficiency on τ-bench Retail and AppWorld using Qwen3-4B-Instruct and Qwen3-8B (non-thinking). Under Observation, SELF consistently outperforms SDPO and OEL on SR, TGC, and SGC. With Qwen3-4B-Instruct, SELF achieves 38.7% SR, exceeding SDPO and OEL by 8.3 and 8.1 percentage points; the corresponding gains with Qwen3-8B are 6.4 and 5.3 points. These results indicate more effective use of environmental feedback than existing self-distillation methods. SELF also surpasses Mem0 under both Observation and Full: with Qwen3-4B-Instruct, Mem0 achieves 19.2% and 27.4% SR, respectively, versus SELF’s 38.7%. SELF further outperforms SDPO under Full feedback. Indeed, Full feedback reduces SDPO’s SR from 30.4% to 22.5% with Qwen3-4B-Instruct and from 36.8% to 30.6% with Qwen3-8B, alongside lower AppWorld TGC and SGC. Additional ground-truth information and success/failure labels therefore do not necessarily improve feedback-based learning. Finally, SELF exceeds reward-trained GRPO by 5.4 and 4.1 SR points on the two backbones, respectively, while also achieving higher TGC and SGC. Together, these comparisons demonstrate effective policy learning from environmental feedback without verifiable rewards. 4.3 Mutual Benefits of Feedback Modeling and Self-Distillation (RQ2) We examine the interaction between the two objectives in SELF. Specifically, we test whether environmental feedback modeling improves feedback-derived hindsight supervision, and whether hindsight self-distillation improves the shared model’s ability to predict environmental feedback. Figure 3: Ablation studies and general capability evaluation with Qwen3-4B-Instruct. (a) Student and privileged teacher success rates (SR, pass@4) and average interaction turns on τ-bench Retail across different values of λo _o. (b) Environmental feedback prediction accuracy on τ-bench Retail and AppWorld across different values of λa _a. (c) Performance on MMLU, MMLU-Pro, and IFEval; SELF† denotes the privileged teacher. Vertical dashed lines in (a) and (b) indicate default settings. Table 3: Effect of the environmental feedback modeling weight λo _o on τ-bench Retail with Qwen3-4B-Instruct. SR is reported as pass@4 (%), and Turns denotes the average number of interaction turns. Bold indicates the best result in each column. Setting Student Privileged teacher SR ↑ Turns ↓ SR ↑ Turns ↓ Base 27.2 ± 1.4 10.5 ± 0.3 – – λo=0 _o=0 29.4 ± 2.3 11.2 ± 0.5 30.1 ± 1.1 10.5 ± 0.2 λo=0.1 _o=0.1 36.5 ± 0.9 10.8 ± 0.4 49.1 ± 1.8 12.1 ± 0.6 λo=0.2 _o=0.2 38.7 ± 1.5 10.4 ± 0.4 46.7 ± 2.4 10.9 ± 0.4 λo=0.5 _o=0.5 35.7 ± 2.1 12.5 ± 0.6 44.0 ± 1.3 10.7 ± 0.3 λo=1.0 _o=1.0 34.2 ± 1.2 11.3 ± 0.3 44.8 ± 2.0 11.2 ± 0.5 Environmental feedback modeling strengthens the hindsight teacher and improves feedback-derived supervision. We vary the observation-modeling weight λo∈0,0.1,0.2,0.5,1.0 _o∈\0,0.1,0.2,0.5,1.0\ while fixing the self-distillation weight, using Qwen3-4B-Instruct on τ-bench Retail. Table 3 and Figure 3(a) report student and privileged-teacher performance; the teacher evaluation protocol is detailed in the appendix. At the default λo=0.2 _o=0.2, the student achieves its highest SR of 38.7%, exceeding Base by 11.5 percentage points, while the teacher reaches 46.7%. The teacher peaks at 49.1% with λo=0.1 _o=0.1, but the corresponding student achieves only 36.5%, showing that higher privileged-teacher performance does not necessarily produce the best distilled student. Compared with no observation modeling (λo=0 _o=0), the default improves teacher SR from 30.1% to 46.7% and student SR from 29.4% to 38.7%, gains of 16.6 and 9.3 points, respectively. These results support the interpretation that feedback modeling strengthens the feedback-conditioned teacher and its ability to provide effective policy supervision, with the student gains demonstrating benefits for hindsight self-distillation, even when learning relies on observed environmental responses. Table 4: Effect of the self-distillation weight λa _a on τ-bench Retail and AppWorld with Qwen3-4B-Instruct. SR is reported as pass@4, and Turns denotes the average number of interaction turns. All metrics except Turns are reported as percentages. Setting τ-bench Retail AppWorld SR ↑ Turns ↓ Acc. ↑ TGC ↑ SGC ↑ Acc. ↑ Base 27.2 ± 1.4 10.5 ± 0.3 32 ± 1.8 16.67 ± 1.19 5.36 ± 0.62 11 ± 1.3 λa=0 _a=0 20.1 ± 2.1 11.2 ± 0.5 37 ± 2.6 5.52 ± 0.83 1.21 ± 0.34 9 ± 1.7 λa=0.1 _a=0.1 31.5 ± 1.3 12.0 ± 0.6 55 ± 1.5 26.67 ± 2.07 8.62 ± 0.91 39 ± 2.4 λa=0.2 _a=0.2 33.2 ± 1.8 10.7 ± 0.3 62 ± 2.2 29.91 ± 1.43 8.45 ± 1.16 47 ± 1.6 λa=0.5 _a=0.5 36.5 ± 0.9 10.2 ± 0.2 68 ± 1.3 31.10 ± 1.86 9.73 ± 0.78 54 ± 2.1 λa=1.0 _a=1.0 38.7 ± 1.5 10.4 ± 0.4 73 ± 1.7 32.14 ± 1.61 12.50 ± 0.96 56 ± 1.2 Self-distillation strengthens environmental feedback modeling, supporting the reverse direction of their mutually reinforcing relationship. We vary the self-distillation weight λa∈0,0.1,0.2,0.5,1.0 _a∈\0,0.1,0.2,0.5,1.0\ while fixing λo _o, using Qwen3-4B-Instruct on τ-bench Retail and AppWorld. We measure feedback prediction accuracy (Acc.) on a shared, fixed set of held-out transitions collected by a reference policy in each environment. All variants receive identical histories and actions and are evaluated against the same observed feedback under identical evaluation settings (see appendix for details). As shown in Table 4 and Figure 3(b), accuracy increases monotonically across the tested λa _a values, peaking at λa=1.0 _a=1.0: 73% on Retail and 56% on AppWorld, compared with 37% and 9% without self-distillation. Policy performance improves alongside prediction accuracy: increasing λa _a from 0 to 1.0 raises Retail SR from 20.1% to 38.7% and AppWorld TGC from 5.52% to 32.14%. Without self-distillation, task performance falls below Base on both benchmarks, highlighting the importance of the policy-learning objective. Together with the λo _o ablation, results support a mutually reinforcing relationship: feedback modeling improves feedback-derived policy supervision, while hindsight self-distillation enhances the shared model’s ability to predict action consequences. Feedback modeling benefits from policy learning while supporting it. Table 5: General capability evaluation on MMLU, MMLU-Pro, and IFEval using Qwen3-4B-Instruct. Results are mean ± standard deviation. Light and darker blue shading identify the SELF student and teacher, respectively. Higher is better; bold entries indicate the highest mean per column. Method Setting MMLU ↑ MMLU-Pro ↑ IFEval ↑ Base – 59.60 ± 0.18 34.38 ± 0.27 82.62 ± 0.35 GRPO Reward 59.61 ± 0.24 34.25 ± 0.36 82.19 ± 0.48 SDPO Observation 57.96 ± 0.31 31.02 ± 0.42 81.17 ± 0.53 SELF Observation 59.57 ± 0.22 34.49 ± 0.33 82.61 ± 0.41 SELF (teacher) Observation 59.71 ± 0.20 34.92 ± 0.29 82.58 ± 0.38 4.4 General Capability Retention (RQ3) Finally, we examine whether SELF’s gains in interactive environments come at the expense of broader capabilities by evaluating the Qwen3-4B-Instruct student and EMA teacher on MMLU, MMLU-Pro, and IFEval, covering general knowledge, reasoning, and instruction following (Table 5 and Figure 3). The SELF student closely matches the base model: MMLU changes from 59.60 to 59.57, MMLU-Pro from 34.38 to 34.49, and IFEval from 82.62 to 82.61, with absolute differences of at most 0.11 points. In contrast, observation-based SDPO reduces these scores to 57.96, 31.02, and 81.17, respectively. The EMA teacher similarly retains general performance. These results support capability retention rather than general capability improvements, showing that SELF’s interactive-task gains accompany minimal changes on the evaluated general benchmarks. 5 Conclusion We introduced SELF, a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation, using realized action consequences both as prediction targets for learning action–feedback relationships and as privileged context for self-teacher supervision. Experiments on τ-bench Retail and AppWorld show that SELF consistently outperforms existing feedback-learning methods and reward-trained GRPO while largely preserving the general capabilities of the base model. Our analysis further reveals a mutually reinforcing mechanism: environmental feedback modeling strengthens hindsight self-distillation, while self-distillation improves feedback prediction. These findings suggest that effective learning from environmental feedback requires not only conditioning on the observed feedback, but also explicitly learning the action–feedback regularities encoded in interaction. SELF therefore provides a simple framework for jointly learning how the environment responds and how the agent should act. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §3.2. Cai et al. (2026) G. Cai, K. Yang, S. He, Y. Li, S. Yang, J. Lv, and L. Feng Beyond next-observation prediction: agent-authored world modeling for sequential decision making. arXiv preprint arXiv:2606.25421. External Links: Link, Document Cited by: §2. Chhikara et al. (2025) P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav Mem0: building production-ready AI agents with scalable long-term memory. In ECAI 2025, p. 2993–3000. External Links: Document, Link Cited by: §B.2, §4.1. Choudhury and Sodhi (2025) S. Choudhury and P. Sodhi Better than your teacher: LLM agents that learn from privileged AI feedback. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §3.2. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §B.1, §3.2. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations, External Links: Link Cited by: 1st item, §4.1. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §B.1. Hübotter et al. (2026) J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. In International Conference on Machine Learning, External Links: Link Cited by: §B.2, §C.2, §1, §1, §2, §4.1. Jin et al. (2026) X. Jin, D. Wang, V. Lingam, and V. Kumar AHEAD: adaptive hindsight with environment-augmented distillation for agentic rl. arXiv preprint arXiv:2608.24114. External Links: Document Cited by: §2. Kleine Buening et al. (2026) T. Kleine Buening, J. Hübotter, B. Pásztor, I. Shenfeld, G. Ramponi, and A. Krause Aligning language models from user interactions. arXiv preprint arXiv:2603.12273. External Links: Document Cited by: §2. Lambert et al. (2025) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, X. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. Le Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi Tülu 3: pushing frontiers in open language model post-training. In Second Conference on Language Modeling, External Links: Link Cited by: §1. Liu et al. (2026a) F. Liu, Y. Miao, Z. He, Y. Liu, F. Sun, and M. Du DynaCF: mitigating shortcut learning in reward models via dynamic counterfactual sensitivity. arXiv preprint arXiv:2606.09043. External Links: Document, Link Cited by: §1. Liu et al. (2026b) H. Liu, Y. Zhang, X. Li, B. Lyu, and J. Shang HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation. arXiv preprint arXiv:2606.11559. External Links: Link Cited by: §2. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §B.1. Lu et al. (2026) N. Lu, B. Lin, S. Liu, J. Wu, H. Lv, Y. Wei, L. Zhu, S. Qian, X. Wang, Y. Chen, Q. Wang, and K. Tang Policy and world modeling co-training for language agents. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: §B.2, §C.2, §1, §1, §4.1. Shenfeld et al. (2026) I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal Self-distillation enables continual learning. In International Conference on Machine Learning, External Links: Link Cited by: §2. Shrivastava et al. (2026) V. Shrivastava, P. Kauffmann, A. Awadallah, and D. Papailiopoulos ECHO: terminal agents learn world models for free. arXiv preprint arXiv:2605.24517. External Links: Link Cited by: §2. Song et al. (2026) Y. Song, L. Chen, F. Tajwar, R. Munos, D. Pathak, J. A. Bagnell, A. Singh, and A. Zanette Expanding the capabilities of reinforcement learning via text feedback. arXiv preprint arXiv:2602.02482. External Links: Document Cited by: §2. Tarvainen and Valpola (2017) A. Tarvainen and H. Valpola Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, Vol. 30. External Links: Link Cited by: §D.3, §3.2, §4.1. Tian et al. (2026) J. Tian, F. Liu, J. Han, Y. Jiang, Y. Wu, Y. Liu, H. Li, F. Xu, and W. Li Auto-Rubric as reward: from implicit preferences to explicit multimodal generative criteria. arXiv preprint arXiv:2605.08354. External Links: Document, Link Cited by: §1. Trivedi et al. (2024) H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian AppWorld: a controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 16022–16076. External Links: Document, Link Cited by: 2nd item, §B.3, §1, §1, §4.1. Wang et al. (2026) Y. Wang, X. Chen, X. Jin, M. Wang, and L. Yang OpenClaw-rl: train any agent simply by talking. arXiv preprint arXiv:2603.10165. External Links: Document Cited by: §2. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: 2nd item, §4.1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: Link Cited by: 1st item, 2nd item, 3rd item, §1, §4.1, §4.1. Yang et al. (2026a) C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan Self-distilled rlvr. arXiv preprint arXiv:2604.03128. External Links: Document Cited by: §2. Yang et al. (2026b) Y. Yang, C. Qin, X. Liu, C. Chen, Q. Dong, Y. Zhang, C. Liu, Z. Yang, L. Pan, J. Lin, and Y. Feng Agentic reinforcement learning with observation-calibrated self-distillation. arXiv preprint arXiv:2608.04788. External Links: Link Cited by: §2. Yao et al. (2025) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ-bench: a benchmark for tool-agent-user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: 1st item, §1, §4.1. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §B.1, §1, §4.1. Ye et al. (2026) T. Ye, L. Dong, Q. Dong, X. Wu, S. Huang, and F. Wei Online experiential learning for language models. arXiv preprint arXiv:2603.16856. External Links: Link Cited by: §B.2, §C.2, §1, §2, §4.1. Zhang et al. (2026) K. Zhang, X. Chen, B. Liu, T. Xue, Z. Liao, Z. Liu, X. Wang, Y. Ning, Z. Chen, X. Fu, J. Xie, Y. Sun, B. Gou, Q. Qi, Z. Meng, J. Yang, N. Zhang, X. Li, A. Shah, D. Huynh, H. Li, Z. Yang, X. Cao, L. K. Jang, S. Zhou, J. Zhu, H. Sun, J. E. Weston, Y. Su, and Y. Wu Agent learning via early experience. In Forty-third International Conference on Machine Learning, Cited by: §2. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. In International Conference on Machine Learning, External Links: Link Cited by: §2. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. External Links: Link Cited by: 3rd item, §4.1. Appendix A Datasets, Benchmarks, and Models This section introduces the interactive environments, general capability benchmarks, and models used to evaluate SELF. Interactive environments assess learning from action consequences, while general capability benchmarks measure retention of knowledge, reasoning, and instruction following after agent training. A.1 Datasets and Benchmarks Interactive Benchmarks. We use τ-bench Retail and AppWorld to evaluate learning from agent–environment interactions. Learning uses the official training splits, and task performance is measured on held-out test splits. Training trajectories are collected by interacting with the environments, rather than provided as expert demonstrations. For SELF, the resulting observations serve as feedback-prediction targets and as hindsight context for self-distillation. Task-success annotations are reserved for evaluation. • τ-bench Retail. The retail domain of τ-bench evaluates agents through multi-turn conversations with simulated users and calls to domain-specific tools (Yao et al., 2025). Agents must interpret user requests, obtain relevant information, and perform actions consistent with retail policies. Tool responses reveal the consequences of actions and inform subsequent decisions, but successful execution of a tool call does not necessarily establish that the user’s overall request has been fulfilled. We use Retail to assess learning from tool feedback in policy-constrained interactions. • AppWorld. AppWorld evaluates agents that generate and execute code to complete tasks across multiple applications (Trivedi et al., 2024). Tasks combine application APIs with program logic, and execution outputs guide subsequent actions. A single code-based action can compose multiple operations, creating dependencies both within actions and across interaction steps. Programmatic tests assess the resulting application states, including task completion and unintended changes. We use AppWorld to assess learning from code-execution feedback, complementing Retail’s tool-based interactions. General Capability Benchmarks. MMLU, MMLU-Pro, and IFEval are used exclusively for evaluation. We compare the initial Qwen3-4B-Instruct checkpoint, the trained SELF student, and its EMA teacher to assess changes in general capabilities after agent training. Each checkpoint receives only the inputs provided by the corresponding benchmark. These evaluations measure the teacher’s general capabilities separately from its use of privileged hindsight information during interactive learning. • MMLU. MMLU evaluates knowledge and problem solving through multiple-choice questions across 57 subjects, including mathematics, history, computer science, and law (Hendrycks et al., 2021). Its broad subject coverage extends beyond the retail and application domains used for agent training. We use it to assess whether interactive adaptation preserves general knowledge and question-answering ability. • MMLU-Pro. MMLU-Pro emphasizes reasoning-intensive questions and increases the number of answer options to as many as ten (Wang et al., 2024). It complements MMLU with a more demanding assessment of reasoning and answer selection. We use it to examine whether these capabilities are retained after interactive training. • IFEval. IFEval evaluates instruction following through programmatically verifiable requirements, such as constraints on response length and keyword usage (Zhou et al., 2023). It assesses whether generated responses satisfy explicit instructions without relying on a subjective model judge. We use it to measure retention of instruction-following behavior beyond tool use and code execution. A.2 Models We use Qwen3-4B-Instruct and Qwen3-8B as trainable backbones to evaluate SELF across model scales, and include Qwen3-30B-A3B-Instruct as external comparisons. • Qwen3-4B-Instruct. Qwen3-4B-Instruct is an instruction-tuned language model in the Qwen3 family (Yang et al., 2025). We use it as a trainable backbone for the main interactive-task comparisons, coefficient ablations, mechanism analyses, and general capability evaluations. • Qwen3-8B. Qwen3-8B is an open-weight language model in the Qwen3 family (Yang et al., 2025). We use its non-thinking configuration as a second trainable backbone to evaluate SELF on Retail and AppWorld at a larger model scale. • Qwen3-30B-A3B-Instruct. Qwen3-30B-A3B-Instruct is an instruction-tuned mixture-of-experts model in the Qwen3 family (Yang et al., 2025). We report its performance on Retail and AppWorld to compare SELF with a larger model from the same family. Appendix B Implementation Details This section specifies the training hyperparameters, dataset partitions, and evaluation protocols used for SELF. Benchmark and model descriptions are provided in Appendix A, and the training objectives are derived in Appendix D.3. B.1 Hyperparameters Optimization Settings. We train Qwen3-4B-Instruct and Qwen3-8B using LoRA (Hu et al., 2022) with AdamW (Loshchilov and Hutter, 2019), a learning rate of 3×10−43× 10^-4, and a training batch size of 16. Qwen3-8B uses its non-thinking configuration. The feedback-modeling and hindsight-distillation weights are λo=0.2 _o=0.2 and λa=1.0 _a=1.0, respectively. Both objectives update the same student adapters. Table 6 summarizes the default configuration, including the backbone-specific LoRA settings. Interaction Collection and Optimization. Agents use a ReAct-style interface (Yao et al., 2023), selecting actions from the task instruction and accumulated interaction history. At the start of each training round, the current student collects trajectories containing transitions (hi,ai,fi)(h_i,a_i,f_i), where hih_i is the history preceding an action, aia_i is the executed action, and fif_i is the resulting environmental feedback. The collected trajectories remain fixed during optimization within that round. The updated student then collects the next round of interactions. Optimization samples a trajectory uniformly and then a turn uniformly from that trajectory. Within each transition, feedback-prediction and action-distillation losses are normalized separately by their target lengths. This corresponds to averaging over trajectories, turns, and target tokens, rather than giving longer trajectories greater outer weight. Recorded actions and feedback are treated as fixed data; gradients do not propagate through sampling or environment execution. Table 6: Default SELF training configuration. Backbone-specific LoRA settings are listed separately; other settings are shared. Hyperparameter Qwen3-4B-Instruct Qwen3-8B Optimizer AdamW Learning rate 3×10−43× 10^-4 Training batch size 16 LoRA rank r 16 96 LoRA alpha 32 128 LoRA dropout 0.05 0.05 Gradient checkpointing Enabled Feedback-modeling weight λo _o 0.2 Hindsight-distillation weight λa _a 1.0 Interaction interface ReAct-style Collection policy Current student at the start of each round Within-round data Fixed trajectory collection tT_t Action-prefix alignment Same recorded prefixes for student and teacher Teacher gradients Stopped during distillation Feedback-prediction loss Token-averaged negative log-likelihood Action-distillation loss Full-vocabulary reverse KL Loss averaging order Trajectories, turns, then target tokens Deployment Student action policy only Training Targets. Feedback prediction conditions on the history and executed action, and uses the realized environmental response as its target. During training, the response is scored through teacher forcing with token-averaged negative log-likelihood. For hindsight distillation, the student and teacher score the same recorded action prefixes. The student receives the available interaction history, while the teacher additionally receives the realized feedback. Their token distributions are aligned by position within the action, accounting for the different conditioning-context lengths. Distillation uses reverse KL from the student distribution to the detached teacher distribution over the full vocabulary (Gu et al., 2024). Feedback and action losses are evaluated only at their respective target positions; conditioning and padding tokens are excluded. At deployment, action generation uses only the student policy, without an additional feedback-prediction or teacher-scoring step. B.2 Dataset Setup Training and Test Partitions. We use the official training and test partitions of τ-bench Retail and AppWorld. Retail is evaluated on its official test split, and AppWorld is evaluated on test-normal. Training trajectories are collected through interactions with training tasks, rather than provided as expert solutions. Held-out test interactions do not contribute to parameter updates. Supervision Available During Learning. SELF learns from environmental observations, including ordinary execution outputs, error messages, and status information. It does not add separate task-success labels to either training objective. In the comparison settings, Observation supplies environmental responses; Full additionally supplies ground-truth information and explicit success/failure feedback; Reward supplies verifiable task rewards. OEL (Ye et al., 2026) and SELF use Observation, SDPO Hübotter et al. (2026) and Mem0 (Chhikara et al., 2025) are evaluated under both Observation and Full, and GRPO (Shao et al., 2024) uses Reward. Task-success annotations remain available to the evaluator for scoring held-out performance. Feedback-Prediction Evaluation Data. We construct a fixed transition collection for each interactive benchmark using the initial Qwen3-4B-Instruct model. This model runs over the benchmark’s evaluation tasks three times, producing recorded histories, actions, and actual environmental responses. The resulting transitions are shared across the feedback predictors being compared. These three collection passes define the diagnostic evaluation data; they are distinct from the four attempts per task used to calculate Retail pass@4pass@4. B.3 Evaluation Protocols Interactive Task Performance. The deployable student is evaluated on held-out tasks using only the information available before each action. Task completion is determined by the benchmark evaluator. Retail reports success rate (SR), calculated as pass@4pass@4, and average interaction turns (Turns). AppWorld reports its official Task Goal Completion (TGC) and Scenario Goal Completion (SGC) metrics (Trivedi et al., 2024). SR, TGC, and SGC are expressed as percentages. Retail Success Rate and Interaction Length. For each of the N test tasks, we conduct four independent attempts. Let sq,r∈0,1s_q,r∈\0,1\ indicate whether attempt r on task q succeeds. A task contributes to pass@4pass@4 if at least one attempt succeeds: SR=pass@4=100N∑q=1N[∑r=14sq,r≥1].SR=pass@4= 100N _q=1^N1\! [ _r=1^4s_q,r≥ 1 ]. (9) Thus, each task receives equal weight, regardless of how many of its attempts succeed. This measures success within four attempts, rather than the proportion of tasks successful in all four attempts. Let Lq,rL_q,r be the interaction length of attempt r on task q. We average over all attempts, including failures: Turns=14N∑q=1N∑r=14Lq,r.Turns= 14N _q=1^N _r=1^4L_q,r. (10) Turns is interpreted together with SR, since a short failed interaction does not indicate efficient task completion. Feedback-Prediction Accuracy. Let ℰbE_b denote the fixed transition collection for benchmark b obtained from the three collection passes above. For each recorded transition (hi,ai,fi)(h_i,a_i,f_i), the evaluated model generates a feedback prediction f^i f_i conditioned on hih_i and aia_i. The realized response fif_i is withheld from the predictor and used only as the reference for scoring. This evaluation measures generated feedback, rather than token likelihood under teacher forcing. GPT-5.6-Luna, with reasoning effort set to low, compares f^i f_i with fif_i and returns a binary judgment: ci=J(f^i,fi)∈0,1,Acc.b=100|ℰb|∑i∈ℰbci.c_i=J( f_i,f_i)∈\0,1\,\;Acc._b= 100|E_b| _i _bc_i. (11) Here, ci=1c_i=1 means that the judge considers the predicted feedback consistent with the actual feedback, and ci=0c_i=0 means that it does not. The judge receives only the two feedback texts, without the interaction history or action. Consequently, Acc. measures agreement with observed feedback, not whether the action itself is appropriate or completes the task. All recorded steps receive equal weight, so longer trajectories contribute more evaluation examples. The same transition collection is used for every predictor to separate prediction quality from differences in the trajectories generated by their own policies. Privileged-Teacher Task Evaluation. The privileged teacher is evaluated on complete tasks, using the same task-success criteria as the student. At each decision step, we save the current environment state. The student generates a trial action and executes it to obtain environmental feedback. The teacher then receives the student’s pre-action context augmented with that feedback. The student’s trial action is not included in the teacher prompt; the feedback is its only additional information relative to the student’s context. The environment is then restored to the saved pre-action state, and the teacher generates and executes its action from that state. The resulting teacher-action transition is added to the ongoing interaction history and determines the state for the next step. The student trial is used only to obtain the additional feedback; its state changes are rolled back. This process is repeated until the task terminates, so task success is evaluated on the trajectory executed by the teacher rather than by editing a completed student trajectory. Teacher task scores assess performance with privileged feedback unavailable to the student at its original decision. They are reported separately from deployable student scores and do not represent ordinary inference without access to additional environmental information. General Capability Retention. We evaluate the initial Qwen3-4B-Instruct model, the adapted student, and the teacher checkpoint on MMLU, MMLU-Pro, and IFEval. Each checkpoint receives the benchmark inputs without privileged environmental feedback. These evaluations assess knowledge, reasoning, and instruction following after agent training and are excluded from the training objectives. The teacher’s general capability scores are therefore distinct from its privileged interactive-task performance. Appendix C Additional Results C.1 Effect of Cross-Rollout Feedback Modeling Experimental Design. We examine whether the benefit of environmental feedback modeling depends on being coupled to the same interaction experience used for hindsight self-distillation. In standard SELF, the two objectives are computed from the same recorded transitions: the realized feedback of a transition is used both as privileged context for hindsight self-distillation and as the prediction target for environmental feedback modeling. We compare this setting with a cross-rollout variant, in which the two objectives are trained on transitions from different rollouts. Let ℬ=(hi,ai,fi)i=1BB=\(h_i,a_i,f_i)\_i=1^B denote a batch of transitions used for hindsight self-distillation. Standard SELF also computes environmental feedback modeling on the same batch, ℒSELF=λaB∑i=1Bℓsd(hi,ai,fi)+λoB∑i=1Bℓenv(hi,ai,fi).L_SELF= _aB _i=1^B _sd(h_i,a_i,f_i)+ _oB _i=1^B _env(h_i,a_i,f_i). (12) For the cross-rollout variant, we independently sample another batch ℬ′=(hi′,ai′,fi′)i=1BB =\(h _i,a _i,f _i)\_i=1^B from different collected rollouts and optimize ℒcross=λaB∑i=1Bℓsd(hi,ai,fi)+λoB∑i=1Bℓenv(hi′,ai′,fi′).L_cross= _aB _i=1^B _sd(h_i,a_i,f_i)+ _oB _i=1^B _env(h _i,a _i,f _i). (13) Importantly, each transition in ℬ′B preserves its original history, action, and realized feedback. Thus, the feedback-prediction task remains well defined: (hi′,ai′)→fi′(h _i,a _i)→ f _i is still an actual environment transition. The intervention only removes the requirement that feedback modeling and hindsight self-distillation operate on the same interaction experience. This comparison allows us to distinguish the general benefit of learning environmental feedback from the additional benefit of coupling feedback modeling and hindsight supervision on the same transitions. All other training settings are kept fixed. Each condition is evaluated in a single run. We report Retail success rate (SR; pass@4pass@4) and average interaction turns, together with AppWorld Task Goal Completion (TGC) and Scenario Goal Completion (SGC). Table 7: Effect of coupling environmental feedback modeling and hindsight self-distillation on the same rollout with Qwen3-4B-Instruct. In the cross-rollout condition, feedback modeling uses intact transitions from independently collected rollouts, while hindsight self-distillation uses the original rollout. Results are from one run per condition. SR, TGC, and SGC are percentages, with higher values indicating better performance. Turns is interpreted alongside SR. Bold values indicate the best completion scores. Condition Retail AppWorld SR ↑ Turns ↓ TGC ↑ SGC ↑ Base 27.1 10.7 17.00 5.56 GRPO 33.0 13.4 28.67 10.50 SELF (cross-rollout) 36.6 11.7 31.21 8.67 SELF (same-rollout) 38.6 10.3 32.16 12.44 Results and Analysis. Table 7 shows that environmental feedback modeling remains effective even when its training transitions are decoupled from those used for hindsight self-distillation. The cross-rollout variant achieves 36.6% SR on Retail and 31.21% TGC on AppWorld, substantially outperforming the unadapted backbone and remaining competitive with the full SELF objective. This result suggests that the benefit of feedback modeling does not arise solely from using the same transition for both objectives: learning to predict environmental responses from valid interaction experience can improve the shared model even when that experience comes from a different rollout. Using the same rollout for both objectives provides an additional benefit. Compared with cross-rollout training, standard SELF improves Retail SR from 36.6% to 38.6% while reducing average interaction length from 11.7 to 10.3 turns. On AppWorld, TGC increases from 31.21% to 32.16%, while SGC increases from 8.67% to 12.44%. These results indicate that environmental feedback modeling provides a broadly useful learning signal on its own, while coupling it with hindsight self-distillation on the same interaction experience yields further gains. C.2 Training Cost Measurement Setup. We compare SELF with OEL (Ye et al., 2026), SDPO (Hübotter et al., 2026) under the Observation setting, and GRPO (Shao et al., 2024) using four NVIDIA A100 GPUs 80 GB. For each method, we aggregate the training costs of its runs in Table 1 across the reported backbones and interactive benchmarks. The accounting includes interaction collection and optimization, but excludes final evaluation and additional ablation experiments. Table 8 reports local GPU-hours and wall time. GPU-hours sum the device time used across runs, while wall time sums the elapsed duration of the individual training runs. Table 8: Training costs for the main experiments in Table 1 using four NVIDIA A100 80 GB GPUs. Values are aggregated across the reported training runs and exclude final evaluation and additional ablations. Lower values indicate lower cost. Method GPU-hours Wall time(h) OEL 20.3 6.12 SDPO (Observation) 11.8 3.67 GRPO 41.2 12.46 SELF 16.4 4.93 Results and Analysis. SELF requires 16.4 GPU-hours and 4.93 hours of wall time. Compared with GRPO, it reduces GPU usage by 60.2% and wall time by 60.4%. It also uses 19.2% fewer GPU-hours and 19.4% less wall time than OEL. Relative to SDPO (Observation), SELF incurs an additional 4.6 GPU-hours and 1.26 hours, corresponding to increases of 39.0% and 34.3%, respectively. SELF therefore has a higher training cost than SDPO (Observation), while remaining less expensive than OEL and GRPO in these experiments. Its feedback-prediction and hindsight-distillation objectives reuse collected transitions without requiring additional environment execution to construct their targets. The additional computation is confined to training: deployment uses only the student action policy. Appendix D SELF Details This section presents the complete SELF training procedure, then explains the construction of hindsight supervision and the two learning objectives. We distinguish the data held fixed within a collection round from the model quantities recomputed at each optimization step, and specify how losses are aligned, normalized, and differentiated. D.1 Training Workflow Interaction Collection and Recorded Transitions. SELF alternates between collecting interactions and optimizing the student. At the start of round t, the current student interacts with the training environment through its deployable action policy. A recorded transition (hi,ai,fi)(h_i,a_i,f_i) contains the task instruction and pre-action history hih_i, the executed action aia_i, and the resulting environmental feedback fif_i. Earlier observations are included in hih_i; the response to aia_i becomes available only after that action is executed. The resulting trajectory collection tT_t remains fixed throughout the round’s optimization steps. SELF does not resample actions or rerun the environment to construct targets from these transitions. Instead, feedback prediction uses the recorded response as a target, and hindsight distillation uses that response as additional teacher context. After optimization, the updated student collects the next round. Student and Teacher Roles. The student uses shared trainable parameters θ for action generation and feedback prediction, with separate instructions for the two uses. In the LoRA implementation, both objectives update the same student adapters while the backbone weights remain frozen. The teacher, denoted by parameters θ¯ θ, is initialized from the student and maintained through EMA updates. Teacher outputs are detached during distillation. Algorithm 1 treats teacher maintenance as a separate step rather than a gradient-based update through the teacher. Sampling and Optimization. Each minibatch contains 16 transitions. To give trajectories equal outer weight, each transition is sampled by first selecting a trajectory uniformly from tT_t and then selecting a turn uniformly from that trajectory. Both losses are computed for each sampled transition and normalized by their respective target lengths before averaging across the minibatch. The resulting estimator matches the objective in Equation 28. Algorithm 1 uses K optimization steps per collection round. AdamW updates only the student parameters; teacher outputs, recorded prefixes, and environmental feedback are treated as constants for that update. The teacher distributions are recomputed using the current teacher checkpoint at subsequent steps, even though the underlying transitions remain unchanged. Default optimizer and LoRA settings are given in appendix. Algorithm 1 SELF: training with environmental feedback and hindsight distillation 1: Initial student parameters θ0 _0; training environment; rounds T; updates per round K; batch size B=16B=16 2: Loss weights λo=0.2 _o=0.2, λa=1.0 _a=1.0; learning rate η=3×10−4η=3× 10^-4; student, teacher, and feedback prompts 3: Initialize student θ←θ0θ← _0 and teacher θ¯←θ0 θ← _0 4: Initialize the AdamW optimizer for the student trainable parameters 5: for t=0,…,T−1t=0,…,T-1 do 6: Collect tT_t using the current student action policy 7: Store each transition (hi,ai,fi)(h_i,a_i,f_i); hold tT_t fixed in this round 8: for s=1,…,Ks=1,…,K do 9: Form ℬB by sampling B times: a uniform trajectory, then a uniform turn 10: for all (hi,ai,fi)∈ℬ(h_i,a_i,f_i) do 11: Compute student action distributions pi,jstup^stu_i,j at recorded prefixes 12: Compute detached teacher distributions pi,jteap^tea_i,j using hi,fi,ai,<jh_i,f_i,a_i,<j 13: Align student and teacher outputs by action-token position j 14: ℓsd(i)←1Li∑j=1LiDKL(pi,jstu∥sg[pi,jtea]) _sd^(i)← 1L_i _j=1^L_iD_KL\! (p^stu_i,j \|sg[p^tea_i,j] ) 15: Compute pi,kenvp^env_i,k using hi,ai,fi,<kh_i,a_i,f_i,<k 16: ℓenv(i)←−1Mi∑k=1Milogpi,kenv(fi,k) _env^(i)←- 1M_i _k=1^M_i p^env_i,k(f_i,k) 17: end for 18: ℒ^←1B∑i∈ℬ(λoℓenv(i)+λaℓsd(i)) L← 1B _i ( _o _env^(i)+ _a _sd^(i) ) 19: Clear student gradients and backpropagate ℒ L 20: Update student trainable parameters with AdamW at learning rate η 21: Update the teacher using the maintained EMA rule, without backpropagation 22: end for 23: end for 24: return Student action policy conditioned on the task and available history D.2 Hindsight Self-Distillation Constructing Hindsight Supervision. Consider a collected transition (hi,ai,fi)(h_i,a_i,f_i), where hih_i contains the task instruction and the interaction history before action aia_i, and fif_i is the feedback returned after its execution. The feedback records the realized consequence of the action, but does not directly specify a corrected action or a target policy. SELF constructs such supervision through a hindsight teacher that evaluates the recorded action with access to both the original task context and its observed consequence. Let ai=(ai,1,…,ai,Li)a_i=(a_i,1,…,a_i,L_i). At action position j, the student and teacher distributions are pi,jstu(v) p^stu_i,j(v) =πθ(v∣hi,ai,<j,promptstu), = _θ\! (v h_i,a_i,<j,prompt_stu ), (14) pi,jtea(v) p^tea_i,j(v) =πθ¯(v∣hi,fi,ai,<j,prompttea), = _ θ\! (v h_i,f_i,a_i,<j,prompt_tea ), (15) for vocabulary token v∈v . The teacher parameters θ¯ θ are maintained as an exponential moving average of the student. The task instruction remains available through hih_i, so hindsight reassessment is conditioned on what the agent was asked to accomplish, not merely on the contents of the feedback. The teacher and student evaluate the same recorded action prefix ai,<ja_i,<j. The complete action is not provided as additional teacher context; at position j, only its prefix is used to define the next-token prediction being compared. The teacher additionally receives fif_i, while the task and pre-action history remain available through hih_i. This places both distributions at the same action-token decision, with hindsight feedback as the teacher’s privileged information. The target is a learned distribution, not an externally supplied corrected action. Because feedback adds tokens to the teacher’s input, the student and teacher logits must be matched by position within the recorded action, not by absolute position in their full input sequences. Conditioning and padding positions are excluded from the comparison. In the formulas below, LiL_i and MiM_i denote the numbers of scored action and feedback tokens, respectively, and are assumed to be positive. Action-Prefix Alignment and Distillation. SELF transfers hindsight supervision through the length-normalized reverse KL ℓsd(hi,ai,fi)=1Li∑j=1LiDKL(pi,jstu∥sg[pi,jtea]), _sd(h_i,a_i,f_i)= 1L_i _j=1^L_iD_KL\! (p^stu_i,j\, \|\,sg[p^tea_i,j] ), (16) where sgsg denotes stop-gradient. The comparison uses distributions over candidate next tokens, rather than treating the recorded token ai,ja_i,j as a correct action label. Consequently, the sampled action determines the prefixes at which learning occurs without requiring that the sampled action itself be successful. For a fixed position, abbreviate p=pi,jstup=p^stu_i,j and q=sg[pi,jtea]q=sg[p^tea_i,j]. The local objective is DKL(p∥q)=∑v∈p(v)(logp(v)−logq(v)).D_KL(p\|q)= _v p(v) ( p(v)- q(v) ). (17) This objective allows the teacher to redistribute support across the vocabulary. Unlike hard-label imitation, it need not reinforce the recorded next token; the update depends on the teacher’s distribution relative to the student’s current prediction. The exact form of this update is derived below. Gradient Boundaries and Deployment. The observation fif_i enters the teacher context during training, but not the deployable student’s context for action aia_i. Distillation therefore transfers information through parameter updates rather than through access to future observations at inference time. Earlier observations remain available in hih_i, as in the original interaction process. Within each gradient update, the recorded transition and detached teacher prediction are fixed supervision. Gradients do not pass through the teacher, action sampling, or environment execution. When trajectories are reused across optimization steps, the recorded prefixes remain fixed even as the student parameters change. Equation 16 therefore defines distillation at behavior-policy prefixes, with the hindsight context supplied by each recorded transition. This training computation differs from the privileged-teacher full-task evaluation in appendix. Training scores recorded prefixes without executing teacher actions. In that diagnostic evaluation, a student trial provides feedback, the environment is restored to its pre-action state, and the teacher generates and executes its own action. Ordinary deployment uses neither this trial-and-restore procedure nor future feedback. D.3 SELF Objective We derive the feedback-prediction and hindsight-distillation gradients, then express their contributions to the shared student under the trajectory–turn–token weighting. Feedback Prediction as Forward KL. Hindsight supervision depends on the teacher’s ability to interpret the feedback it receives. SELF complements this supervision with a predictive objective that directly trains the shared model on action–consequence relationships. Write fi=(fi,1,…,fi,Mi)f_i=(f_i,1,…,f_i,M_i) and define pi,kenv(v)=penv,θ(v∣hi,ai,fi,<k,promptenv).p^env_i,k(v)=p_env,θ\! (v h_i,a_i,f_i,<k,prompt_env ). (18) These token distributions define the autoregressive feedback model penv,θ(fi∣hi,ai,promptenv)=∏k=1Mipi,kenv(fi,k).p_env,θ(f_i h_i,a_i,prompt_env)= _k=1^M_ip^env_i,k(f_i,k). (19) Let δenv,i,k _env,i,k be a point mass on the observed token fi,kf_i,k. This is an empirical token target from a recorded response, rather than the full distribution of possible environmental responses. Its entropy is zero, giving DKL(δenv,i,k∥pi,kenv) D_KL\! ( _env,i,k\, \|\,p^env_i,k ) =∑v∈δenv,i,k(v)logδenv,i,k(v)pi,kenv(v) = _v _env,i,k(v) _env,i,k(v)p^env_i,k(v) =−logpi,kenv(fi,k). =- p^env_i,k(f_i,k). (20) Here, zero-probability terms use the standard convention 0log0=00 0=0. Averaging over the feedback tokens yields ℓenv(hi,ai,fi) _env(h_i,a_i,f_i) =1Mi∑k=1MiDKL(δenv,i,k∥pi,kenv) = 1M_i _k=1^M_iD_KL\! ( _env,i,k\, \|\,p^env_i,k ) =−1Mi∑k=1Milogpi,kenv(fi,k) =- 1M_i _k=1^M_i p^env_i,k(f_i,k) =−1Milogpenv,θ(fi∣hi,ai,promptenv). =- 1M_i p_env,θ(f_i h_i,a_i,prompt_env). (21) Thus, forward-KL feedback modeling is exactly length-normalized maximum-likelihood learning from the realized environment response. For a feedback-prediction logit ui,k,vu_i,k,v, softmax differentiation gives ∂ℓenv∂ui,k,v=1Mi(pi,kenv(v)−v=fi,k). ∂ _env∂ u_i,k,v= 1M_i (p^env_i,k(v)-1\v=f_i,k\ ). (22) For the response recorded in a transition, the target is available regardless of whether the action ultimately contributes to task success. For example, an execution error remains a valid observation to predict. The objective learns what follows an action, rather than labeling the action as desirable. This distinction explains why feedback modeling complements, rather than replaces, policy learning. Deriving the Reverse-KL Gradient. At a fixed action position, write p=pi,jstup=p^stu_i,j and q=sg[pi,jtea]q=sg[p^tea_i,j]. Let zvz_v denote the student logit, with p(v)=exp(zv)/∑uexp(zu)p(v)= (z_v)/ _u (z_u). Assuming positive probabilities, define A(v)=logq(v)−logp(v),A¯=∑u∈p(u)A(u).A(v)= q(v)- p(v), A= _u p(u)A(u). (23) The quantity A(v)A(v) measures relative teacher support, and A¯=−DKL(p∥q) A=-D_KL(p\|q). Since q is detached, ∂DKL(p∥q)∂p(u)=logp(u)−logq(u)+1. ∂ D_KL(p\|q)∂ p(u)= p(u)- q(u)+1. (24) Combining this expression with the softmax Jacobian, ∂p(u)∂zv=p(u)(u=v−p(v)), ∂ p(u)∂ z_v=p(u) (1\u=v\-p(v) ), (25) gives ∂DKL(p∥q)∂zv ∂ D_KL(p\|q)∂ z_v =∑u∈p(u)(u=v−p(v))(logp(u)−logq(u)+1) = _u p(u) (1\u=v\-p(v) ) ( p(u)- q(u)+1 ) =p(v)[logp(v)−logq(v)−∑u∈p(u)(logp(u)−logq(u))] =p(v) [ p(v)- q(v)- _u p(u) ( p(u)- q(u) ) ] =−p(v)(A(v)−A¯). =-p(v) (A(v)- A ). (26) The constant terms cancel because ∑up(u)=1 _up(u)=1. For the transition-level distillation loss, the corresponding derivative is ∂ℓsd∂zi,j,v=−1Lipi,jstu(v)(Ai,j(v)−A¯i,j). ∂ _sd∂ z_i,j,v=- 1L_ip^stu_i,j(v) (A_i,j(v)- A_i,j ). (27) If logits were optimized independently, gradient descent would increase a token’s logit when its relative teacher support exceeds the student-weighted average, and decrease it when that support is below average. In the actual model, these logit derivatives propagate through shared trainable parameters, so their effects across tokens and examples are coupled. The centering arises from probability normalization, not from a reward-based advantage estimator. Although A resembles an advantage notation, it is computed entirely from the student and teacher distributions. This derivation explains how an observation can produce dense policy supervision without specifying a reference action. The usefulness of the correction depends on whether the feedback-conditioned teacher provides informative relative support for candidate tokens. The gradient expression characterizes how that support is transferred; it does not establish that every teacher reassessment improves task performance. Joint Objective and Effective Sample Weights. For a fixed collection tT_t, let nτn_τ be the number of turns in trajectory τ. Transition indices below are local to each trajectory; the trajectory subscript on (hi,ai,fi)(h_i,a_i,f_i) is omitted for readability. SELF combines the two transition-level objectives as follows: ℒSELF,t=1|t|∑τ∈t1nτ∑i=1nτ[λoℓenv(hi,ai,fi)+λaℓsd(hi,ai,fi)].L_SELF,t= 1|T_t| _τ _t 1n_τ _i=1^n_τ [ _o _env(h_i,a_i,f_i)+ _a _sd(h_i,a_i,f_i) ]. (28) For nonempty trajectories and scored sequences, the effective coefficients on individual feedback and action token losses are, respectively, wτ,i,kenv=λo|t|nτMi,wτ,i,jsd=λa|t|nτLi.w^env_τ,i,k= _o|T_t|\,n_τM_i, w^sd_τ,i,j= _a|T_t|\,n_τL_i. (29) Each trajectory receives equal outer weight, turns are averaged within their trajectory, and each objective is normalized by its own target length. Long feedback sequences therefore do not receive greater total weight merely because they contain more tokens. The coefficients λo _o and λa _a control the explicit weighting of the two normalized objectives, although their gradient magnitudes also depend on the model predictions. This formulation corresponds to the trajectory-then-turn sampling procedure in Algorithm 1. Uniformly sampling a flattened pool of transitions would instead favor trajectories containing more turns, unless equivalent weighting corrections were applied. Similarly, a single average over all tokens in a batch generally differs from averaging the length-normalized transition losses. Shared-Parameter Learning across Updates. Let genv g_env =1|t|∑τ∈t1nτ∑i=1nτ∇θℓenv(i), = 1|T_t| _τ _t 1n_τ _i=1^n_τ _θ _env^(i), (30) gsd g_sd =1|t|∑τ∈t1nτ∑i=1nτ∇θℓsd(i). = 1|T_t| _τ _t 1n_τ _i=1^n_τ _θ _sd^(i). With the collected data and teacher outputs fixed within the update, linearity gives ∇θℒSELF,t=λogenv+λagsd. _θL_SELF,t= _og_env+ _ag_sd. (31) The two gradients act on the same student parameters, but provide different supervision: genvg_env learns the observed consequence of the recorded action, whereas gsdg_sd transfers the teacher’s hindsight action distribution into the deployable policy. Their interaction also extends beyond the current optimization step. Feedback-modeling updates alter the student and, through subsequent EMA updates (Tarvainen and Valpola, 2017), the teacher used for future hindsight supervision. Policy distillation also changes these shared parameters and can therefore affect feedback prediction, although the direction of that effect is an empirical question. It can additionally change the actions and observations collected in later rounds. Because the teacher is detached, this interaction is not a same-step differentiable cycle. This decomposition describes how the two objectives update the shared model, while the coefficient studies provide empirical evidence for their complementary benefits. These benefits are not guaranteed by gradient addition alone; the component gradients may reinforce or oppose one another at different optimization steps. Appendix E Illustrative Prompts and Examples [q] Task instruction Fix the Python program analyze_sales.py. The script should read data/orders.csv, consider only orders whose status is completed, compute the average revenue of those orders, and print the result with exactly two decimal places. You may inspect files, execute Python or shell commands, and edit the implementation. Do not modify the tests. [o0o_0] Initial environment observation Working directory: /workspace/sales_analysis Repository contents: analyze_sales.py data/orders.csv tests/test_analyze_sales.py README.md [a1a_1] Agent action sed -n ’1,220p’ analyze_sales.py [f1f_1] Environmental feedback import csv def main(): total = 0.0 count = 0 with open("data/orders.csv") as f: reader = csv.DictReader(f) for row in reader: total += float(row["revenue"]) count += 1 print(f"Average revenue: total/count:.1f") if __name__ == "__main__": main() [a2a_2] Agent action head -5 data/orders.csv && pytest -q [f2f_2] Environmental feedback order_id,status,total 1001,completed,32.50 1002,cancelled,80.00 1003,completed,62.00 1004,pending,44.00 F ============================= FAILURES ============================= KeyError: ’revenue’ analyze_sales.py:10 [a3a_3] Agent action sed -n ’1,220p’ tests/test_analyze_sales.py [f3f_3] Environmental feedback def test_completed_orders_only(capsys): main() out = capsys.readouterr().out.strip() assert out == "Average completed-order revenue: 47.25" [a4a_4] Agent action Inspect all rows to verify which records should contribute: cat data/orders.csv … [f6f_6] Environmental feedback F ============================= FAILURES ============================= Expected: Average completed-order revenue: 47.25 Received: Average completed-order revenue: 47.2 1 failed in 0.08s [a7a_7] Agent action python - <<’PY’ from pathlib import Path p = Path("analyze_sales.py") s = p.read_text().replace(’:.1f’, ’:.2f’) p.write_text(s) PY [f7f_7] Environmental feedback Command completed successfully. [a8a_8] Agent action pytest -q [f8f_8] Environmental feedback . 1 passed in 0.07s [a9a_9] Agent action python analyze_sales.py [f9f_9] Environmental feedback Average completed-order revenue: 47.25 [a10a_10] Agent action git diff -- analyze_sales.py [f10f_10] Environmental feedback - total += float(row["revenue"]) - count += 1 + if row["status"] != "completed": + continue + total += float(row["total"]) + count += 1 - print(f"Average revenue: total/count:.1f") + print(f"Average completed-order revenue: total/count:.2f") [ana_n] Final agent response Implemented the fix in analyze_sales.py. The program now filters for completed orders, uses the total field, and prints the average with two decimal places. The test suite passes successfully. Example 1 --- Python execution Correct prediction Task q: Debug a Python script that parses a JSONL dataset. History hih_i: … The agent inspected the schema and found that each record contains id, text, and label. Current action aia_i: python validate_dataset.py data/train.jsonl Realized feedback fif_i: KeyError: ’user_id’ at validate_dataset.py:27 Predicted feedback f^i f_i: KeyError: ’user_id’ at validate_dataset.py:27 Result: The prediction correctly identifies both the exception type and the missing field. Example 2 --- Retail tool interaction Incorrect prediction Task q: Determine whether the user’s wireless-headphone order is eligible for return. History hih_i: … The user account has been verified and Order 1042 was identified as the relevant purchase. Current action aia_i: get_order_details(order_id="1042") Realized feedback fif_i: Order 1042 was delivered 12 days ago and remains within the return window. Predicted feedback f^i f_i: Order 1042 was cancelled before shipment and is eligible for an automatic refund. Result: The model predicts the wrong order state and therefore misinterprets the consequence of the action. Example 3 --- SQL database interaction Incorrect prediction Task q: Count the number of pending orders in the database. History hih_i: … The agent connected successfully to orders.db and verified the table schema. Current action aia_i: SELECT COUNT(*) FROM orders WHERE status=’pending’; Realized feedback fif_i: 17 Predicted feedback f^i f_i: 0 Result: The predicted database response is incorrect despite the action itself being syntactically valid. Example 4 --- Filesystem interaction Correct prediction Task q: Locate the configuration used by the training script. History hih_i: … The repository contains train.py, configs/, and several experiment directories. Current action aia_i: cat configs/base.yaml Realized feedback fif_i: model: qwen3-4b batch_size: 16 learning_rate: 1e-6 Predicted feedback f^i f_i: model: qwen3-4b batch_size: 16 learning_rate: 1e-6 Result: The predicted environmental feedback matches the observed file contents. E.1 Prompts Figure 4: Illustrative prompting interfaces used in SELF. The student prompt promptstuprompt_stu generates the next action aia_i from the interaction history hih_i; the teacher prompt promptteaprompt_tea additionally conditions on the realized environmental feedback fif_i as privileged hindsight information; and the environmental-feedback modeling prompt promptenvprompt_env predicts fif_i from hih_i and the current action aia_i.