Paper deep dive
Learning from Revision Consequences: Hindsight Meta-Experience Distillation for Self-Improving Agents
Qianhan Feng, Zhongzhen Huang, Yakun Zhu, Xiaofan Zhang, Qi Dou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As agents continuously improve by generating and revising Skills, the process that discovers and refines those Skills becomes a learnable object in its own right. Task-Skills directly act on task execution, whereas Meta-Skills govern how agents discover and improve future Skills; their value therefore emerges through the subsequent search processes they induce. Existing approaches improve Meta-Skills from observed raw Skill-search trajectories and branch outcomes. However, branch performance entangles the effects of the initial discovery state and the Meta-Skill revision that generated the search process, making it difficult to characterize what a particular revision actually changed, and pushing updates toward revisions that benefit from favorable states rather than those that improve the process. We introduce HMED (Hindsight Meta-Experience Distillation), a mechanism for constructing Meta-Experience for self-improving agents. HMED revisits the completed event from which a revision originates and re-executes the incumbent and revised Meta-Skills from the same restored discovery state, so that the changes associated with the revision can be observed under a shared condition. Each comparison is distilled into a Meta-Experience, a structured record that can be reused by future updates, so that even revisions that are not ultimately retained still contribute a learning signal. Across three interactive agent benchmarks and both open-source and closed-source models, HMED consistently improves Skill discovery performance over strong baselines, shifting Meta-Skill learning beyond branch outcomes toward the consequences of changing the improvement process.
Tags
Links
- Source: https://arxiv.org/abs/2610.07979v1
- Canonical: https://arxiv.org/abs/2610.07979v1
Trouble viewing inline? Open PDF directly →
Full Text
48,265 characters extracted from source content.
Expand or collapse full text
Learning from Revision Consequences: Hindsight Meta-Experience Distillation for Self-Improving Agents Qianhan Feng Zhongzhen Huang Yakun Zhu Xiaofan Zhang* Qi Dou*1 The Chinese University of Hong Kong 2 Shanghai Jiao Tong University 3 Shanghai Innovation Institue Abstract As agents continuously improve by generating and revising Skills, the process that discovers and refines those Skills becomes a learnable object in its own right. Task-Skills directly act on task execution, whereas Meta-Skills govern how agents discover and improve future Skills; their value therefore emerges through the subsequent search processes they induce. Existing approaches improve Meta-Skills from observed raw Skill-search trajectories and branch outcomes. However, branch performance entangles the effects of the initial discovery state and the Meta-Skill revision that generated the search process, making it difficult to characterize what a particular revision actually changed, and pushing updates toward revisions that benefit from favorable states rather than those that improve the process. We introduce HMED (Hindsight Meta-Experience Distillation), a mechanism for constructing Meta-Experience for self-improving agents. HMED revisits the completed event from which a revision originates and re-executes the incumbent and revised Meta-Skills from the same restored discovery state, so that the changes associated with the revision can be observed under a shared condition. Each comparison is distilled into a Meta-Experience, a structured record that can be reused by future updates, so that even revisions that are not ultimately retained still contribute a learning signal. Across three interactive agent benchmarks and both open-source and closed-source models, HMED consistently improves Skill discovery performance over strong baselines, shifting Meta-Skill learning beyond branch outcomes toward the consequences of changing the improvement process. 1 Introduction Agents increasingly improve by turning interaction into reusable capabilities. A household agent that repeatedly has to locate objects can accumulate a Skill for finding them, so that a lesson learned once is available later without being rediscovered (Shinn et al., 2023; Zhao et al., 2023; Wang et al., 2023). As agents operate in increasingly open environments, however, improvement is no longer limited to acquiring individual capabilities. An agent must also improve the process through which it discovers, evaluates, and refines those capabilities. We refer to the policy governing Skill discovery and evolution as a Meta-Skill, and to the capabilities it produces as Task-Skills. A Task-Skill determines how an agent performs a task and can usually be evaluated through task outcomes. A Meta-Skill instead shapes how future capabilities are generated. Its value is realized only through the search processes it subsequently guides. Existing approaches improve Meta-Skills using signals derived from Skill-search trajectories, descendant outcomes, and accumulated evolution histories (Fernando et al., 2023; Agrawal et al., 2025; Wang et al., 2026). These signals are attached to complete search branches rather than to the consequences of individual Meta-Skill revisions. A search branch reflects the combined influence of where the search started and how the improvement policy guided it. As a result, its final outcome does not directly reveal the contribution of a particular Meta-Skill revision: a favorable discovery state may make a weak revision appear effective, while a useful revision may appear ineffective when evaluated from a difficult starting point. In other words, a good branch is not necessarily a good improver. Figure 1: Learning signals in Skill evolution. Task-Skill methods update from task outcomes; existing Meta-Skill evolution updates from branch-level search signals, where the starting discovery state and the Meta-Skill are mixed; HMED updates from matched revision consequences, where the same discovery state is restored and only the Meta-Skill is varied. This limitation comes from a mismatch between the experience collected by existing Skill evolution methods and the experience required for learning a reusable improvement policy. A trajectory records what happened during one improvement process, whereas it does not characterize what consequences are associated with a particular revision. What Meta-Skill learning needs is an experience unit centered on the revision itself: what changed in the improvement process, and what followed from that change. We call this unit Meta-Experience. Conceptually, it links four aspects of a revision: the completed event that motivated it, the modification to the Meta-Skill, the changes observed in the subsequent search process, and the resulting changes in discovered capabilities. For example, a revision that adjusts the retrieval component of the Meta-Skill may show which prior experience was consulted differently and whether that change led to different discoveries. Unlike Task-level Experience, which describes whether an existing capability can solve a task, Meta-Experience describes what a change to the improvement process produced. We introduce Hindsight Meta-Experience Distillation (HMED) to construct and reuse this experience. HMED anchors each Meta-Skill revision to the completed improvement event that motivated it, restores the discovery state from which that event began, and re-runs the search under both the incumbent and revised Meta-Skills from that shared starting point. The resulting paired traces are distilled into a compact Meta-Experience record. This record enters the next round of Meta-Skill updates, where it can inform how future revisions are proposed. A comparison therefore serves two roles: it decides whether to retain the current revision, and it contributes a record that shapes later revisions. Because the record is kept regardless of whether the revision is adopted, even a rejected revision leaves a reusable learning signal. We evaluate HMED on three interactive agent benchmarks, using DeepSeek-v4-Flash and Qwen3-32B. HMED achieves the highest mean downstream performance in all six model–benchmark configurations, and an ALFWorld ablation shows that returning revision consequences to later updates provides additional gains over using comparison only for selection. When the learned Meta-Skill is frozen and the Task-Skill library is reset, it continues to guide effective discovery at new origins, indicating that HMED learns a reusable improvement policy rather than fitting the trajectories that produced it. Together, these results suggest that learning an improvement policy benefits from an experience representation aligned with the object being learned. Our contributions are threefold: • Meta-Experience for Self-Improving Agents. We identify the mismatch between branch-level Skill-evolution outcomes and the revision-level experience required for learning reusable improvement policies, and introduce Meta-Experience as a distinct experience unit that links the completed event, the revision, the observed process changes, and the resulting discovery outcomes of a Meta-Skill update. • Hindsight Meta-Experience Distillation. We introduce HMED, a mechanism that constructs Meta-Experience by grounding Meta-Skill revisions in completed improvement events, re-running the incumbent and revised Meta-Skills from matched discovery states, and distilling the resulting comparison into records that future updates can read. • Empirical Evidence for Learning from Revision Consequences. We evaluate HMED across interactive agent environments and distinguish the value of consequence feedback from comparison-based selection, showing that learned Meta-Skills remain effective beyond the trajectories that produced them. 2 Related Work 2.1 Self-Improving Agents and Evolving Improvement Procedures Agents increasingly transform interaction experience into reusable capabilities. Early work such as Reflexion converts failures into verbal feedback for future trials (Shinn et al., 2023), ExpeL distills reusable lessons from repeated interactions (Zhao et al., 2023), and Voyager maintains a growing Skill library that can be accumulated, retrieved, and composed (Wang et al., 2023). As the learned object expands from task behavior to the mechanisms that produce such behavior, self-improvement becomes increasingly recursive. PromptBreeder evolves the prompts that guide generation itself (Fernando et al., 2023), GEPA updates textual instructions through reflection on execution trajectories and feedback (Agrawal et al., 2025), and SkillEvolver iteratively refines reusable Skill artifacts through execution-driven improvement (Zhang et al., 2026a). Related directions in recursive self-improvement further explore agents that modify their own reasoning, search, or capability construction procedures (Zhang et al., 2025a; Kim et al., 2026; Zhang et al., 2026b). These approaches progressively shift the focus from improving what an agent does to improving how an agent improves. Recent work further advances this direction by making the improvement procedure itself an editable object. MetaSkill-Evolve introduces a two-timescale evolution framework in which Task-Skills and Meta-Skills co-evolve, with the Meta-Skill governing components such as failure analysis, experience retrieval, budget allocation, candidate proposal, and Skill revision (Wang et al., 2026). Its updates incorporate accumulated evolution signals such as descendant improvements and diagnoses, while maintaining branch-local Skill and Meta-Skill states. This line of work establishes the improver itself as a persistent object that can be represented, revised, and inherited. As the scope of learning shifts from task capabilities to the improvement process, a natural question emerges: what form of experience should guide the evolution of such an improver? 2.2 Learning Improvement Policies from Experience Learning how to improve a search or optimization procedure from previous experience has a long history in machine learning. Meta-learning studies how experience across tasks can produce better initializations or adaptation strategies (Finn et al., 2017; Ravi and Larochelle, 2017); learned optimizers parameterize update rules and learn optimization behaviors from prior training trajectories (Andrychowicz et al., 2016; Oh et al., 2020); and evolutionary computation explores adaptive operator selection and search strategies based on historical outcomes (Lange et al., 2023). These approaches share a common principle: a search process produces not only a solution, but also experience that can shape future search. Large language models extend this idea by representing improvement procedures as natural-language policies and open-ended agent workflows. EvoPrompt evolves natural-language prompts through evolutionary operators (Guo et al., 2024), while recent agent systems learn from execution feedback, memory, and accumulated experience to improve future behavior (Zhang et al., 2025b; Ouyang et al., 2025; Wu et al., 2025). As the optimized object moves from parameterized optimization rules toward language-based improvement policies that govern diagnosis, retrieval, proposal, and revision, the experience available to such policies becomes increasingly heterogeneous. A final search outcome indicates whether a branch succeeded, but it reflects both the starting discovery state and the improvement procedure that generated it. HMED instead organizes experience around individual Meta-Skill revisions within open-ended evolution. This distinction connects to several neighboring methodological areas. Hindsight Experience Replay makes failed attempts useful by relabeling the objective associated with an observed trajectory (Andrychowicz et al., 2017); in contrast, hindsight in HMED identifies which Meta-Skill revision should be revisited and which completed improvement event provides the basis for examining its consequences. Paired comparisons under shared conditions are widely used to reduce variance and improve interpretability in simulation and experimental design (Owen, 2013), while off-policy evaluation estimates the performance of one policy from observations generated by another (Thomas et al., 2015; Kallus, 2018; Agarwal et al., 2014). HMED follows a direct comparison perspective by running matched continuations rather than estimating the value of an unseen policy from logged trajectories. These directions motivate our study of revision-centered experience for learning language-based improvement procedures. 3 Method 3.1 Meta-Experience for Meta-Skills An agent that improves itself operates on two levels. At the task level, it executes a policy π on an environment and receives a task outcome r∈ℝr . At the meta level, it maintains a Meta-Skill m that governs how Task-Skills are discovered, evaluated, and revised across windows. A Task-Skill determines how the agent performs a task; a Meta-Skill instead shapes the capability-production process itself. Let x denote the discovery state at the start of a window: the Skill library, the visible history, and the current task context. Running Skill discovery from x under m produces an observed branch outcome y=f(x,m)y=f(x,m). The branch outcome is therefore a function of two arguments: the state the search started from and the Meta-Skill that guided it. This distinction creates a core difficulty for Meta-Skill learning: branch outcomes mix the two arguments of f. A revision may appear effective simply because it was evaluated from a favorable discovery state, and a useful revision may appear weak because it was evaluated from a difficult one. Increasing the number of branch observations does not resolve the mixture, because each branch still corresponds to a different x. As a result, Meta-Skill updates may select revisions that benefit from favorable states rather than revisions that improve the discovery process. Meta-Skill learning therefore requires a revision-centered form of experience, one that records the consequences of changing the improvement process. We call this experience Meta-Experience. Its structure is determined by what is needed to characterize a revision and its consequences: the completed event that motivated it, the modification it makes, the changes it produces in the subsequent search process, and the resulting changes in discovered capabilities. We write the record as ℰt=(et,Δmt,Δprocess,Δdiscovery),E_t=(e_t, m_t, _process, _discovery), where ete_t provides the revision’s provenance, Δmt m_t describes the Meta-Skill change, Δprocess _process captures how the intermediate discovery trajectory changed—for example, how prior experience was utilized, how candidates were generated, and whether a recurring failure was resolved—and Δdiscovery _discovery records the resulting change in discovered capabilities. The learning object of Meta-Experience is thus not task execution itself, but the consequences of changing the improvement process. 3.2 Hindsight Meta-Experience Distillation We propose Hindsight Meta-Experience Distillation (HMED), a mechanism that constructs Meta-Experience for Meta-Skill learning. HMED revisits a completed improvement event, restores the discovery state in which the revision was produced, and re-runs the search under the Meta-Skills before and after the modification from the same starting point. Here, hindsight refers to revisiting a completed revision after its consequences become observable, and distillation refers to converting the resulting paired traces into a compact update-level record that later revisions can read. One Meta-Skill update therefore proceeds in three steps: a revision is proposed and grounded in a completed event; the incumbent and revised Meta-Skills are re-run from matched discovery states; and the resulting comparison is distilled into a Meta-Experience record. The comparison produces two outputs: a selection signal for whether the revision is retained, and a Meta-Experience record for future updates. A candidate revision is first produced from the existing improvement process and bound to its source event, giving the modification a clear problem context and verification location. The incumbent and revised Meta-Skills are then re-run from matched discovery states, and the search differences they induce are compared. This comparison is finally distilled into Meta-Experience and returned to future Meta-Skill updates. One revision comparison therefore produces two outputs: a selection signal that determines whether the current revision continues to be used, and a Meta-Experience record that carries what the comparison revealed about the proposed change. Figure 2: Overview of Hindsight Meta-Experience Distillation (HMED). A Meta-Skill revision is anchored to the completed event that motivated it, revisited through matched continuations from the same restored state, and distilled into a Meta-Experience record that informs both the current adoption decision and future revisions. 3.3 Revision Grounding in Completed Events A Meta-Skill revision is the central event around which Meta-Experience is constructed, and the completed event it originates from provides the provenance of the resulting record. To make the revision consequence observable, each revision is bound to a completed Skill discovery attempt. A Skill discovery attempt refers to a segment of search run by the agent under a certain discovery state; it may succeed, fail, or be terminated early. When the agent observes a search bottleneck during this process and proposes a Meta-Skill modification accordingly, we record that attempt as an event. For example, a search may fail because the candidate directives do not cover the current task type; this failure is a discovery event. Grounding revisions in completed events is essential because Meta-Experience describes observed consequences rather than expectations about a proposed change: an attempt that has not finished has no search outcome against which the revision can be evaluated. Let the current Meta-Skill be mtm_t, the search window be WtW_t, and the historical Meta-Experience be ℰ<tE_<t. The candidate revision and its source event are represented as (mt′,et)=G(mt,Wt,ℰ<t),(m _t,e_t)=G(m_t,W_t,E_<t), where G denotes the Meta-Skill update procedure and ete_t denotes the completed discovery attempt corresponding to this revision. A revision modifies the Meta-Skill representation while preserving the underlying discovery framework; it is a textual change to the policy that governs search, not a replacement of the search procedure itself. The event ete_t preserves the context in which the revision was produced, enabling HMED to return to the location where the problem first appeared and check whether the modification changes subsequent search. Given the event, HMED restores its corresponding discovery state: xtsrc=Restore(et).x_t^src=Restore(e_t). This state asks: if the revision had been adopted at the location where the problem arose, would the subsequent discovery process have changed? To observe whether the revision has a more stable effect, HMED also considers an additional discovery state: t=xtsrc,xtcov.X_t=\x_t^src,x_t^cov\. The source state keeps the comparison consistent with the original problem, while the additional state supplies a bounded supplementary observation. Event grounding thus allows the revision to be examined at the location where the problem first appeared, and the subsequent comparison revolves around the improvement change that actually occurred. In the implementation, each component of ℰtE_t is instantiated from the corresponding comparison record. 3.4 Matched Meta Continuations Matched Meta Continuations are the core mechanism by which HMED constructs Meta-Experience. They hold the discovery state fixed while varying the Meta-Skill, so that the change associated with the revision can be observed under a shared reference condition. For example, in the failure scenario from the previous section, HMED returns to the discovery state at the time of that failure and re-runs the search with the Meta-Skills before and after the modification, recording whether the modified version avoids the same failure or produces a different search trajectory. For any discovery state x∈tx _t, HMED runs the current Meta-Skill mtm_t and the candidate revision mt′m _t separately: τ−(x)=(x,mt),τ+(x)=(x,mt′),τ^-(x)=S(x,m_t), τ^+(x)=S(x,m _t), where S denotes a Skill discovery continuation. The two continuations start from the same discovery state, so they share the same initial conditions, and the Meta-Skill is the controlled difference between them. The difference associated with the revision is therefore Δ(x)=g((x,mt′))−g((x,mt)), (x)=g(S(x,m _t))-g(S(x,m_t)), where g(⋅)g(·) computes the best-so-far utility improvement along the executed continuation steps. For multiple evaluation states, HMED aggregates: Δt=1|t|∑x∈tΔ(x). _t= 1|X_t| _x _t (x). Because the discovery state is held fixed, the resulting quantity reflects the change in subsequent search that is associated with the revision rather than the advantage of starting from a favorable state. HMED re-runs both Meta-Skills, so it compares the search difference that actually occurs under the matched condition rather than a quantity estimated from logged trajectories. Based on the matched comparison, the current Meta-Skill is updated according to the revision consequence: mt+1=mt′,Δt>0and the comparison passes validity checks,mt,otherwise.m_t+1= casesm _t,& _t>0\ and the comparison passes validity checks,\\ m_t,&otherwise. cases A comparison is treated as valid when both continuations complete normally, the candidate text is delivered on every step its own arm realizes, and the candidate differs textually from the incumbent. Comparisons that fail these conditions are recorded but not used to update the Meta-Skill. 3.5 Distilling Revision Consequences into Meta-Experience Distillation is the step that converts the paired traces from a matched comparison into a compact update-level record. An ordinary execution log describes what happened during search; Meta-Experience instead captures what changing the improvement process produced. HMED retains the information required to construct future Meta-Experience records. The resulting record uses the representation defined in Section 3.1, with ete_t providing the revision provenance, Δmt m_t describing the strategy change, Δprocess _process describing how the search process changed, and Δdiscovery _discovery describing how the capability discovery outcome changed. This record enters the future Meta-Skill update process: (mt+1′,et+1)=G(mt+1,Wt+1,ℰ≤t)(m _t+1,e_t+1)=G(m_t+1,W_t+1,E_≤ t) Historical Meta-Experience gives later revisions information about the consequences of past modifications, allowing new Meta-Skill updates to distinguish changes that altered discovery behavior from those that did not. A rejected revision still contributes a record: it captures what followed from a direction of change that was tried and found not to improve discovery. HMED retains the consequences of both adopted and rejected revisions, accumulating experience about how discovery procedures evolve. 4 Experiments 4.1 Experimental Setup Benchmarks. We evaluate on three interactive agent benchmarks covering task execution, open-ended exploration, and scientific discovery. ALFWorld (Shridhar et al., 2021) provides a text-based embodied environment in which agents complete everyday tasks through long-horizon planning, tool use, and environment feedback; task success depends on accumulating and reusing effective action strategies. ScienceWorld (Wang et al., 2022) provides a multi-step exploration setting in which agents iteratively test hypotheses and refine strategies, so a useful Skill encodes a procedure rather than a single answer. BiomniBench-DA (Qu et al., 2026) targets complex biomedical analysis tasks that require coordinating multi-step reasoning, tool use, and knowledge retrieval. Following the official setup, BiomniBench-DA results are scored by DeepSeek-v4-Flash; the other benchmarks are evaluated using the task success rates provided by the environment. Models and settings. We evaluate on Qwen3-32B (Qwen Team, 2025) and DeepSeek-v4-Flash. All methods run under identical inference configurations, with reasoning/thinking modes disabled. Across all experiments, each method interacts only with the training tasks specified by the benchmark and updates its Task-Skills or Meta-Skills through its own mechanism; the resulting Skill library and Meta-Skill are then frozen and evaluated on test tasks not used during evolution. Unless otherwise stated, results are averaged over three independent runs. Table 1: Downstream task performance after Skill evolution. ALFWorld and ScienceWorld report success rate (%). BiomniBench-DA reports a judge score out of 100. Model Method ALFWorld ScienceWorld BiomniBench-DA Qwen3-32B No Skill 45.15 16.68 14.33 Static Skill 47.76 21.55 16.36 SkillEvolver 49.25 20.59 18.33 MetaSkill-Evolve 51.49 20.84 17.80 HMED (ours) 54.11 22.56 20.78 DeepSeek-v4-Flash No Skill 71.46 23.58 24.70 Static Skill 73.51 26.33 22.84 SkillEvolver 76.87 24.30 25.67 MetaSkill-Evolve 76.49 28.70 36.80 HMED (ours) 80.60 32.47 43.19 4.2 Overall Skill Discovery Performance We evaluate whether HMED improves downstream task performance after Skill evolution. We compare with SkillEvolver (Zhang et al., 2026a), which continually updates Task-Skills, and MetaSkill-Evolve (Wang et al., 2026), which additionally allows the Meta-Skill itself to evolve. We additionally include No Skill and Static Skill as reference points measuring the contribution of accumulated Skill knowledge without an evolving mechanism. Table 1 reports the results. HMED achieves the highest mean in all six model–benchmark configurations. Relative to MetaSkill-Evolve, HMED improves ALFWorld from 51.49 to 54.11 (+2.62+2.62), ScienceWorld from 20.84 to 22.56 (+1.72+1.72), and BiomniBench-DA from 17.80 to 20.78 (+2.98+2.98) on Qwen3-32B. On DeepSeek-v4-Flash, HMED reaches 80.60 on ALFWorld, 32.47 on ScienceWorld, and 43.19 on BiomniBench-DA. These results indicate that organizing revision consequences as Meta-Experience is compatible with improved downstream performance across different agent backbones and environments. Two observations follow from the baseline ordering. First, Static Skill improves over No Skill on most benchmarks, indicating that accumulated historical Skills contribute to capability expansion; however, the benefit depends on whether the current task matches the existing library, as illustrated by the DeepSeek-v4-Flash BiomniBench-DA cell where Static Skill falls below No Skill. Second, SkillEvolver and MetaSkill-Evolve are not uniformly better than Static Skill, and MetaSkill-Evolve does not consistently outperform SkillEvolver. These comparisons suggest that how improvement signals are organized and reused can influence downstream self-improvement performance. 4.3 Ablation Study on Meta-Experience Construction and Use The main results show that HMED improves downstream performance. This section examines which aspects of Meta-Experience construction and use drive that improvement. Table 2 reports four variants, all retaining matched comparison. Table 2: Ablation of Meta-Experience construction and feedback. All variants retain matched comparison. Comparison-only removes the explicit feedback returned to later updates; Outcome-only restricts its content. Values are success rate (%) on ALFWorld. Variant Event Matched Consequence ALFWorld grounding comparison feedback HMED (full) ✓ ✓ ✓ 54.11 - Comparison-only ✓ ✓ × 49.25 - Outcome-only Meta-Experience ✓ ✓ Partial 51.49 - No hindsight grounding × ✓ ✓ 47.76 Selection versus learning. Matched comparison has two possible uses: deciding whether to adopt a revision, and informing how future revisions are proposed. The Comparison-only variant isolates the second by withholding revision-consequence feedback from later updates while retaining event grounding, matched continuations, and the adoption rule. According to Table 2, the full method exceeds this variant by 4.86 points. If the benefit of matched comparison were primarily due to selection, removing consequence feedback would lead to a smaller performance difference. This comparison isolates the contribution of consequence feedback under matched comparison. Content and source. Two further variants target the construction of the record. Removing event grounding reduces the score to 47.76, indicating that anchoring the comparison to the event that motivated the revision matters for downstream performance. Restricting the record to outcome-level information reduces the score to 51.49. Outcome-only remains competitive, suggesting that outcome information contributes substantially, while the additional process-level observations provide further benefit in this setting. 4.4 Analysis of Branch Outcomes and Revision Consequences The ablation shows which components of Meta-Experience matter. A separate question is whether the matched comparison merely reproduces the branch outcome that existing methods rely on. If branch outcome already captured the consequence of a revision, additional Meta-Experience feedback would provide limited additional information. For each evaluated revision, we compare the sign of its original branch outcome with the sign of its matched-continuation Δ . A revision whose branch outcome is positive but whose matched Δ is negative, or vice versa, is a case where branch outcome alone and matched comparison give opposite directional signals. Table 3: Agreement between branch outcome and matched-continuation Δ on Qwen3-32B. Agreement rate is computed over comparable cases (Agree + Disagree). Tie indicates that the matched Δ is zero or the sign cannot be determined. Benchmark Agree Disagree Tie Agreement rate ALFWorld 18 15 15 54.5% ScienceWorld 6 18 12 25.0% Table 3 reports the results. In ALFWorld, 45.5% of comparable cases show opposite signs; in ScienceWorld, the disagreement rate is 75.0%. These results indicate that branch outcome and matched-continuation evaluation often provide different directional signals. This supports treating Meta-Experience as a learning signal that contains information beyond branch outcome alone. The analysis characterizes the comparison signal; the feedback ablation above addresses its use in later updates. Appendix A provides a trace showing how a distilled Meta-Experience record is incorporated into a later update request and corresponds to a subsequent modification of the Meta-Skill. The trace illustrates that the record reaches the updater rather than being merely stored; it does not establish that the record alone caused the subsequent gain. 4.5 Reuse of Learned Meta-Skills under New Discovery Origins The preceding sections evaluate the full evolution pipeline. A separate question is whether the learned Meta-Skill is a reusable improvement policy or a fit to the trajectories that produced it. We test this by freezing the Meta-Skill and resetting the Task-Skill library before new discovery. In this protocol, we reuse the final Meta-Skill produced by each method on the training origins; the agent then starts from an initial Skill library on the held-out set and runs one complete new round of Skill discovery guided by the frozen Meta-Skill; the resulting Skill library is frozen and evaluated on the new test set. The held-out and new test sets are obtained by randomly partitioning the test set into two disjoint halves, neither of which overlaps with the training tasks; this prevents the reuse phase from adapting directly to the final evaluation tasks. Table 4: Reuse of learned Meta-Skills on unseen discovery origins in ALFWorld, reults reported in success rate (%) on new set. The learned Meta-Skill is frozen and the Task-Skill library is reset before new discovery. Model No Skill Static Skill SkillEvolver MetaSkill-Evolve HMED (ours) Qwen3-32B 41.05 43.28 42.54 49.25 50.41 DeepSeek-v4-Flash 68.65 70.15 73.13 76.12 76.67 Table 4 reports the results. HMED achieves the highest mean on both base models, with margins of 1.16 and 0.55 points over MetaSkill-Evolve. Since the Task-Skill library is reset before reuse, the remaining performance difference provides evidence that the learned Meta-Skill contributes beyond the stored Skill library. The learned Meta-Skill remains effective under the reuse protocol at new discovery origins, suggesting that HMED captures an improvement strategy that transfers beyond the original discovery trajectory. 5 Conclusion Self-improving agents increasingly accumulate experience from interaction to discover and revise Skills. As the improvement process itself becomes an editable object, an important question arises: what form of experience should guide Meta-Skill learning? We introduce Hindsight Meta-Experience Distillation (HMED), which shifts the experience unit for Meta-Skill learning from search-branch outcomes to the consequences of Meta-Skill revisions on subsequent Skill discovery. HMED anchors each revision to the completed event that motivated it, compares the incumbent and revised Meta-Skills from matched discovery states, and distills the resulting revision consequences into a revision-centered Meta-Experience record. The comparison therefore provides both an adoption signal for the current revision and a Meta-Experience record for future updates. Across three interactive agent benchmarks and two base model families, HMED consistently improves downstream Skill discovery performance over strong baselines. Further experiments show that its benefit comes not only from comparison-based selection, but also from the learning signal provided by Meta-Experience, and that learned Meta-Skills remain effective when reused under new discovery origins. Together, these results suggest that once the improvement process becomes an editable object, these results support organizing experience around revision consequences in self-improving agents. References Agarwal et al. (2014) A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire Taming the monster: a fast and simple algorithm for contextual bandits. In Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 32, p. 1638–1646. External Links: Link Cited by: §2.2. Agrawal et al. (2025) L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab GEPA: reflective prompt evolution can outperform reinforcement learning. External Links: 2507.19457, Link Cited by: §1, §2.1. Andrychowicz et al. (2016) M. Andrychowicz, M. Denil, S. Gómez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. de Freitas Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, External Links: 1606.04474 Cited by: §2.2. Andrychowicz et al. (2017) M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba Hindsight experience replay. In Advances in Neural Information Processing Systems, External Links: 1707.01495 Cited by: §2.2. Fernando et al. (2023) C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel Promptbreeder: self-referential self-improvement via prompt evolution. External Links: 2309.16797, Link Cited by: §1, §2.1. Finn et al. (2017) C. Finn, P. Abbeel, and S. Levine Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, External Links: 1703.03400 Cited by: §2.2. Guo et al. (2024) Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y. Yang Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations, External Links: 2310.06201 Cited by: §2.2. Kallus (2018) N. Kallus Instrument-armed bandits. In Proceedings of Algorithmic Learning Theory, Proceedings of Machine Learning Research, Vol. 83, p. 529–546. External Links: Link Cited by: §2.2. Kim et al. (2026) Z. M. Kim, Y. Lee, S. Jwa, and D. Kang Metan^n: recursive self-improvement through emergent depth. External Links: 2608.24735, Link Cited by: §2.1. Lange et al. (2023) R. T. Lange, Y. Zhang, E. Aleti, et al. Evolving reinforcement learning algorithms. In International Conference on Learning Representations, External Links: 2301.09846 Cited by: §2.2. Oh et al. (2020) J. Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. van Hasselt, S. Singh, and D. Silver Discovering reinforcement learning algorithms. In Advances in Neural Information Processing Systems, External Links: 2007.08794 Cited by: §2.2. Ouyang et al. (2025) S. Ouyang, J. Yan, I. Hsu, Y. Chen, K. Jiang, Z. Wang, R. Han, L. T. Le, S. Daruki, X. Tang, V. Tirumalashetty, G. Lee, M. Rofouei, H. Lin, J. Han, C. Lee, and T. Pfister ReasoningBank: scaling agent self-evolving with reasoning memory. External Links: 2509.25140, Link Cited by: §2.2. Owen (2013) A. B. Owen Monte carlo theory, methods and examples. Stanford University. External Links: Link Cited by: §2.2. Qu et al. (2026) Y. Qu, Y. Lu, X. Tu, S. Zhang, T. She, A. G. Shaw, J. Shih, B. Zhao, M. Shen, H. Yang, J. Yan, R. Zhang, X. Wu, T. Li, B. Zhou, N. Wang, A. Ma, L. Cong, X. Hu, Y. Jiang, J. Dong, T. Peng, J. Leskovec, and K. Huang BiomniBench: process-level evaluation of LLM agents for real-world biomedical research. bioRxiv. External Links: Document, Link Cited by: §4.1. Qwen Team (2025) Qwen Team Qwen3 technical report. External Links: 2505.09388, Link Cited by: §4.1. Ravi and Larochelle (2017) S. Ravi and H. Larochelle Optimization as a model for few-shot learning. In International Conference on Learning Representations, External Links: Link Cited by: §2.2. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §2.1. Shridhar et al. (2021) M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht ALFWorld: aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. Thomas et al. (2015) P. Thomas, G. Theocharous, and M. Ghavamzadeh High confidence policy improvement. In Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 37, p. 2380–2388. External Links: Link Cited by: §2.2. Wang et al. (2023) G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar Voyager: an open-ended embodied agent with large language models. External Links: 2305.16291, Link Cited by: §1, §2.1. Wang et al. (2022) R. Wang, P. Jansen, M. Côté, and P. Ammanabrolu ScienceWorld: is your agent smarter than a 5th grader?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, External Links: 2203.07540, Link Cited by: §4.1. Wang et al. (2026) Z. Wang, M. Yan, J. Bi, S. Yan, V. Tresp, and Y. Ma MetaSkill-Evolve: recursive self-improvement of LLM agents via two-timescale meta-skill evolution. External Links: 2607.05297, Link Cited by: §1, §2.1, §4.2. Wu et al. (2025) R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y. Shen, Y. Wang, and B. Shi EvolveR: self-evolving LLM agents through an experience-driven lifecycle. External Links: 2510.16079, Link Cited by: §2.2. Zhang et al. (2026a) G. Zhang, E. Zhu, J. Zhou, C. Jia, and H. Wang SkillEvolver: skill learning as a meta-skill. External Links: 2605.10500, Link Cited by: §2.1, §4.2. Zhang et al. (2025a) J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune Darwin gödel machine: open-ended evolution of self-improving agents. External Links: 2505.22954, Link Cited by: §2.1. Zhang et al. (2026b) J. Zhang, B. Zhao, W. Yang, J. Foerster, J. Clune, M. Jiang, S. Devlin, and T. Shavrina Hyperagents. External Links: 2603.19461, Link Cited by: §2.1. Zhang et al. (2025b) Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V. Kamanuru, J. Rainton, C. Wu, M. Ji, H. Li, U. Thakker, J. Zou, and K. Olukotun Agentic context engineering: evolving contexts for self-improving language models. External Links: 2510.04618, Link Cited by: §2.2. Zhao et al. (2023) A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang ExpeL: LLM agents are experiential learners. External Links: 2308.10144, Link Cited by: §1, §2.1. Appendix A Case Study: Tracing Meta-Experience Through the Update Loop This appendix traces how a distilled Meta-Experience record is incorporated into later updates rather than merely stored. We analyze one training run of DeepSeek-v4-Flash on ALFWorld (seed 2). A Meta-update window is one complete cycle of revision proposal, matched evaluation, and updater update; the run contains four such windows. The case is chosen by position rather than by outcome: we take the earliest complete chain in the run and include the last comparison in the same run as a counterpoint. All numbers and texts are quoted directly from the run records. Throughout this appendix, Gain denotes the best-so-far utility improvement relative to the starting utility under the same continuation: maxhuh−ustart _hu_h-u_start, where the maximum is taken over the executed continuation steps. Steps reports executed steps over allowed steps. Each row in a comparison table corresponds to a single continuation A.1 Anadopted Chain: Window 0 to Window 1 Step 1: The original comparison record. Window 0 produced the first Meta-Skill comparison in this run. Table 5 lists its key fields. Table 5: Comparison record at window 0. The candidate gains a positive Δ at the source state and zero at the coverage state. Start Continuation ustartu_start Steps ubestu_best Gain 1 (source) incumbent m 0.80 2/2 [0.80, 0.80] 0.0000 1 (source) candidate m′m 0.80 2/2 [0.84, 0.84] 0.0400 2 (coverage) incumbent m 0.84 2/2 [0.84, 0.84] 0.0000 2 (coverage) candidate m′m 0.84 2/2 [0.84, 0.84] 0.0000 The outcome is that the candidate isadopted. The record contains both starting states, the per-arm best-so-far trajectories, the gains, the step counts, and the inheritance decision. This record becomes the Meta-Experience that the next window’s updater receives. Step 2: How the next revision uses the Meta-Experience. Window 1’s candidate rewrites three of the five components: the allocator, the proposer, and the evolver. The new allocator rule is conditioned on gain values of the same form as those reported in window 0’s record, as shown in Table 6. Table 6: Correspondence between the gains in window 0’s record and the condition in window 1’s allocator rule. Gains in window 0’s record Condition in window 1’s allocator rule source: +0.0400+0.0400; coverage: 0.00000.0000 “If the previous window produced exactly one archived child with a small positive Δ (≤0.05≤ 0.05) and at least one non-archived child, keep K at 2 but require the two children to differ in edit action.” The phrase “small positive Δ ” is numerically compatible with the +0.0400+0.0400 and 0.00000.0000 in the record. The condition also refers to the archive status of Task-Skill children, which the paired gains alone do not determine. We therefore read this as a correspondence between the record’s content and a later rule rather than a derivation of the rule from the record alone. The same candidate additionally introduces an evolver instruction that prefers modify over add, with a rationale consistent with the pattern in the record, where the candidate gained at the source state and was flat at the coverage state. Step 3: The next comparison. Window 1’s own comparison provides a subsequent positive outcome for that revision, as shown in Table 7. Table 7: Comparison result at window 1. The candidate outperforms the incumbent at both starting states; ΔJ=+0.04 J=+0.04, and the candidate isadopted. Start Continuation ustartu_start Steps ubestu_best Gain 1 (source) incumbent m 0.76 2/2 [0.76, 0.76] 0.0000 1 (source) candidate m′m 0.76 2/2 [0.80, 0.84] 0.0600 2 (coverage) incumbent m 0.80 2/2 [0.80, 0.84] 0.0200 2 (coverage) candidate m′m 0.80 2/2 [0.84, 0.84] 0.0400 The candidate text is delivered at every step; retrieval and proposals differ from the incumbent arm at both states, while the failure tag, the allocator width, and the operation type remain the same. The candidate isadopted and becomes the incumbent for the following windows. Step 4: Evidence that the record reached the updater. Window 0’s comparison is written into a feedback block that appears verbatim in both of window 1’s requests. This establishes that the record was included in the updater’s input; it does not by itself establish that the record caused the subsequent proposal. A.2 A Non-adopted Revision The last update in the same run provides a counterpoint. Window 2’s comparison is shown in Table 8. The candidate leads by a wide margin at the source state but loses at the coverage state, resulting in a tie and keeping the incumbent. Table 8: Comparison record at window 2. The candidate leads at the source state but falls behind at the coverage state; the outcome is a tie. Start Continuation ustartu_start Steps ubestu_best Gain 1 (source) incumbent m 0.76 2/2 [0.84, 0.84] 0.0800 1 (source) candidate m′m 0.76 2/2 [0.88, 0.88] 0.1200 2 (coverage) incumbent m 0.76 1/2 [0.88] 0.0600 2 (coverage) candidate m′m 0.76 2/2 [0.80, 0.84] 0.0200 Window 3’s candidate receives a feedback block containing the comparisons from windows 1 and 2, and proposes a sharpening of the rules for inventory_loop failures. That candidate loses in its own comparison, as shown in Table 9. Table 9: Comparison result at window 3. The candidate ties with the incumbent at the source state and loses at the coverage state; ΔJ=−0.03 J=-0.03, and the candidate is rejected. Start Continuation ustartu_start Steps ubestu_best Gain 1 (source) incumbent m 0.76 2/2 [0.84, 0.84] 0.0800 1 (source) candidate m′m 0.76 2/2 [0.84, 0.84] 0.0800 2 (coverage) incumbent m 0.76 1/2 [0.88] 0.1200 2 (coverage) candidate m′m 0.76 2/2 [0.80, 0.84] 0.0600 The candidate is notadopted, and its changes do not enter the final Meta-Skill. This counterpoint shows that the feedback channel carries information rather than a guarantee of success: the updater proposed a new revision after receiving the record, while the final adoption was still decided by the matched comparison. A.3 Summary of the Case Study Together, the two chains illustrate the Meta-Experience loop of HMED: • a comparison is recorded and distilled into Meta-Experience; • the record is included in a later update request, and part of its content corresponds to a subsequent proposal; • the new proposal is again tested by a matched comparison, and is either adopted or rejected. The case study provides record-level evidence that Meta-Experience is incorporated into the update loop rather than merely stored. It illustrates how the distillation and return of revision consequences enter a later proposal, without establishing that this return is the sole cause of any downstream outcome.