Paper deep dive
Learning to Accumulate Knowledge with Mutual Information
Yuyang Zhao, Lizi Liao, Leyang Shen, Xiaoyan Zhao, Yang Zhang, Fuli Feng, Xiangnan He
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/11/2026, 3:40:52 AM
Summary
The paper introduces Knowledge Weaver, a reinforcement learning framework that trains a language model curator to accumulate reusable knowledge from agent trajectories. It addresses the redundancy problem in existing methods by combining token-wise mutual information (MI) feedback with marginal success rewards. This approach encourages the curator to generate distinct, complementary knowledge entries rather than general, overlapping guidance. The method achieves significant performance improvements on ALFWorld and WebShop benchmarks compared to Group Relative Policy Optimization (GRPO) and other baselines.
Entities (7)
Relation Signals (6)
Knowledge Weaver → evaluatedon → ALFWorld
confidence 99% · On ALFWorld and WebShop, Knowledge Weaver achieves mean success rates...
Knowledge Weaver → evaluatedon → WebShop
confidence 99% · On ALFWorld and WebShop, Knowledge Weaver achieves mean success rates...
Knowledge Weaver → incorporates → Mutual Information
confidence 95% · We couple feedback inspired by token-wise mutual information (MI) with marginal success rewards...
Knowledge Weaver → outperforms → Group Relative Policy Optimization
confidence 95% · Knowledge Weaver achieves mean success rates of 54.0% and 42.0%... exceeding GRPO by 16.9 and 18.7 percentage points, respectively.
Knowledge Weaver → uses → Group Relative Policy Optimization
confidence 90% · Therefore, we propose Knowledge Weaver, a reinforcement learning framework... We retain GRPO for policy optimization...
Knowledge Weaver → usesbackbonemodel → Qwen3.5-4b
confidence 90% · We use Qwen3.5-4B as the training model.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model (LLM) agents can improve their performance by reusing knowledge distilled from past interactions. However, curating new experiences into a knowledge bank that becomes more useful as it grows remains challenging. Effective knowledge accumulation should limit redundant overlap among entries and ensure that new knowledge contributes beyond what the bank already provides. Yet training a curator with Group Relative Policy Optimization (GRPO) on standalone task success can reinforce general guidance even when it duplicates existing knowledge. Therefore, we propose Knowledge Weaver, a reinforcement learning framework that trains a language model to curate reusable knowledge from agent trajectories. We couple feedback inspired by token-wise mutual information (MI) with marginal success rewards to guide knowledge accumulation. Together, these signals encourage the curator to preserve distinct information from experience and produce entries that improve task success when added to existing knowledge. Standalone success rewards also favor entries that are useful on their own. On ALFWorld and WebShop, Knowledge Weaver achieves mean success rates of 54.0\% and 42.0\% with k=10 retrieved entries, exceeding GRPO by 16.9 and 18.7 percentage points, respectively. Its knowledge banks also outperform the evaluated prompt-based and established banks, including human-written banks, in overall ALFWorld success rate and WebShop score with the executor frozen. Our codebase is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.10042v1
- Canonical: https://arxiv.org/abs/2610.10042v1
Trouble viewing inline? Open PDF directly →
Full Text
78,089 characters extracted from source content.
Expand or collapse full text
Preprint. Under review. LEARNING TO ACCUMULATE KNOWLEDGE WITH MUTUAL INFORMATION Yuyang Zhao 1 Lizi Liao 2 Leyang Shen 3 Xiaoyan Zhao 3 Yang Zhang 3 Fuli Feng 1 Xiangnan He 1 1 University of Science and Technology of China 2 Singapore Management University 3 National University of Singapore ABSTRACT Large language model (LLM) agents can improve their performance by reusing knowledge distilled from past interactions. However, curating new experiences into a knowledge bank that becomes more useful as it grows remains challenging. Effective knowledge accumulation should limit redundant overlap among entries and ensure that new knowledge contributes beyond what the bank already pro- vides. Yet training a curator with Group Relative Policy Optimization (GRPO) on standalone task success can reinforce general guidance even when it duplicates existing knowledge. Therefore, we propose Knowledge Weaver, a reinforcement learning framework that trains a language model to curate reusable knowledge from agent trajectories. We couple feedback inspired by token-wise mutual in- formation (MI) with marginal success rewards to guide knowledge accumulation. Together, these signals encourage the curator to preserve distinct information from experience and produce entries that improve task success when added to existing knowledge. Standalone success rewards also favor entries that are useful on their own. On ALFWorld and WebShop, Knowledge Weaver achieves mean success rates of 54.0% and 42.0% with k = 10 retrieved entries, exceeding GRPO by 16.9 and 18.7 percentage points, respectively. Its knowledge banks also outperform the evaluated prompt-based and established banks, including human-written banks, in overall ALFWorld success rate and WebShop score with the executor frozen. Our codebase is available at https://github.com/LaoKuiZe/Knowledge-Weaver. 1INTRODUCTION Large language model (LLM) agents interact with external environments to solve complex tasks (Yao et al., 2022b; Gur et al., 2024; Yang et al., 2024; Shen et al., 2025). Reusing knowledge from these interactions offers a route to agent self-improvement, allowing experience from previous tasks to inform future decisions. Prior work shows that high-quality, task-specific knowledge can improve agent performance (Wang et al., 2024; Li et al., 2026; Ouyang et al., 2026). Such knowledge includes reusable workflows and executable skills that guide action sequences (Wang et al., 2023; 2024; Yuan et al., 2024), as well as distilled lessons about environmental constraints and effective strategies (Majumder et al., 2023; Chen et al., 2024). Stored in external knowledge banks, relevant entries can be retrieved and incorporated into the agent’s context without updating the executor’s parameters (Zheng et al., 2024; Fu et al., 2024). This makes knowledge accumulation a means of retaining experience beyond individual interactions and supporting its reuse across tasks. While these benefits establish the value of using knowledge, producing it reliably remains a separate challenge. Existing work relies on pipeline design or training-based methods for knowledge cura- tion. Prompt-based methods extract reflective lessons, conditional guidelines, and reusable work- flows from trajectories (Shinn et al., 2023; Zhao et al., 2024; Fu et al., 2024; Wang et al., 2024), then consolidate them through memory or playbook revision (Zhang et al., 2026a; Ouyang et al., 2026; Suzgun et al., 2026). Training-based methods integrate knowledge into policy improvement through skill internalization (Lu et al., 2026) or the joint evolution of task execution and reusable skills (Xia et al., 2026; Shi et al., 2026; Yao et al., 2026). When curation is rewarded through downstream task success, this feedback assesses whether generated knowledge helps execution. However, these 1 arXiv:2610.10042v1 [cs.AI] 7 Oct 2026 Preprint. Under review. d examine the pillow with the lamp put a cool apple in countertop buy a blue 3-pack under $30 General Guidance Trajectory-Specific Guidance Task space Execution Knowledge Curation KnowledgeBank Quality Complementary Coverage Check Act Verify GRPO Knowledge Weaver Redundant Coverage Find Take Search Inspect? Cool? Buy? Toggle lamp Look Match options Chill in fridge Place Order Additional Coverage Cool an apple Inspect a pillow Buy the requested variant examine the pillow with the lamp put a cool apple in countertop buy a blue 3-pack under $30 Cool an apple Inspect a pillow Buy the requested variant Figure 1: Conceptual illustration of general guidance and complementary knowledge. General guid- ance can remain useful for individual tasks while providing limited additional information across entries. Knowledge Weaver combines MI-inspired feedback with execution rewards to encourage experience-dependent knowledge and incremental utility. advances in knowledge construction and use leave a distinct question of how to train a curator to produce better knowledge for a fixed executor. Addressing this question requires considering both the utility of individual entries and what they contribute as knowledge accumulates. Building on this, we propose Knowledge Weaver, an RL framework that trains a language model as a knowledge curator to extract reusable knowledge from agent trajectories. A good curator should produce knowledge that improves task performance and adds useful information as the bank grows. Effective accumulation therefore calls for both limited redundant overlap and substantial incremen- tal utility. Figure 1 illustrates these considerations through knowledge coverage over a task space. Optimizing only for standalone task success can favor general guidance that repeatedly covers the same tasks, yielding substantial overlap and limited additional coverage as the bank grows. Com- plementary entries extend the bank’s coverage, with the shaded region indicating the incremental contribution of a new entry. We therefore approach knowledge accumulation through both the dis- tinct information retained across entries and the additional utility each entry provides. Knowledge Weaver couples token-level feedback inspired by mutual information (MI) with marginal success rewards to preserve distinct information and encourage incremental contributions to the bank. The MI-inspired signal contrasts generation probabilities across trajectory contexts, assigning greater credit to context-sensitive tokens than to tokens whose probabilities remain similar across experiences. To reward incremental contributions, we measure the gain in task success when a new entry is added to retrieved knowledge. This marginal reward favors entries that improve execution beyond what the bank already provides. We also reward task success with each entry alone to encourage individual utility. These signals jointly train the curator, whose generated knowledge is retrieved to guide a frozen executor. We evaluate whether learned knowledge improves a frozen executor on ALFWorld, a household task environment (Shridhar et al., 2020), and WebShop, an online shopping environment (Yao et al., 2022a). Before each task, we retrieve k = 10 knowledge entries from the bank using the task query and provide them to the frozen executor as guidance. Knowledge Weaver achieves success rates of 54.0% and 42.0%, respectively. These results exceed task-success-only GRPO by 16.9 and 18.7 percentage points under the same executor. These gains demonstrate improved task performance and support a curation strategy aimed at reducing redundant overlap among knowledge entries. Our contributions are summarized as follows. • Redundancy in knowledge curation. We identify a limitation of GRPO using only task success as reward, which can produce general guidance with substantial informational re- dundancy across knowledge entries. • MI-guided curator training. We propose Knowledge Weaver, which combines token- level information feedback with execution rewards to train useful and complementary knowledge generation. 2 Preprint. Under review. • Improved downstream performance. On ALFWorld and WebShop, Knowledge Weaver improves SR over task-reward-only GRPO by 16.9 and 18.7 percentage points at k = 10, respectively. Its banks also exceed human-written banks on the reported aggregate metrics. 2RELATED WORK Knowledge Construction from Experience. LLM agents construct reusable knowledge through memory organization, experience distillation, and skill synthesis. Memory systems extract and or- ganize information for later retrieval (Chhikara et al., 2025; Xu et al., 2026; Zhang et al., 2026b), while experience distillation produces reflective lessons and generalizable strategies (Shinn et al., 2023; Zhao et al., 2024; Majumder et al., 2023; Ouyang et al., 2026). Structured representations include conditional guidelines (Fu et al., 2024; Chen et al., 2024), reusable workflows (Wang et al., 2024), and executable skills (Wang et al., 2023; Yuan et al., 2024; Zheng et al., 2025). AutoSkill (Yang et al., 2026) maintains skills derived from interaction traces, while Trace2Skill (Ni et al., 2026) consolidates trajectory-level lessons into transferable guidance. Skill-Pro (Mi et al., 2026) uses execution feedback to generate and verify procedural skills without parameter updates. RL- based approaches further learn to use and construct reusable knowledge. Skill0 (Lu et al., 2026) trains agents to internalize supplied skills, while Skill0.5 (Zhu et al., 2026) combines general skill internalization with task-specific skill utilization. SkillRL (Xia et al., 2026) couples skill distillation and library evolution with executor training. Skill1 (Shi et al., 2026) jointly learns skill selection, utilization, and distillation, while SkillRise (Yao et al., 2026) assigns curation credit through sub- sequent execution outcomes. Knowledge Weaver complements execution feedback with explicit information feedback to train knowledge generation while keeping the executor fixed. Mutual Information. Mutual information (MI) provides a general criterion for learning informa- tive representations and encouraging dependence between related variables. In representation learn- ing, Contrastive Predictive Coding captures information predictive of future observations, while Deep InfoMax promotes dependence between inputs and learned representations (Oord et al., 2018; Hjelm et al., 2018). InfoGAN uses MI between latent codes and generated observations to en- courage interpretable factors of variation (Chen et al., 2016). In reinforcement learning, DIAYN maximizes MI between skill identities and visited states to discover distinguishable behaviors (Ey- senbach et al., 2018). In dialogue generation, MI-based objectives discourage generic responses and favor responses that convey information about the conversational context (Li et al., 2016; Zhang et al., 2018). AMI further models source-target dependence through adversarial optimization of forward and backward generation networks (Pan et al., 2020). For LLM agents, RAGEN-2 intro- duces MI-inspired proxies to assess the dependence of generated reasoning on input prompts (Wang et al., 2026). In knowledge curation, input dependence concerns whether generated entries retain the distinct information carried by their source trajectories. Knowledge Weaver applies token-level MI-inspired feedback during curator training to encourage experience-specific content. 3METHOD 3.1PRELIMINARIES As shown in Figure 2, a trainable curator π θ generates a knowledge entry z ∼ π θ (· | x) from a context x of trajectories sampled from a poolT , each recording a task’s observations and actions. Training maintains an initially empty bankB, expanded between updates. After training, the curator generates a separate evaluation bankC =z (i) K i=1 of valid entries from sampled contexts. A frozen executor uses a knowledge set S to solve task q. Writing its binary success as s(q,S), we measure empirical utility on a finite reward-task setD by U (S) = 1 |D| X q∈D s(q,S).(1) During evaluation, S = R k (q,C) contains at most k entries retrieved once from the initial query and held fixed throughout execution. 3 Preprint. Under review. (a) AgentInteraction (b) MI-guidedKnowledge Curation (c) Knowledge Evaluation Frozen Executor Environment a Use the fridge to cool Use the fridgeto cool r1 r2 r3 r4 r5 Cooling Context Heating Context MI Credit Find the fridge and use the fridge to cool the apple Task Reward Environment With Knowledge Curator top-3 a Cool an apple Heat an apple <Action>: Take the apple <Obs>: You are holding the apple <Action>: Cool the apple using fridge <Obs>: The apple is now cool <Action>: Take the apple <Obs>: You are holding the apple <Action>: Place it in the basin <Obs>: The apple in the basin unheated update (a) (b) SR Single Marginal low high prob Figure 2: Overview of Knowledge Weaver.Token-level MI feedback promotes experience- dependent knowledge, while execution rewards assess individual utility and marginal gains over retrieved knowledge. Only the curator is trained and the executor remains frozen. 3.2EFFECTIVE KNOWLEDGE ACCUMULATION Effective accumulation requires both individual utility and contributions beyond existing knowl- edge. With two equally weighted task types, a routine for an already-supported type and one for an uncovered type can be equally useful alone, yet only the latter extends the bank’s capabilities. We formalize this distinction with a nonnegative, integrable utility profile f i (u) for each entry z (i) over an idealized task coordinate u ∈ R. The profile describes where and how strongly an en- try helps; its area represents total individual utility. To avoid counting the same available utility repeatedly, bank coverage takes the best profile at each location, E (C) = Z R max 1≤i≤K f i (u)du.(2) Thus, repeated profiles add no coverage. This model assumes ideal selection of an entry without retrieval errors or interactions between entries; it illustrates utility overlap without predicting execu- tion success. For a tractable comparison, use Gaussian profiles with fixed centers c 1 < · < c K , K ≥ 2, common height h > 0, and width σ > 0, f i (u;h,σ) = h exp − (u− c i ) 2 2σ 2 .(3) The center locates peak utility, while height and width describe its strength and extent. Each entry has total utility A = √ 2πσh; write the resulting coverage asE (h,σ). Proposition 1 (Utility scaling). For fixed centers and width,E (h,σ) = hE (1,σ) is strictly increas- ing in h > 0. To compare overlap at equal individual utility, we instead fix each profile’s area, so height must decrease as width grows. Proposition 2 (Coverage under a fixed utility budget). For fixed centers and per-entry utility A > 0, let h A (σ) = A/( √ 2πσ). ThenE A (σ) =E (h A (σ),σ) strictly decreases with σ > 0. The first result isolates stronger individual utility. In the second, total individual utility remains KA, but broader profiles overlap more and yield less bank coverage. Narrowing profiles at fixed height need not improve coverage. A new entry adds coverage only where its utility exceeds the existing envelope (Figure 1). Appendix A gives both proofs and the marginal-coverage identity. Why standalone-success GRPO fails. Group Relative Policy Optimization (GRPO) (Shao et al., 2024) samples G ≥ 2 candidates z (g) from a curator snapshot π old (· | x) for the same source context x. Under outcome supervision, every token t in candidate g shares an advantage, the relative learning signal for the policy update, A GRPO g,t = R g − ̄ R σ R + ε num ,(4) 4 Preprint. Under review. where R g is the candidate reward, ̄ R and σ R are its group’s reward mean and standard deviation, and ε num > 0 ensures numerical stability. With standalone success R g = U (z (g) ), this advantage contains no bank information. An already- covered routine receives positive advantage when R g > ̄ R, with no preference for an equally suc- cessful complementary entry. Shared outcome credit also treats source-specific and generic tokens alike. We therefore introduce bank-relative gains to reward contributions beyond existing knowledge and source-dependent credit to encourage retention of the distinct information in each experience. These goals motivate two complementary signals. Marginal success feedback rewards contributions beyond existing knowledge (Section 3.3), while MI-inspired token credit encourages retention of source-dependent information (Section 3.4). 3.3MARGINAL SUCCESS FEEDBACK Marginal success-rate (MSR) feedback encourages additions that improve execution beyond what the bank already supports. For each task q ∈ D, retrieve up to m entries B q = R m (q,B) from a fixed training-bank snapshot and compute MSR(z) = 1 |D| X q∈D [s(q,B q ∪z)− s(q,B q )].(5) We also evaluate standalone success U (z), the success rate given only that entry, while MSR measures change relative to retrieved guidance. For source context x i , candidates and all three conditions (standalone, bank-only, and bank-plus-candidate) share the reward-task set D = D i , with B q fixed per task. The candidate is appended without replacing retrieved entries. An empty bank gives a no-knowledge baseline. Unlike ideal coverage, execution gains can be negative under interference. They operationalize incremental utility without estimating the coverage integral. Between updates, eligible valid entries absent from the bank are ranked by positive MSR and ad- mitted under a quota. If none has positive gain, the bank is unchanged. The updated bank is used in the next training batch; Algorithm 1 gives the schedule. 3.4MI-INSPIRED INFORMATION FEEDBACK Marginal success feedback evaluates what an entry contributes beyond the existing bank. We com- plement this signal with information feedback that encourages the curator to retain the distinct infor- mation contained in its input experience. Mutual information, I(X;Z) = H(Z)− H(Z | X), with H denoting entropy, motivates measuring how source context X changes uncertainty about gener- ated knowledge Z (Cover et al., 1991). For an entry z = (z 1 ,...,z T ) generated from x i among N ≥ 2 contexts, we contrast token probabilities under source and alternative contexts, δ t = log (N − 1)π old (z t | x i ,z <t ) P ℓ̸=i π old (z t | x ℓ ,z <t ) , r MI t = b t minc MI ,λ MI |δ t |. (6) The token and its prefix are held fixed when comparing source likelihood with the mean likelihood under alternative contexts. We take the absolute value to reward the magnitude of input dependence, whether the source context increases or decreases a token’s likelihood. Generic tokens with similar probabilities across contexts have small |δ t | and receive little credit, whereas experience-specific tokens with larger contrasts receive more. The mask b t limits this credit to knowledge-body tokens, with scale λ MI and cap c MI . Assigning credit locally favors input-specific content over generic wording, encouraging the curator to retain more information from each source experience. Joint feedback and policy update. We construct the joint feedback by extending the SR-only task reward used in our GRPO baseline. SR rewards an entry’s standalone utility without accounting for overlap with existing knowledge. We first add MI-inspired token credit to encourage preservation of information specific to each source experience. Because downstream evaluation concerns the utility of a knowledge bank, we further incorporate MSR to reward contributions beyond its existing entries. A formatting term additionally rewards valid structure and penalizes malformed outputs. Retaining GRPO’s group-relative normalization (Equation 4), we combine these signals in a single 5 Preprint. Under review. token-level advantage, A g,t = λ SR U (z (g) ) + λ MSR MSR(z (g) ) + r fmt (z (g) )− ̄ R exec σ exec + ε num + r MI g,t ,(7) where ̄ R exec and σ exec are the within-group mean and standard deviation of the combined SR, MSR, and formatting reward. The weights λ SR and λ MSR control the two utility terms. MSR augments GRPO’s reward comparison with each entry’s contribution to the bank, while MI supplies local credit for source-dependent content. Invalid candidates receive only the formatting penalty and no MI credit. We retain GRPO for policy optimization, using the combined feedback to update only the curator through its clipped objective (Shao et al., 2024). Optimization details are provided in Appendix F.1. 4EXPERIMENTS 4.1EXPERIMENTAL SETUP Environments. We evaluate our method on two benchmarks: ALFWorld (Shridhar et al., 2020) and WebShop (Yao et al., 2022a). ALFWorld is a text-based environment in which an agent com- pletes household tasks through sequential natural-language interactions. We evaluate knowledge banks on the valid_unseen split across six canonical task types: Pick, Clean, Heat, Cool, Look, and Pick Two. We report the success rate (SR) for each task type and across all evaluation tasks. WebShop is a simulated web shopping environment in which an agent searches for products, navi- gates product pages, and purchases an item that satisfies a user’s requirements. We sample 100 tasks from the test set to evaluate the knowledge banks and report both the environment score and SR. Training and Evaluation Protocol. Throughout training, we freeze the executor and optimize only the curator. We assign equal weights of 0.5 to standalone SR and marginal SR. Token-level MI credit is scaled by 0.1 and capped at 0.15 after scaling. We use Qwen3.5-4B as the training model. After training, we use the trained curator to construct knowledge banks in a separate generation stage. We then evaluate the resulting banks with the frozen executor. Each knowledge bank contains 50 entries(except for SkillRL), with each entry limited to 100 words. To generate each entry, we randomly sample four trajectories from a trajectory pool. For each evaluation task, we retrieve the top-k entries using the all-mpnet-base-v2 encoder (Reimers & Gurevych, 2019), ranking entries by their cosine similarity to the initial task query. Retrieval occurs once, before the first action, and the selected entries remain fixed throughout the episode. We use k = 10 for all banks in the main comparison and evaluate every k ∈0,..., 15 in the scaling analysis. Baselines. We compare four categories of knowledge banks, following Table 1: (1) LLM Curators: banks generated directly by Qwen3.5-4B, GPT-5 (OpenAI, 2025), Claude Opus 4.8 (Anthropic, 2026), and Gemini 3.7 Flash (Google DeepMind, 2026), (2) Established Knowledge Banks: publicly released banks from SkillRL (Xia et al., 2026) and human-written banks refined by human, (3) RL- Trained: GRPO (Shao et al., 2024) baseline, and SkillRise (Yao et al., 2026) and Skill1 (Shi et al., 2026), which use execution feedback to improve knowledge generation and refinement, (4) Prompt- Based: ExpeL (Zhao et al., 2024), which iteratively extracts and refines insights from successful and failed trajectories, and Mem0 (Chhikara et al., 2025), which extracts and organizes memories for later retrieval. We also include zero-shot execution (k=0) as a reference without external knowledge. 4.2MAIN RESULTS Improved Performance across Benchmarks. Table 1 shows that our method achieves the highest values on all three aggregate metrics with the same frozen executor at k = 10. Overall SR reaches 54.0% on ALFWorld and 42.0% on WebShop, exceeding GRPO by 16.9 and 18.7 percentage points. On WebShop, its mean environment score reaches 66.4, compared with 49.0 for GRPO, indicating higher average reward alongside improved task success. Our method also outperforms SkillRise and Skill1 on both benchmarks. It exceeds SkillRL, the strongest ALFWorld baseline, by 2.8 percentage points in overall SR. Human-written banks achieve the highest baseline mean environment score on WebShop, but our method surpasses them on all three aggregate metrics. Gains over GRPO span 6 Preprint. Under review. Table 1: Performance of knowledge banks with a frozen Qwen3.5-4B executor on ALFWorld and WebShop. Each bank supplies k = 10 retrieved entries per task and zero-shot execution uses no external knowledge. We report success rate (SR, %) and mean environment score on WebShop. All results are averaged over 3 seeds, with standard deviations shown as subscripts for the Overall and WebShop columns. Bold and underlining mark the best and second-best in each column. ALFWorld (SR%)WebShop MethodPick Clean Heat Cool Look Pick Two OverallScoreSR% Zero-shot30.6 10.8 31.9 30.2 24.125.5 24.6 ±1.5 53.5 ±1.9 27.3 ±1.5 LLM CURATORS Qwen3.5-4B40.3 43.0 24.6 41.3 48.115.7 36.3 ±3.0 40.6 ±2.2 14.7 ±2.1 GPT-540.3 39.8 27.5 61.950.021.6 40.3 ±7.9 45.2 ±6.8 18.7 ±7.4 Claude Opus 4.840.3 30.1 40.6 36.5 72.23.9 37.1 ±6.2 55.2 ±2.5 24.7 ±0.6 Gemini 3.7 Flash25.0 11.8 18.8 25.4 64.811.8 24.6 ±6.4 61.3 ±5.4 35.7 ±5.9 ESTABLISHED KNOWLEDGE BANKS SkillRL34.7 50.5 50.7 68.3 66.7 39.2 51.2 ±2.4 36.6 ±3.0 15.7 ±0.6 Human-written 47.2 48.4 63.8 52.4 55.625.5 49.5 ±1.1 63.1 ±1.5 32.0 ±1.0 RL-TRAINED GRPO30.6 26.9 34.8 44.4 55.6 39.2 37.1 ±3.7 49.0 ±4.8 23.3 ±2.1 SkillRise43.8 22.6 32.6 28.6 47.229.4 33.2 ±1.6 40.0 ±5.6 18.7 ±3.5 Skill137.5 25.8 14.5 34.9 44.425.5 29.9 ±8.2 38.5 ±0.9 17.0 ±1.0 PROMPT-BASED ExpeL38.9 18.3 23.2 27.0 37.031.4 28.4 ±2.7 58.4 ±1.2 24.7 ±2.1 Mem036.8 37.1 52.2 47.6 47.225.5 41.2 ±0.9 50.1 ±1.3 19.0 ±2.0 Our45.848.463.854.087.027.554.0 ±0.4 66.4 ±2.6 42.0 ±2.0 five of six ALFWorld task types, although performance on Pick Two remains lower. These results support training knowledge curators with feedback beyond standalone task success. Strong Base Models Do Not Guarantee Useful Knowledge. In Table 1, we evaluate strong general-purpose models as knowledge curators with the same frozen executor. Our trained Qwen3.5- 4B curator outperforms GPT-5, Claude Opus 4.8, and Gemini 3.7 Flash on all three aggregate met- rics. On ALFWorld, GPT-5 and Claude banks achieve overall SRs of 40.3% and 37.1%, respec- tively, above the zero-shot SR of 24.6%. On WebShop, however, their banks yield SRs of 18.7% and 24.7%, below the zero-shot SR of 27.3%. Gemini achieves the highest WebShop SR among these general-purpose curators (35.7%), whereas its overall ALFWorld SR matches the zero-shot mean (24.6%). These results show that strong general capabilities do not necessarily translate into knowledge that benefits downstream execution. 4.3KNOWLEDGE BANK QUALITY ANALYSIS Useful Knowledge Requires Relatively Little Context. Figure 3 examines whether the main- table gains persist when the number of retrieved entries changes. Knowledge Weaver achieves the highest mean WebShop SR among the eight plotted methods at every positive retrieval count. At k = 10, its knowledge context averages approximately 660 tokens on ALFWorld and 594 on WebShop, less than half the corresponding human-written context lengths. It nevertheless achieves higher SR on both benchmarks. The advantage therefore accompanies compact guidance, rather than a larger volume of injected text. As retrieval increases from k = 9 to 15, SR remains within 51.5-55.0% on ALFWorld and 42.0-45.7% on WebShop. The gains persist across this range de- spite fluctuations, reducing dependence on a single retrieval setting. ALFWorld performance is less uniformly dominant: human-written banks perform better at several smaller retrieval counts, and SkillRL slightly leads at k = 12. Additional Knowledge Can Help or Hinder Execution. Increasing retrieval exposes clear dif- ferences in how banks support the executor. On ALFWorld, Gemini’s SR declines from 38.8% at k = 3 to 20.6% at k = 15, whereas Knowledge Weaver largely retains its gains. On WebShop, Qwen3.5-4B, GPT-5, Claude, and GRPO remain below their own no-knowledge controls at every 7 Preprint. Under review. (a) ALFWorld(b) WebShop Figure 3: Success rate as the number of retrieved knowledge entries k varies from 0 to 15. The horizontal axis reports the number of tokens in the formatted knowledge block injected into the frozen executor, averaged over tasks and runs. Lines and shading show the mean and one standard deviation across three runs. Each curve uses its own no-knowledge control at k = 0. positive retrieval count. Their additional context therefore fails to recover the performance lost when guidance is introduced. This pattern is consistent with unhelpful or conflicting guidance, although the curves alone cannot distinguish interference from retrieval mismatch or other content effects. Ef- fective accumulation consequently requires assessing how entries work together, as well as whether individual entries appear useful. This observation motivates the marginal-utility objective and its contribution is examined alongside MI in the ablation. These sweeps retrieve from fixed banks, with observed token lengths rather than matched token budgets. To complement these execution trends, we quantify whether different knowledge entries tend to be retrieved for the same tasks. At k = 10, we compute mean pairwise Jaccard overlap between these task sets for entries retrieved at least once, then average over three banks per method. Overlap is lower for Knowledge Weaver than for GRPO, with values of 12.92% versus 13.58% on ALFWorld and 15.35% versus 24.48% on WebShop. Thus, sustained execution gains with longer context coexist with lower retrieval overlap at k = 10. These observations motivate our joint use of MI feedback and marginal success rewards to preserve useful information, with execution benefits examined in Section 4.4. 4.4ABLATION STUDIES Table 2: Reward ablation at k = 10 with a frozen Qwen3.5-4B executor. SR rewards indi- vidual task success, MSR rewards success-rate gains over retrieved knowledge, and MI pro- vides token-level information feedback. For- matting rewards are shared across configura- tions. Values are three-run means under differ- ent seeds. Bold marks column best. RewardsALFWorldWebShop SRScoreSR SR37.149.023.3 SR + MI46.065.840.7 SR + MI + MSR54.066.442.0 MI and MSR Improve Knowledge Utility. Table 2 compares SR-only training with two pro- gressively augmented objectives, SR + MI and SR + MI + MSR. The first extension adds token- level MI, increasing ALFWorld SR from 37.1% to 46.0% and WebShop SR from 23.3% to 40.7%. WebShop score also improves from 49.0 to 65.8, supporting information feedback as a useful addi- tion to standalone-success training. Building on SR + MI, the full objective adds MSR to reward contributions beyond existing knowl- edge. ALFWorld SR increases further to 54.0%, an additional 8.0 percentage points, comparable to the 8.9-point improvement from adding MI. On WebShop, SR reaches 42.0% and score reaches 66.4, with a smaller additional SR gain of 1.3 points. Both extensions therefore improve per- formance, with similarly sized SR gains on ALF- World and a larger gain from MI on WebShop. 8 Preprint. Under review. Knowledge Banks Transfer across Frozen Executors. We reuse banks generated by a cura- tor trained with feedback from a frozen Qwen3.5-4B executor. Table 3 reports their performance with eight additional frozen executors at k = 8. Knowledge Weaver improves ALFWorld SR and both WebShop metrics over no-knowledge execution for all eight executors. Relative to GRPO, it achieves higher ALFWorld SR and WebShop scores for seven of eight executors on each metric, and higher WebShop SR for six, with one tie. GRPO banks improve ALFWorld SR for all eight executors but reduce WebShop SR for four. These results support reusing Knowledge Weaver banks across the evaluated executors without retraining the curator for each target model. Table 3: Transfer of knowledge banks across frozen executors. None denotes execution without knowledge (k = 0), while GRPO and Knowledge Weaver (Ours) retrieve k = 8 entries per task. Knowledge-augmented results are averaged over three banks per method. Bold marks the best result for each executor and metric. ExecutorALFWorld SR (%)WebShop ScoreWebShop SR (%) NoneGRPOOursNoneGRPOOursNoneGRPOOurs Qwen3.5-4B24.638.149.353.549.067.527.320.741.0 Qwen3.5-27B48.561.961.254.955.666.925.023.041.7 Llama-3-8B-Instruct6.77.212.72.710.310.50.04.74.0 Llama-3.1-8B-Instruct9.714.921.116.06.719.76.02.311.7 Phi-4-mini-instruct7.59.513.92.38.95.41.03.33.3 Phi-419.432.638.157.051.365.213.016.733.3 DeepSeek-V4-Flash56.064.264.957.056.269.230.026.046.3 MiniMax-M2.535.153.756.562.864.873.630.033.048.0 MiniMax-M2.170.973.974.473.268.775.843.037.049.7 5CONCLUSION We introduced Knowledge Weaver, an RL framework that trains knowledge curators to preserve information through token-level MI feedback and rewards for marginal gains over retrieved knowl- edge, while rewarding standalone success. On ALFWorld and WebShop, the resulting banks out- perform the evaluated baselines with a frozen executor, and the gains hold across retrieval sizes. Ablation indicates that MI adds value beyond task success alone. The banks also transfer to other executors, though the gains vary by model. These results support information preservation as a useful principle for building reusable knowledge from agent experience. We plan several directions that follow from this work in the future. First, because our experiments keep the executor frozen, a natural extension is to jointly train the curator and executor, allowing knowledge construction and utilization to improve in tandem. Second, the uneven transfer gains suggest that knowledge effective for one executor may not transfer directly to another, motivating the development of knowledge banks that generalize across models. Finally, extending Knowledge Weaver to longer-horizon tasks, where the bank is continually updated as experience accumulates, would provide a stronger test of how well knowledge preservation scales over time. More broadly, we view learning to curate reusable knowledge as a step toward agents that self-improve continually through their own experience. AI USE STATEMENT In this work, we used generative AI tools to assist with the implementation of software code and the analysis and interpretation of experimental results. We did not use generative AI tools to formulate the core research ideas, hypotheses, or methodological design of this work. Additionally, we used generative AI tools to improve the readability and language of the manuscript and to create or refine minor visual elements in some figures. All AI-assisted code was reviewed and tested by the authors, and all AI-assisted analyses of experi- mental results were independently verified against the original experimental outputs. All AI-assisted text and visual elements were reviewed and revised by the authors. We take responsibility for the 9 Preprint. Under review. final content of this work, including text, claims, code, analyses, and artifacts produced with the aid of generative AI. REPRODUCIBILITY STATEMENT Section 3 defines the feedback signals and training objective. Section 4.1 describes the ALFWorld and WebShop evaluation protocol. Appendix C provides the prompts used for knowledge generation and execution. Appendix E reports the hyperparameter settings in Table 4. Appendix F presents the training pseudocode and GRPO update. Appendix D explains how the baselines were implemented and adapted to the shared evaluation protocol. ETHICS STATEMENT Human volunteers wrote knowledge entries from sampled agent trajectories to construct the Human- written baseline. Participation was voluntary, and informed consent was obtained from all volun- teers. REFERENCES Anthropic.Claude Opus 4.8 System Card.System card, 2026.URL https://w. anthropic.com/claude-opus-4-8-system-card. Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He. Automanual: Constructing instruction manuals by llm agents via interactive environmental learning. Advances in Neural Information Processing Systems, 37:589–631, 2024. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025. Thomas M Cover, Joy A Thomas, and John Kieffer. Elements of information theory, volume 2. wiley New York, 1991. Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018. Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of context-aware guide- lines for large language model agents. Advances in Neural Information Processing Systems, 37: 119919–119948, 2024. Google DeepMind. Gemini 3.7 Flash Model Card. Model card, August 2026. URL https: //deepmind.google/models/model-cards/gemini-3-7-flash/. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. In International Conference on Learning Representations, volume 2024, p. 52690–52717, 2024. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, p. 110–119, 2016. 10 Preprint. Under review. Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Ky- oung Whan Choe, Jiankai Sun, Shuyi Wang, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026. Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Skill0: In-context agentic reinforcement learning for skill internalization. arXiv preprint arXiv:2604.02268, 2026. Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. Clin: A continually learning language agent for rapid task adaptation and generalization. arXiv preprint arXiv:2310.10134, 2023. Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, and Jun Wang. Skill-pro: Learning reusable skills from experience via non-parametric ppo for llm agents. arXiv preprint arXiv:2602.01869, 2026. Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Er- chao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158, 2026. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018. OpenAI. GPT-5 System Card. System card, 2025. URL https://openai.com/index/ gpt-5-system-card/. Siru Ouyang, Jun Yan, I Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long Le, Samira Daruki, Xiangru Tang, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory. In International Conference on Learning Representations, volume 2026, p. 94327–94354, 2026. Boyuan Pan, Yazheng Yang, Kaizhao Liang, Bhavya Kailkhura, Zhongming Jin, Xian-Sheng Hua, Deng Cai, and Bo Li. Adversarial mutual information for text generation. In International Con- ference on Machine Learning, p. 7476–7486. PMLR, 2020. Nils Reimers and Iryna Gurevych.Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), p. 3982–3992, 2019. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Leyang Shen, Yang Zhang, Chun Kai Ling, Xiaoyan Zhao, and Tat-Seng Chua. Carl: Critical action focused reinforcement learning for multi-step agent. arXiv e-prints, p. arXiv–2512, 2025. Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi Gu, Xunliang Cai, Xiang Wang, and An Zhang. Skill1: Unified evolution of skill-augmented agents via reinforcement learning. arXiv preprint arXiv:2605.06130, 2026. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheat- sheet: Test-time learning with adaptive memory. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7080–7106, 2026. 11 Preprint. Under review. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, et al. Ragen-2: Reasoning collapse in agentic rl. arXiv preprint arXiv:2604.06268, 2026. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024. Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, et al. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234, 2026. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. Advances in Neural Information Processing Systems, 38:17577–17604, 2026. John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. Yutao Yang, Junsong Li, Qianjun Pan, Bihao Zhan, Yuxuan Cai, Lin Du, Jie Zhou, Kai Chen, Qin Chen, Xin Li, et al. Autoskill: Experience-driven lifelong learning via skill self-evolution. arXiv preprint arXiv:2603.01145, 2026. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744–20757, 2022a. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b. Zhiyuan Yao, Yuxin Chen, Zhengxi Lu, Zishan Xu, Yueqing Sun, Yifu Guo, Yuquan Lu, Zhengzhou Cai, Kangning Zhang, Zhuowen Han, et al. Skillrise: Agentic reinforcement learning for cross- task skill evolution. arXiv preprint arXiv:2607.26784, 2026. Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Fung, Hao Peng, and Heng Ji. Craft: Customizing llms by creating and retrieving from specialized toolsets. In International Conference on Learning Representations, volume 2024, p. 40097–40125, 2024. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kama- nuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. Agentic context engineering: Evolving contexts for self-improving language models. In International Conference on Learning Representations, volume 2026, p. 86069–86100, 2026a. Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maxi- mization. Advances in Neural Information Processing Systems, 31, 2018. Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, and Tat-Seng Chua.Nextmem: Towards latent factual memory for llm-based agents.arXiv preprint arXiv:2603.15634, 2026b. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p. 19632–19642, 2024. Boyuan Zheng, Michael Y Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, et al. Skillweaver: Web agents can self-improve by discovering and honing skills. arXiv preprint arXiv:2504.07079, 2025. 12 Preprint. Under review. Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. In International Conference on Learning Rep- resentations, volume 2024, p. 19036–19066, 2024. Jiapeng Zhu, Jianxiang Yu, Yibo Zhao, Chengcheng Han, Qi Gu, Xunliang Cai, Xiang Li, and Wein- ing Qian. Skill0. 5: Joint skill internalization and utilization for out-of-distribution generalization in agentic reinforcement learning. arXiv preprint arXiv:2605.28424, 2026. 13 Preprint. Under review. APROOFS We use the Gaussian utility profiles from Equation 3, with K ≥ 2 fixed centers c 1 < · < c K and common height h > 0 and scale σ > 0. Specializing Equation 2 gives the envelope coverage E (h,σ) = R R max i f i (u;h,σ)du. Write ∆ i = c i+1 − c i > 0 for the adjacent gaps, and let φ and Φ denote the standard normal density and cumulative distribution function. A.1PROOF OF PROPOSITION 1 Proof. For fixed σ, changing the common height rescales every profile by the same positive factor. This factor can be taken outside the pointwise maximum and the integral, giving E (h,σ) = h Z R max 1≤i≤K exp − (u− c i ) 2 2σ 2 du = hE (1,σ).(8) To establish strict monotonicity, it remains to check that the coefficientE (1,σ) is positive and finite. The envelope is bounded below by any individual profile and above by their sum. Since each unit- height profile integrates to √ 2π σ, these bounds imply 0 < √ 2π σ ≤E (1,σ)≤ K √ 2π σ <∞. Thus coverage is linear and strictly increasing in the common height h. A.2PROOF OF PROPOSITION 2 Proof. Fix the per-entry mass A > 0. The Gaussian integral gives A = √ 2π σh, so the height must vary with σ according to h A (σ) = A √ 2π σ , f i (u;h A (σ),σ) = A σ φ u− c i σ .(9) Each profile therefore has mass A for every σ > 0. We writeE A (σ) =E (h A (σ),σ) for the coverage under this constraint. Because all profiles share the same height and scale, the nearest center provides the largest utility at each location. Define the adjacent midpoints m i = (c i + c i+1 )/2 for i = 1,...,K − 1, with m 0 = −∞ and m K = +∞. On [m i−1 ,m i ], the center c i is nearest to u, so f i attains the upper envelope. We can therefore integrate the envelope one interval at a time: E A (σ) = K X i=1 Z m i m i−1 f i (u;h A (σ),σ)du = A K X i=1 Φ m i − c i σ − Φ m i−1 − c i σ .(10) For each interior interval, the upper and lower bounds relative to its center are ∆ i /2 and−∆ i−1 /2. Using Φ(−z) = 1 − Φ(z), its contribution is A[Φ(∆ i /(2σ)) + Φ(∆ i−1 /(2σ)) − 1]. The two end intervals contribute AΦ(∆ 1 /(2σ)) and AΦ(∆ K−1 /(2σ)), respectively. Thus each adjacent gap contributes twice, while each of the K − 2 interior intervals contributes one subtraction of A. Summing gives E A (σ) = A " 2 K−1 X i=1 Φ ∆ i 2σ − (K− 2) # .(11) This expression also holds for K = 2, when there are no interior intervals. Differentiating with A and the centers fixed, and using Φ ′ = φ, yields dE A (σ) dσ =− A σ 2 K−1 X i=1 ∆ i φ ∆ i 2σ < 0.(12) The sum contains at least one term because K ≥ 2, and every term is positive because ∆ i > 0 and φ is strictly positive. Hence coverage strictly decreases with σ under the fixed per-entry mass constraint, proving the proposition. 14 Preprint. Under review. Redundancy and marginal coverage. At each location, utility counted by the sum of pro- files but omitted by their maximum represents redundant overlap.Its integral is R A (σ) = R R [ P i f i (u;h A (σ),σ)− max i f i (u;h A (σ),σ)]du. Subtracting Equation 11 from the total mass KA gives R A (σ) = KA−E A (σ) = 2A K−1 X i=1 Φ − ∆ i 2σ .(13) SinceR ′ A =−E ′ A > 0, increasing the common scale increases redundant overlap. As σ → 0 + , each Φ term in Equation 11 tends to 1, giving coverage KA in the limit. As σ → ∞, each term tends to 1/2, giving limiting coverage A. Consequently, the coverage efficiencyE A /(KA) decreases from a limit of 1 to a limit of 1/K. More generally, for any nonnegative, integrable candidate profile f z , its marginal coverage is ∆ cov (z |C) =E (C∪z)−E (C) = Z R f z (u)− max 1≤i≤K f i (u) + du, (14) where [a] + = max(a, 0). This follows pointwise from maxa,b−b = [a−b] + . Apply the identity with a = f z (u) and b = max i f i (u), then integrate. It holds for any nonnegative, integrable profiles and requires no Gaussian assumption. For the Gaussian specialization, consider adding an entry with the same A and σ at c K + ∆, where ∆ > 0. The coverage formula gains one gap term and one additional subtraction of A, so the marginal coverage isA[2Φ(∆/(2σ))−1]. It is strictly increasing in ∆ and lies in (0,A), approaching 0 as ∆→ 0 + and A as ∆→∞. The same argument applies to an entry added beyond the leftmost center. 15 Preprint. Under review. BCASE STUDIES OF KNOWLEDGE BANK CONTENT We inspect knowledge banks generated from step-40 curator checkpoints to examine what repeated and distinct entries contain. All examples below are complete entries from the GRPO and Knowl- edge Weaver used in Table 1. Each source bank contains 50 entries. We select cases from the knowledge banks of GRPO and Knowledge Weaver. B.1ALFWORLD GRPO | Step 40 Possession before placement locate your target object, verify it is on the surface with take, then immediately navigate to the destination and execute move to place it. Avoid navigating without picking up the object or performing unnecessary inventory checks that delay the action. Possession before placement always take the object from the source before moving or attempting to place it there, even if the action seems redundant, to ensure possession is maintained and progress is made. Possession before placement Check your inventory before moving to the destination to confirm you are carrying the correct object, then proceed directly to the destination and place the object there immediately without returning to the source. Knowledge Weaver | Step 40 Cooling with the fridge When cooling an object for placement, immediately pick it up from the countertop after nav- igating there, then head to the fridge and cool the object there, rather than returning to the countertop or searching for the object multiple times. Heating with the microwave If you need to heat an item before placing it: verify the item is accessible by picking it up first (e.g., from the countertop), then heat it in the microwave using valid actions like move and heat; this ensures the object is in your possession and eliminates the error of heating an object that is still sitting in one location while you navigate to gain access to another, which consumes steps without progressing toward success. Inspection under a lamp Before looking at an object in light, verify you are holding the object; if not, locate it by navigating to its current position and pick it up, then turn on the nearby lamp by using it only after the object is in your inventory. The three GRPO entries reduce to the same possession-before-placement routine despite their dif- ferent wording. The selected Knowledge Weaver entries retain distinct operations: cooling with a fridge, heating with a microwave, and activating a lamp while holding the object. These differ- ences concern the prescribed action and appliance, rather than only an object-name substitution. They demonstrate task-specific content in individual entries, although bank-wide redundancy is not directly assessed by these examples. 16 Preprint. Under review. B.2WEBSHOP GRPO | Step 40 Verify color before purchase Before clicking Buy Now, explicitly verify the visible "color" filter has been applied by se- lecting a specific color option, rather than relying on assumptions that a search query string initializes this setting. Verify color before purchase Before clicking the Buy Now button, explicitly verify that the selected color option has actually been applied by checking the instruction text for a confirmation message (e.g., "You have clicked dark gray") rather than relying on the absence of an error message. This prevents the "invisible update" failure where missing a confirmation causes the price filter to reset, allowing unselected colors to bypass the intended price constraint and wasting search steps. Verify color before purchase Before clicking Buy Now, verify the selected color filter option is actually selected on the product interface-based on the URL or page state, not solely on the search query or user intent. Knowledge Weaver | Step 40 Select a product attribute When a product page lists multiple color or size options alongside the current selection, always click the specific correct option to apply the attribute before clicking Buy Now. Constrain the search query When combined product attributes like color or material type are critical to the search intent, explicitly specify these color and material constraints in the search query rather than relying solely on the displayed product page for confirmation, as clicking a product page to select a specific color attribute often triggers an error stating "INVALID(click target is not currently visible)." Continue to the next page When a product page displays no purchasable option of the specified color or size, explicitly advance to the next page to find the correct attribute before clicking Buy Now, rather than leaving the system with a non-compliant selection field. The GRPO examples provide general guidance on checking color before purchase, varying the sug- gested confirmation signal while repeating the same decision rule. Across this bank, 48 of 50 entries begin with “Before”. The selected Knowledge Weaver entries cover three different decisions: ap- plying an available attribute, expressing constraints in the search query, and continuing to another page when a suitable option is absent. CCURATOR AND EXECUTOR PROMPTS We report the system and user message templates used for knowledge curation and execution on ALFWorld and WebShop. The curator receives four trajectories from the same task type and re- turns a single knowledge entry of at most 100 words, enclosed in <skill> and </skill> tags. We reproduce the evidence-conditioned prompts, including their output requirements, and retain the implementation’s term “skill.” The prompt wording is preserved; whitespace is adjusted for typesetting. Fields such as trajectories, observation, and guidance represent content in- serted at runtime. The trajectory field contains the rendered source histories, and the guidance field contains the knowledge supplied to the executor. For fixed-bank evaluation, multiple retrieved en- tries are numbered under “Retrieved guidance skills:” and remain fixed throughout the episode. The memory block is omitted when no prior interactions are available, and the WebShop invalid-response 17 Preprint. Under review. line is included only when retrying an invalid response. In the ALFWorld action vocabulary, obj, recep, and appliance are literal action-template slots. The executor fields are shown after preprocessing and before model-specific chat formatting. The recorded evaluations retain up to five recent interactions. ALFWorld supplies the final 1,800 obser- vation characters and at most 140 admissible commands; WebShop limits the current observation to 5,000 characters and lists at most 60 visible click targets. Command and click-target lists are sorted before truncation. C.1ALFWORLD CURATOR ALFWorld curator | System You extract one useful guidance skill from agent trajectories. The acting agent already receives general operating guidance to consult its legal actions, navigate where needed, reveal accessible contents, and carry out the requested interaction. Treat that guidance as given rather than rewriting it as the learned skill. Study the complete trajectories as behavioral evidence. Keep investigating beyond convenient surface explanations until the histories support an additional consequential insight that can improve future decisions. The conclusion must depend on what actually happened in the trajectories. Do not merely summarize the task or action-space description, imitate a fixed writing pattern, or invent unsupported details. Reason privately and output only the final natural-language skill required by the output contract. Return exactly one skill wrapped by <skill> and </skill>, with no text outside the tags. ALFWorld curator | User # Task category category (category_note) # Legal action vocabulary The agent may use these action templates: go to recep, open/close recep, take obj from recep, put obj in/on recep, use obj, heat/cool/clean obj with appliance, examine obj, look. This list only constrains which actions are legal. It is not trajectory evidence and must not be restated as the learned skill. The agent only sees the current room view. Receptacles must be navigated to, and some must be opened before their contents become visible. # Complete trajectories trajectories # Skill content requirements - Read across the complete action histories and discover a consequential issue or insight that is easy to miss from the task description alone, then give useful guidance for addressing it. - Ground the guidance in differences, consequences, or lack of progress visible in the supplied trajectories. If the same advice could be written without reading them, investigate further. - Do not mechanically recap the action sequence. - Use only legal actions from the action vocabulary. Do not invent actions or unsupported facts. - Write one compact natural-language paragraph of at most 100 words, without following a fixed rhetorical template. # Strict output requirements - Return exactly: <skill><one natural-language guidance paragraph></skill> - The opening tag must be the first non-whitespace text and the closing tag the last. - Use exactly one opening tag and one closing tag; do not add analysis, markdown, labels, nested tags, or text outside the wrapper. 18 Preprint. Under review. C.2ALFWORLD EXECUTOR ALFWorld executor | System You are an ALFWorld TextWorld agent. Choose the next action using this exact format: Thought: <concise reasoning, max 30 words> Action: <copy exactly one command from the admissible command list> Write the Action line immediately after the Thought line. Do not add any other text. Guidance skill: guidance ALFWorld executor | User Task: task_description Observation: observation Recent online episodic memory: online_memory Admissible commands: admissible_commands Respond with Thought and Action: C.3WEBSHOP CURATOR WebShop curator | System You extract one useful guidance skill from shopping-agent trajectories. Read the complete histories as behavioral evidence and begin with the concrete details that explain why an action helped, failed, or changed the outcome. Infer one non-obvious, consequential insight that would materially improve a future agent’s decision in a similar situation. Let the evidence determine what the skill is about instead of starting from a predetermined workflow or restating advice that could be written without these histories. Make the insight specific enough to guide a decision. Ground every claim in an observed state, action consequence, or outcome difference. Do not produce generic boilerplate, a reusable checklist, a trajectory recap, or unsupported product facts. Reason privately and output only the final skill required by the output contract. Return exactly one <skill>...</skill> wrapper and no text outside it. WebShop curator | User # Task category WebShop product search, option selection, and purchase # Legal action vocabulary The agent can call search[concise query] when a search box is present and click[exact visible target] for products, navigation controls, product options, and Buy Now. This vocabulary only constrains legal behavior; it is not trajectory evidence and must not be restated as the skill. # Complete trajectory evidence trajectories # Skill content requirements 19 Preprint. Under review. WebShop curator | User (continued) - Extract a consequential insight whose usefulness depends on reading these histories closely, including evidence from failed progress, misleading page state, or a successful change in behavior. - Prefer the smallest behaviorally meaningful correction or decision principle supported by the evidence. If the same advice could be written without the histories, investigate further. - Let concrete trajectory evidence determine both the content and wording. Do not fall back on generic boilerplate, a reusable checklist, or a trajectory recap. - Use only legal WebShop actions and facts supported by the supplied histories. - Write one compact natural-language paragraph of at most 100 words. # Strict output requirements - Return exactly: <skill><one natural-language guidance paragraph></skill> - The opening tag must be the first non-whitespace text and the closing tag the last. - Use exactly one opening tag and one closing tag. Do not add analysis, markdown, labels, nested tags, or text outside the wrapper. C.4WEBSHOP EXECUTOR WebShop executor | System You are a WebShop text agent. Select the next action to satisfy the shopping instruction and maximize the final WebShop reward. Respond using exactly two lines: Thought: <concise reasoning, at most 35 words> Action: <one valid action> Use search[concise keywords] only when search is available. Use click[exact target] by copying one currently visible clickable target. Do not add any other text. Guidance skill: guidance WebShop executor | User Shopping instruction: shopping_instruction Current page observation: observation Recent online memory: online_memory Search action: available_or_unavailable Visible click targets: visible_click_targets Previous response was invalid: error_reason Return Thought and Action now: DBASELINE REPRODUCTION We reproduced Skill1, SkillRise, and Mem0 with Qwen3.5-4B under a shared bank-evaluation pro- tocol. The trained Skill1 and SkillRise checkpoints served as curators for separate bank generation. Each bank contained 50 entries of at most 100 words, constructed from matched bundles of four tra- jectories. Evaluation used the frozen Qwen3.5-4B executor and all-mpnet-base-v2 retrieval, with the top 10 entries selected once and held fixed throughout each episode. Skill1 and SkillRise results are reported as means and standard deviations across three runs. 20 Preprint. Under review. D.1SKILL1 We retained Skill1’s query generation, skill reranking, library updates, and distillation objective (Shi et al., 2026). We trained query-generation and reranking responses while excluding task-execution responses from the policy loss. Complete trajectories still supplied outcome rewards. We used the step-40 checkpoints to generate the evaluation banks. D.2SKILLRISE We retained SkillRise’s cross-task training and decoupled credit assignment (Yao et al., 2026). Each trial comprised three tasks from the same family, with the skill document updated between tasks and reset between trials. Solving responses received feedback from their own task outcomes, whereas curation responses received feedback from subsequent tasks. D.3MEM0 We evaluated both semantic and procedural banks constructed with Mem0 v2.1.0 (Chhikara et al., 2025) and Qwen3.5-4B, retaining the package’s native memory-writing prompts and exporting memories from the same trajectory bundles into the shared bank format. For each evaluation seed, we averaged the results of the two bank types, then reported the mean and standard deviation across three seeds. D.4HUMAN KNOWLEDGE CONSTRUCTION For the Human-written baseline, we sampled trajectories using the same procedure and distributed them to volunteers, who wrote knowledge entries based on these trajectories. We then assembled these entries into knowledge banks following the shared construction protocol and evaluated the resulting banks under the same evaluation protocol. EEXPERIMENTAL DETAILS E.1HYPERPARAMETER DETAILS Table 4 summarizes the training configuration for the Qwen3.5-4B curator. We update all cura- tor parameters while keeping the executor and reference model frozen. After training, the curator generates separate knowledge banks for downstream evaluation. E.2DATA SOURCES AND TASK SPLITS Table 5 summarizes the task sources used by Knowledge Weaver. The full training split is eligible for reward sampling; each group samples eight tasks rather than evaluating the entire split at every update. Training trajectories, reward tasks, and bank-generation sources may share training tasks. For ALF- World, these sources belong to train, whereas final evaluation uses valid_unseen. For Web- Shop, official human training and test tasks have disjoint task IDs. Synthetic bank-generation tasks use a separate ID namespace. E.3TRAINING DETAILS We initialized the curator, executor, and reference policy from Qwen3.5-4B. Training updated all curator parameters, while the executor and reference policy remained frozen. We optimized the curator with the clipped GRPO objective using AdamW, a constant learning rate, and KL regular- ization against the reference policy. Table 4 specifies the principal settings for the main Knowledge Weaver configuration on ALFWorld and WebShop. The training batch size and GRPO group size refer to different sampling levels. Each training batch contained 32 trajectory contexts, each comprising four trajectories from the same task type. For each context, the curator sampled eight candidate knowledge entries from the identical input, giving 21 Preprint. Under review. HyperparameterALFWorldWebShop Optimization OptimizerAdamW Learning rate1× 10 −6 Learning-rate schedule / warmupConstant / none AdamW (β 1 ,β 2 )(0.9, 0.999) AdamW numerical epsilon10 −8 Weight decay0.01 Gradient-norm clipping1.0 Policy-ratio clipping parameter0.2 KL regularization coefficient0.01 PPO minibatches per training batch32 Context Sampling and Generation Trajectory contexts per training batch32 Contexts per MI comparison set8 Input trajectories per context4 Candidate entries per context (GRPO group size)8 Curator sampling temperature1.0 Curator top-p1.0 Maximum generated tokens per candidate256 Maximum words per knowledge entry100 Maximum rendered steps per source trajectory24 Source-observation character limit120400 Execution Feedback and Bank Updates Executor temperature0 Maximum actions per reward rollout5015 Reward-task rollouts per candidate and condition8 Standalone-success weight, λ SR 0.5 Marginal-success weight, λ MSR 0.5 Token-level MI weight, λ MI 0.1 MI credit cap after scaling, c MI 0.15 Valid-format bonus / invalid-format penalty+0.02/− 0.1 Retrieved entries for marginal feedback, m3 Table 4: Key training hyperparameters for the main Qwen3.5-4B Knowledge Weaver configuration. Shared values span both benchmark columns; benchmark-specific values are shown separately. Each training batch contains 32 trajectory contexts and eight candidates per context, for 256 candidate entries before optimization. Only the curator is trained. The table describes training; the separately generated evaluation banks contain 50 entries and use k = 10 in the main comparison. Data roleALFWorldWebShop Training source trajectoriesCollected from train across all six task types. Collected from the full official human training split. SR/MSR reward tasksSampled from all train tasks matching the source context’s task type. Sampled from all 10,587 official human training tasks. Evaluation-bank source trajectories 120 train trajectories, with 20 per task type. 300 synthetic training-task trajectories. Final evaluation tasksAll 134 valid_unseen tasks.A fixed subset of 100 tasks from the 500 official human test tasks. Table 5: Task sources for training, bank construction, and final evaluation. 32 × 8 = 256 candidates per batch. Execution rewards were normalized within each group of eight candidates. The value 32 therefore counts input contexts per training batch, whereas the value 8 counts candidate entries per context. The PPO minibatch count in the table specifies how this generated batch was partitioned for optimization. 22 Preprint. Under review. For execution feedback, we used eight reward-task rollouts per candidate and condition, with matched tasks shared across the standalone, bank-only, and bank-plus-candidate conditions. Stan- dalone success and marginal success received equal weights of 0.5. A valid <skill> response received a formatting bonus of 0.02; an invalid response received a penalty of−0.1 without execu- tion or MI credit. Token-level MI credit was applied only to knowledge-body tokens, with weight 0.1 and a cap of 0.15 after scaling. We used within-group normalization for execution rewards and batch-level normalization for the resulting token advantages. The training bank was initialized empty and held fixed while each batch was scored. Marginal feedback used up to three retrieved entries, and bank admission required a strictly positive marginal success gain. On ALFWorld, the update quota allowed at most one entry from each of two task types per batch during the first 20 updates, and at most one entry thereafter. On WebShop, at most one entry was admitted per batch. After training, the curator generated a separate evaluation bank of 50 entries. The training retrieval count of three is distinct from the main evaluation setting of k = 10. E.4RETRIEVAL-COUNT SCALING We evaluate retrieval counts k ∈0,..., 15 while keeping each knowledge bank, the executor, and the benchmark task set fixed. For methods also shown in Figure 3, we use the same evaluation runs and replace token count with retrieval count on the horizontal axis. Each curve uses a fixed set of runs across all retrieval counts using three runs. Each curve retains its own no-knowledge control at k = 0. (a) ALFWorld(b) WebShop Figure 4: Success rate as the number of retrieved knowledge entries k varies from 0 to 15. The horizontal axis reports the number of retrieved knowledge injected into the frozen executor. Lines and shading show the mean and one standard deviation across three runs. Each curve uses its own no-knowledge control at k = 0. Figure 4 shows that the effect of additional retrieved knowledge varies across banks and environ- ments. Knowledge Weaver achieves SRs of 54.0% and 42.0% at k = 10 on ALFWorld and Web- Shop, respectively. At k = 15, the corresponding rates are 54.2% and 43.7%. FTRAINING ALGORITHM Algorithm 1 summarizes Knowledge Weaver training. Candidates generated from the same tra- jectory context form a GRPO sampling group. Token-level MI feedback compares each candidate across source contexts while preserving its generated tokens and prefix. The executor and reference policy remain frozen throughout training. All execution conditions for a candidate use the same tasks, with B q fixed in the marginal compari- son. With an empty bank, the bank-only condition reduces to execution without knowledge. Invalid candidates receive only the formatting penalty. MI credit is added after execution advantages are computed, so it is not propagated to preceding tokens through return estimation. 23 Preprint. Under review. Algorithm 1 Knowledge Weaver curator training Require: Curator π θ , frozen executor, reference policy π ref , trajectory poolT Require: Updates J , candidates per context G≥ 2, retrieval count m, admission scheduleQ Require: Reward weights λ SR ,λ MSR , MI weight λ MI , MI cap c MI Ensure: Trained curator π θ 1: Initialize training knowledge bankB ← ∅ 2: for j = 1,...,J do 3: π old ← π θ ; fix bank snapshotB snap ←B 4:Sample a batch of trajectory-context sets fromT 5:for each context setX = (x 1 ,...,x N ), with N ≥ 2 do 6:for i = 1,...,N do 7:Sample G candidates z ∼ π old (·| x i ) 8:Select training reward tasksD i , shared by these candidates 9:Retrieve up to m entries B q fromB snap for each q ∈D i 10:for each candidate z generated from x i do 11:Initialize U (z)← 0, MSR(z)← 0, and r MI t ← 0 12:if z has valid formatting then 13:Evaluate s(q,z), s(q,B q ), and s(q,B q ∪z) onD i 14:Average standalone success to obtain U (z) 15:Compute MSR(z) using Equation 5 onD i 16:Rescore z t under each x ℓ ∈X with z <t fixed 17:Compute masked token credits r MI t using Equation 6 18:end if 19:R exec (z)← λ SR U (z) + λ MSR MSR(z) + r fmt (z) 20:end for 21:Normalize R exec (z) within the G candidates generated from x i 22:end for 23:end for 24:Combine normalized execution rewards and local MI credit as A g,t (Equation 7) 25:Compute b A g,t by reference regularization and batch normalization (Equation 16) 26:Update θ using the clipped GRPO objective with π old and b A g,t 27:Select valid candidates absent fromB snap with MSR(z) > 0 28:Admit the highest-gain eligible candidates subject toQ j to obtainB 29: end for 30: return π θ The admission schedule Q j specifies the eligible task types and update quota. On ALFWorld, we admit at most one candidate per scheduled task type, selecting two types per update for the first 20 updates and one thereafter. On WebShop, we admit at most one candidate per update. Admission ranks candidates by MSR alone; if no eligible candidate has positive gain, the bank remains un- changed. The updated bank is used in the next training batch. After training, the curator generates a separate knowledge bank for downstream evaluation. F.1OPTIMIZATION DETAILS Feedback and final advantages. The combined SR, MSR, and formatting rewards in Equation 7 are normalized across candidates from the same source context before being placed at the final generated token. We use zero value estimates, unit discount, and unit GAE parameter. Reference- policy regularization contributes a penalty at each generated token. Let ℓ beh g,t and ℓ ref g,t denote the sampled token log-probabilities under the behavior and frozen reference policies. The sampled KL term is κ g,t = clip ℓ beh g,t − ℓ ref g,t ,−10, 10 .(15) Backward return computation accumulates these penalties from token t to the entry’s final token T g . Local MI credit is added after this computation, so it is not accumulated across token positions. 24 Preprint. Under review. Using the modified advantage A g,t from Equation 7, the final advantages are b A g,t = Norm batch A g,t − β T g X s=t κ g,s .(16) The sum includes only unmasked generated tokens. Batch normalization subtracts the mean and divides by the sample standard deviation plus ε num over all such tokens in the rollout batch. Group reward normalization also uses sample standard deviation, with ε num = 10 −5 in both operations. The configurations use β = 0.01 and clipping radius ε = 0.2. Reference regularization enters through these rewards, without an additional differentiated KL term in the loss. The full configuration uses λ SR = λ MSR = 0.5, λ MI = 0.1, and c MI = 0.15. Formatting adds 0.02 for valid entries and gives invalid entries a reward of −0.1. The backend also clips scalar rewards before group normalization and local token credits before addition to [−20, 20]. This bound is inactive for these reward ranges and does not alter the equations above. Clipped update. We use the clipped policy surrogate underlying GRPO (Shao et al., 2024), aver- aged over valid tokens in each optimization minibatch. For token advantages A = A g,t , the loss is L clip (θ;A) =− 1 |V| X (g,t)∈V mineρ g,t (θ)A g,t , clip(eρ g,t (θ), 1− ε, 1 + ε)A g,t ,(17) whereV contains unmasked token positions in an optimization minibatch and g indexes candidates across source contexts. Training uses A = b A from Equation 16, with all advantage quantities held fixed during differentiation. The ratio eρ g,t compares the current token probability with a detached proximal probability, eρ g,t (θ) = exp logπ θ (z (g) t | x (g) ,z (g) <t )− ℓ prox g,t ,(18) where x (g) is candidate g’s source context. Standard GRPO uses the behavior log-probability as ℓ prox g,t . Our backend uses a log-linear approximation based on the policy versions recorded during generation. For current training version v and token generation version v beh g,t < v, define α g,t = clip v− 1− v beh g,t v− v beh g,t , 0, 1 ! , ℓ prox g,t = (1− α g,t )ℓ beh g,t + α g,t stopgrad logπ θ (z (g) t | x (g) ,z (g) <t ) . (19) Otherwise α g,t = 0, and the behavior log-probability is used directly. Samples from the immedi- ately preceding version also have α g,t = 0. The denominator is detached even when its numerical value incorporates current-policy log-probabilities. We use neither dual clipping nor rejection sam- pling. Algorithm 1 omits this version bookkeeping for readability; it does not change the knowledge rewards or bank-admission criterion. All updates apply only to the curator, with the executor and reference policy frozen. 25