Paper deep dive
AEL: Agent Evolving Learning for Open-Ended Environments
Wujiang Xu, Jiaojiao Han, Minghao Guo, Kai Mei, Xi Zhu, Han Zhang, Dimitris N. Metaxas
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 4/26/2026, 8:48:02 PM
Summary
AEL (Agent Evolving Learning) is a two-timescale framework designed for LLM agents in open-ended environments. It addresses the 'statelessness' of current agents by using a fast-timescale Thompson Sampling bandit to learn memory retrieval policies and a slow-timescale LLM-driven reflection to diagnose failure patterns and inject causal insights. The framework features a three-tier memory system (episodic, semantic, and procedural) and demonstrates that self-diagnosis of how to use experience is more critical for improvement than architectural complexity. In a sequential portfolio benchmark, AEL outperformed several self-improving methods and non-LLM baselines, achieving a Sharpe ratio of 2.13.
Entities (10)
Relation Signals (6)
AEL → evaluatedon → Sequential Portfolio Benchmark
confidence 100% · On a sequential portfolio benchmark (10 sector-diverse tickers, 208 episodes, 5 random seeds), AEL achieves a Sharpe ratio of 2.13
AEL → implements → Episodic Memory
confidence 100% · AEL achieves this through three memory tiers... Episodic memory records each episode's raw outcome
AEL → implements → Semantic Memory
confidence 100% · Semantic memory aggregates episodic records into cross-episode patterns
AEL → implements → Procedural Memory
confidence 100% · Procedural memory promotes high-confidence semantic patterns into executable rules
AEL → uses → Thompson Sampling
confidence 100% · At the fast timescale, a Thompson Sampling bandit learns which memory retrieval policy to apply
AEL → uses → LLM-driven reflection
confidence 100% · at the slow timescale, LLM-driven reflection diagnoses failure patterns
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM agents increasingly operate in open-ended environments spanning hundreds of sequential episodes, yet they remain largely stateless: each task is solved from scratch without converting past experience into better future behavior. The central obstacle is not \emph{what} to remember but \emph{how to use} what has been remembered, including which retrieval policy to apply, how to interpret prior outcomes, and when the current strategy itself must change. We introduce \emph{Agent Evolving Learning} (\ael{}), a two-timescale framework that addresses this obstacle. At the fast timescale, a Thompson Sampling bandit learns which memory retrieval policy to apply at each episode; at the slow timescale, LLM-driven reflection diagnoses failure patterns and injects causal insights into the agent's decision prompt, giving it an interpretive frame for the evidence it retrieves. On a sequential portfolio benchmark (10 sector-diverse tickers, 208 episodes, 5 random seeds), \ael{} achieves a Sharpe ratio of 2.13$\pm$0.47, outperforming five published self-improving methods and all non-LLM baselines while maintaining the lowest variance among all LLM-based approaches. A nine-variant ablation reveals a ``less is more'' pattern: memory and reflection together produce a 58\% cumulative improvement over the stateless baseline, yet every additional mechanism we test (planner evolution, per-tool selection, cold-start initialization, skill extraction, and three credit assignment methods) \emph{degrades} performance. This demonstrates that the bottleneck in agent self-improvement is \emph{self-diagnosing how to use} experience rather than adding architectural complexity. Code and data: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.21725v1
- Canonical: https://arxiv.org/abs/2604.21725v1
Trouble viewing inline? Open PDF directly →
Full Text
75,523 characters extracted from source content.
Expand or collapse full text
AEL: Agent Evolving Learning for Open-Ended Environments Wujiang Xu1 Jiaojiao Han2 Minghao Guo1 Kai Mei1 Xi Zhu1 Han Zhang1 Dimitris N. Metaxas1 1Rutgers University 2Independent Researcher Abstract LLM agents increasingly operate in open-ended environments spanning hundreds of sequential episodes, yet they remain largely stateless: each task is solved from scratch without converting past experience into better future behavior. The central obstacle is not what to remember but how to use what has been remembered, including which retrieval policy to apply, how to interpret prior outcomes, and when the current strategy itself must change. We introduce Agent Evolving Learning (AEL), a two-timescale framework that addresses this obstacle. At the fast timescale, a Thompson Sampling bandit learns which memory retrieval policy to apply at each episode; at the slow timescale, LLM-driven reflection diagnoses failure patterns and injects causal insights into the agent’s decision prompt, giving it an interpretive frame for the evidence it retrieves. On a sequential portfolio benchmark (10 sector-diverse tickers, 208 episodes, 5 random seeds), AEL achieves a Sharpe ratio of 2.13± 0.47, outperforming five published self-improving methods and all non-LLM baselines while maintaining the lowest variance among all LLM-based approaches. A nine-variant ablation reveals a “less is more” pattern: memory and reflection together produce a 58% cumulative improvement over the stateless baseline, yet every additional mechanism we test (planner evolution, per-tool selection, cold-start initialization, skill extraction, and three credit assignment methods) degrades performance. This demonstrates that the bottleneck in agent self-improvement is self-diagnosing how to use experience rather than adding architectural complexity. Code and data: https://github.com/WujiangXu/AEL. 1 Introduction LLM agents are deployed in open-ended environments spanning hundreds of sequential episodes, from coding (Jimenez et al., 2024) to web navigation (Zhou et al., 2024) and sequential prediction (Yao et al., 2023), yet they remain largely stateless. Modern agents comprise a planner for reasoning (Huang et al., 2024), tools for external API access (Schick et al., 2023), and memory for storing past experience (Xu et al., 2025), yet do not reliably transform experience into improved behavior. Several methods enable agents to improve: Reflexion (Shinn et al., 2023) accumulates verbal self-critiques, ExpeL (Zhao et al., 2024) extracts cross-task lessons, and EvoTool (Yang et al., 2026) evolves tool-use policies via blame-aware mutation. However, each evolves only a single module while holding the rest fixed. In open-ended environments, agent competence emerges from the interaction of planning, tool use, and memory; when multiple modules change simultaneously, the resulting performance shift is ambiguous. This multi-module credit assignment problem prevents agents from improving as coordinated systems. We propose Agent Evolving Learning (AEL), a two-timescale framework. At the fast timescale, a Thompson Sampling bandit (Chapelle and Li, 2011) learns which memory retrieval policy suits the current stage; at the slow timescale, LLM-driven reflection diagnoses failure patterns and injects causal insights into the agent’s prompt. When diagnosis warrants structural change, the framework generates new retrieval policies. A systematic credit study shows that uniform credit outperforms sophisticated alternatives, an important negative result. Our contributions are fourfold: ❶Unified evolving framework. Prior methods evolve a single module (memory, tools, or planning) while holding the rest fixed. AEL treats the agent as a coupled system where a Thompson Sampling bandit learns how to access experience and LLM-driven reflection diagnoses when structural change is needed. ❷Two-timescale diagnose-before-prescribe architecture. Fast-timescale bandit updates adapt the memory retrieval policy episode by episode, while slow-timescale LLM reflection aggregates trajectories to produce causal diagnoses. New retrieval policies are generated only when the existing pool is inadequate, preventing changes from noisy signals. ❸Comprehensive “less is more” ablation. We evaluate nine modifications to AEL (removing warm-up or reflection; adding planner evolution, per-tool selection, cold-start, skills; switching to FCC or LLM-FCC credit). All modifications degrade performance, establishing that the simplest configuration is optimal in high-noise, short-horizon domains. ❹Reflection as the key enabler. On a sequential portfolio benchmark (10 tickers, 208 episodes, 5 seeds), AEL achieves Sharpe 2.13± 0.47, ranking first on risk-adjusted metrics with the lowest variance among all LLM methods. An incremental study shows memory yields +24% over the stateless baseline, while reflection adds a further +27%, confirming that the bottleneck is self-diagnosis, not experience accumulation. (a) Previous methods (Zhao et al., 2024; Yang et al., 2026). (b) Ours method. Figure 1: (a) Prior methods evolve a single module (here, Tools) while planner and memory remain fixed, leaving the interaction effects across modules unaddressed. (b) AEL treats the agent as a coupled system: planner, tools, and memory co-evolve under a shared allocator, with credit assigned via a Thompson sampling bandit at the fast timescale and LLM-driven reflection at the slow timescale. 2 Related Work 2.1 LLM Agents ReAct (Yao et al., 2023) interleaves chain-of-thought reasoning (Wei et al., 2022) with action execution, establishing the reason-then-act paradigm. Tree of Thoughts (Yao et al., 2024) expands the reasoning space via branching, while Huang et al. (2024) identify planning as a key bottleneck. Toolformer (Schick et al., 2023) demonstrates autonomous API invocation. For memory, MemGPT (Packer et al., 2023) introduces OS-inspired paging between fast and slow storage, and A-Mem (Xu et al., 2025) proposes self-organized agentic memory. These three modules (planner, tools, memory) form the backbone of modern agents but are typically fixed once deployed. AEL makes all three evolvable. 2.2 Evolving Learning of LLM Agents Several methods enable agents to improve, each evolving a subset of modules. Reflexion (Shinn et al., 2023) accumulates self-critiques in the context window without structured retrieval. Voyager (Wang et al., 2023) builds a skill library through curriculum-driven exploration but does not adapt tools or planning. ExpeL (Zhao et al., 2024) extracts lessons via keyword matching. Meta-Reflexion (Wu et al., 2025) distills reflections into rules (the closest analogue to AEL’s procedural memory) but neither evolves tools or planners. AutoAgent (Wang et al., 2026a) explores elastic memory with evolving cognition. EvoTool (Yang et al., 2026) evolves tool-use policies via blame-aware mutation but holds memory fixed; FactorMiner (Wang et al., 2026b) combines skills with experience memory; Tool-Genesis (Xia et al., 2026) benchmarks tool creation. None jointly evolve tools and memory with reflection-driven self-diagnosis or study credit assignment in multi-module evolution. 2.3 Bandits and Credit Assignment Contextual bandits (Li et al., 2010; Auer et al., 2002) and Thompson Sampling (Chapelle and Li, 2011; Agrawal and Goyal, 2012) provide efficient online learning with natural exploration. For multi-party credit assignment, Shapley values (Shapley, 1953; Lundberg and Lee, 2017; Ghorbani and Zou, 2019) offer a principled cooperative-game framework but largely treat modules as black boxes. AEL systematically evaluates uniform, factored counterfactual, and LLM-driven credit assignment, finding that simpler methods often outperform in high-noise domains and further highlighting credit assignment as an important open challenge. Table 1: Feature comparison with prior methods. AEL is the only method combining multi-tier memory, learned retrieval, and causal diagnosis across two timescales. Method Memory Retrieval Diagnosis Dual Policy Learning Scale Evol. Reflexion (Shinn et al., 2023) Context – – – – Voyager (Wang et al., 2023) Skills – – – – ExpeL (Zhao et al., 2024) Lessons – – – – Meta-Reflexion (Wu et al., 2025) Rules – – – – EvoTool (Yang et al., 2026) – – blame – tools FactorMiner (Wang et al., 2026b) Experience – – – – AEL (ours) 3-tier memory ✓ ✓ ✓ ✓ 3 The AEL Framework 3.1 Overview and Design Rationale Let ett=1T\e_t\_t=1^T denote the episode stream. AEL operates on two episode-based timescales: fast windows kfastW^fast_k of size N update selection preferences, while slow windows jslowW^slow_j of size M≫NM N aggregate evidence for reflection and memory consolidation. At episode t, the agent selects a configuration ct=(pt,zt,mt)c_t=(p_t,z_t,m_t) specifying planner, tool set, and memory retrieval policy, then observes outcome score st∈[−1,1]s_t∈[-1,1]. In the default AEL configuration, the planner and tools are fixed; a Thompson Sampling bandit selects among memory retrieval policies, learning which way of accessing experience suits the current stage. At the slow timescale, LLM-driven reflection diagnoses failure patterns and injects causal insights into the decision prompt. Uniform credit converts outcomes into bandit rewards; the framework additionally supports planner selection (LinUCB), per-tool Thompson Sampling, cold-start, and skill extraction, all evaluated in Table 3. Figure 2 shows the pipeline; episodes are split chronologically into train/val/test, with all learning frozen at test time. 3.2 Memory-Policy Selection and Extended Module Bandits The central online learning mechanism in AEL is memory-policy selection via Thompson Sampling (Chapelle and Li, 2011). Each policy m in a pool of five initial strategies maintains a Beta posterior Beta(αm,βm)Beta( _m, _m); at each episode the agent samples μ~m∼Beta(αm,βm) μ_m ( _m, _m) and selects the policy with the highest sample. After observing the episode outcome, the posterior is updated: αm+=r~t _m += r_t, βm+=(1−r~t) _m +=(1- r_t). The five initial policies range from no retrieval to aggressive multi-tier recall (Appendix M); new policies can be added by the evolution mechanism described in Section 3.4. This learns how to access experience (compressed summaries, full records, or no retrieval), adapting the strategy as the memory store matures. The framework additionally supports two further bandit levels, evaluated in the ablation (Table 3): Planner selection (LinUCB (Li et al., 2010)). A contextual bandit whose context vector ϕt∈ℝ7 _t ^7 encodes sector, 30-day volatility, log market cap, data richness, momentum, options availability, and analyst coverage. It selects πt=argmaxπ(ϕt⊤θ^π+αϕt⊤π−1ϕt) _t= _π ( _t θ_π+α _t A_π^-1 _t ), balancing exploitation of the best-performing planner with exploration of uncertain alternatives (full update equations in Appendix E). Per-tool selection (Thompson Sampling). Each tool a maintains a Beta posterior Beta(αa,βa)Beta( _a, _a) updated from directional hit/miss statistics against realized outcomes; the top-K tools by sampled value are selected, with K shrinking as learning progresses (details in Appendix E). Figure 2: AEL framework overview. At the fast timescale (A), a Thompson Sampling bandit selects memory retrieval policies episode-by-episode, feeding planner-guided predictions and bandit updates. At the slow timescale (B), an LLM reflects on aggregated trajectories to consolidate memory and evolve modules; during deployment (C), all weights are frozen while policy-based retrieval remains active. 3.3 Three-Tier Evolving Memory Raw episode logs are too noisy and voluminous to retrieve directly. The agent needs to progressively distill experience from specific episodes into general patterns and then into actionable rules. AEL achieves this through three memory tiers with automatic promotion. Episodic memory records each episode’s raw outcome: which tools were used, what signals they produced, whether each signal was correct, and the overall prediction quality. This provides the ground-truth training data for distillation. Semantic memory aggregates episodic records into cross-episode patterns, distilled periodically (every 10 episodes). These patterns capture regularities invisible in any single episode, such as “momentum indicators are reliable for trending stocks but misleading during reversals.” Procedural memory promotes high-confidence semantic patterns into executable rules injected directly into planner prompts. These rules represent the agent’s most trusted knowledge, influencing behavior without requiring retrieval at query time. Memory use involves two distinct decisions. First, the Thompson Sampling bandit selects a retrieval policy that specifies which tiers are visible, how many entries to retrieve, and how they are formatted (e.g., compressed summaries vs. full records vs. no retrieval at all). Second, within the selected policy, entries are ranked by a composite relevance score: r(q,e)=fmatch(q,e)⏟ticker, sector, tool×(0.5+0.5qe)⏟quality×(0.3+0.7e−0.01Δ)⏟recency×bτ⏟tier boostr(q,e)\;=\; f_match(q,e)_ticker, sector, tool\;×\; (0.5+0.5\,q_e)_quality\;×\; (0.3+0.7\,e^-0.01 )_recency\;×\; b_τ_tier boost where fmatchf_match sums feature-match bonuses, qeq_e is the entry’s quality score, Δ is the number of episodes since the entry was written, and bτ∈1.0,1.2,1.5b_τ∈\1.0,1.2,1.5\ for episodic, semantic, and procedural tiers respectively. The top-k entries (default k=5k=5) above a quality threshold are returned. This two-layer design means the bandit learns how to access experience, while the scoring function determines which entries surface under that access strategy. 3.4 LLM-Driven Reflection and Code Evolution Selection within a fixed module pool will eventually plateau if the pool itself is inadequate. The reflection system addresses this by letting the LLM diagnose why performance has degraded and then generate new modules to address the diagnosed problem. Cold-start initialization. Before any episodes, the LLM reads tool and planner descriptions and generates informed priors for the bandit system, reducing the exploration cost of uniform initialization. Slow-window reflection. After each slow window, the LLM receives four inputs: (i) per-ticker episode summaries (planner used, score, directional accuracy), (i) per-tool accuracy statistics aggregated over the window, (i) market side information (sector returns, volatility regime, cross-correlation) computed from cached price data but not given to the predictor, and (iv) the last three reflections for temporal continuity. It produces a structured output containing a causal insight (what market conditions caused today’s outcomes and why specific signals were reliable or misleading), a regime label, and a confidence score. This insight is injected directly into the allocator’s prompt at the next episode, giving the predictor an interpretive frame for its evidence without polluting the memory store with noisy LLM-generated narratives. Module selection remains with the bandit; reflection diagnoses conditions, not configurations. Code evolution is triggered by diagnosed structural failures, not fixed schedules. When a planner’s failure streak exceeds a threshold (default 3 consecutive slow windows), the LLM generates a new Python planner class tailored to the diagnosed failure mode; the new class must pass syntax validation and a smoke test before entering the pool. Memory policy evolution fires periodically (every 5 slow windows, starting after window 10) only when the average Thompson Sampling reward across all policies falls below 0.4, indicating that the entire policy pool is underperforming. The LLM then designs a new retrieval policy specifying tier visibility, retrieval depth, and formatting strategy, which is added as a new bandit arm. Note that in the main AEL configuration, planner evolution is disabled; it is active only in the full EAEL variant. The design principle is diagnose before prescribe: the LLM must first build an explanation of why performance degraded before generating an architectural change, making evolution targeted rather than random. 3.5 Learning Signal The training signal combines two complementary mechanisms at different timescales. At the fast timescale, each episode produces a scalar outcome st∈[−1,1]s_t∈[-1,1] that is converted into a uniform reward r~t=clip((st+1)/2,0,1) r_t=clip((s_t+1)/2,0,1) and used directly to update the memory-policy bandit posterior (and, in the full EAEL variant, tool and planner bandits as well). Uniform credit avoids compounding misattribution in our high-noise domain. We also evaluate factored counterfactual credit (FCC, combining structural, counterfactual, and Shapley attribution) and LLM-driven credit (LLM-FCC), but both degrade performance (Table 3; full formulations in Appendix F). At the slow timescale, reflection consumes aggregated trajectories to decide when structural evolution is needed, providing a qualitative diagnostic channel that complements the quantitative bandit signal. Algorithm 1 The training process of AEL 1:Episode stream ett=1T\e_t\_t=1^T; memory-policy pool ℳ=m1,…,mKM=\m_1,…,m_K\; priors αk=βk=1 _k= _k=1; slow-window size M; warm-up W; evolution threshold r¯min r_ 2:Posteriors (αk,βk)\( _k, _k)\, memory D, pool ℳM 3:Partition et\e_t\ into ℰtrain,ℰval,ℰtestE_train,E_val,E_test chronologically 4:for each slow window j⊂ℰtrainW_j _train do 5: for each episode et∈je_t _j do 6: Sample μ~k∼Beta(αk,βk) μ_k ( _k, _k) for each mk∈ℳm_k 7: mt←argmaxkμ~km_t← _k μ_k ⊳ Thompson Sampling 8: t←Retrieve(,mt,et)x_t← Retrieve(D,m_t,e_t) ⊳ Policy-guided recall 9: y^t←Plan(et,t,) y_t← Plan(e_t,x_t,Z) ⊳ Predict with tools Z 10: Observe yty_t; compute st∈[−1,1]s_t∈[-1,1] 11: r~t←clip((st+1)/2, 0, 1) r_t ((s_t+1)/2,\;0,\;1 ) ⊳ Uniform credit 12: αmt+=r~t _m_t += r_t; βmt+=1−r~t _m_t +=1- r_t 13: Write (et,y^t,yt,st)(e_t, y_t,y_t,s_t) to episodic tier of D 14: end for 15: ℐj←Reflect(j,)I_j← Reflect(W_j,D) ⊳ LLM causal diagnosis 16: Inject ℐjI_j into planner prompt for j+1W_j+1 17: Distill episodic → semantic → procedural tiers of D 18: if r¯ℳ<r¯min r_M< r_ and j>jminj>j_ then 19: mnew←EvolvePolicy(ℐj,)m_new← EvolvePolicy(I_j,D); ℳ←ℳ∪mnewM ∪\m_new\ 20: end if 21:end for 22:Select best posteriors on ℰvalE_val; freeze for ℰtestE_test 4 Experiments We design experiments to answer four research questions: Q1: Does AEL outperform prior self-improving agent methods? Q2: Does each component contribute individually, or only through synergy? Q3: How do different credit assignment methods compare? Q4: How robust is performance across random initializations? 4.1 Experiments Setup Dataset. We evaluate on sequential portfolio allocation, a domain requiring multi-episode adaptation due to regime shifts, delayed feedback, and a combinatorial action space. The D-full benchmark comprises 10 sector-diverse tickers spanning 7 GICS sectors at 1-hour resolution (208 episodes: 140 train / 40 val / 28 test). Training covers diverse market regimes (bull, bear, flat); the test set contains a bear-to-bull transition, testing whether the learned architecture generalizes to unseen conditions. While AEL is domain-agnostic, financial prediction provides objective ground truth and regime diversity that stress-tests adaptation (full benchmark details in Appendix K). Baselines. We compare: (i) four non-LLM baselines (equal-weight, momentum-weighted, min-variance, inverse-momentum); (i) five prior self-improving methods: Reflexion (Shinn et al., 2023), ExpeL (Zhao et al., 2024), FactorMiner (Wang et al., 2026b), Meta-Reflexion (Wu et al., 2025), and EvoTool (Yang et al., 2026) (adaptation details in Appendix P); (i) HyperAgent (Zhang et al., 2026), a recursive self-modification framework (Section 5); and (iv) an AEL incremental build (Stateless → +Tools → +Memory → AEL). All methods share the same 12 tools (Appendix L), backbone LLM (Claude Haiku 4.5), and data split. Evaluation & Protocol. We report seven frozen test-phase metrics: Sharpe, Sortino, and Calmar ratios (risk-adjusted return from complementary perspectives), cumulative Return%, maximum drawdown (MaxDD%), win rate (WinR), and tail ratio (TailR); formal definitions are in Appendix C. The matched incremental build uses seeds 42, 123, 456; headline methods extend to 5 seeds. During test, all learning is disabled (frozen bandits, read-only memory, no evolution). Full protocol details including hyperparameters, warm-up schedule, and cost breakdown are in Appendix D. 4.2 Main Results (Q1) Table 2: D-full benchmark results (N=5N=5 seeds for stochastic methods). Bold = best per column. Non-LLM baselines: EqW (equal-weight), Mom (momentum-weighted), MinV (min-variance), InvM (inverse-momentum). Metric definitions in Appendix C. Category Method Sharpe ↑ Sortino ↑ Calmar ↑ Ret% ↑ MaxDD% ↑ WinR ↑ TailR ↑ Non-LLM EqW 0.70 1.32 3.05 +0.34 −-1.47 0.47 1.14 Mom 1.44 2.73 6.89 +1.21 −-1.64 0.48 1.09 MinV −-0.61 −-0.82 −-2.06 −-0.32 −-1.59 0.45 0.89 InvM −-0.20 −-0.30 −-0.66 −-0.17 −-1.73 0.45 0.96 Prior Reflexion −-0.59± 1.33 −-0.57± 1.75 −-1.64± 5.92 −-0.23± 0.54 −-2.30± 0.36 0.48 0.85 ExpeL 0.76± 1.93 1.68± 3.93 3.97± 9.79 +0.24± 0.63 −-1.76± 0.16 0.45 1.05 FactorMiner 0.85± 1.15 1.55± 2.14 3.61± 4.91 +0.22± 0.31 −-1.93± 0.28 0.47 1.02 Meta-Refl. 0.20± 1.16 0.37± 1.65 1.66± 6.01 +0.11± 0.49 −-2.22± 0.23 0.49 0.87 EvoTool 1.37± 1.74 2.73± 3.58 6.28± 7.70 +0.34± 0.46 −-1.46± 0.15 0.47 1.17 HyperAgent 0.72± 0.38 1.00± 0.53 3.90± 2.37 +0.34± 0.20 −-2.49± 0.15 0.49 0.76 AEL Variants Stateless 1.35± 1.03 2.51± 1.93 6.52± 4.98 +0.36± 0.29 −-1.59± 0.26 0.47 1.05 +Tools 1.29± 1.19 2.51± 2.23 6.07± 5.52 +0.33± 0.32 −-1.63± 0.26 0.49 1.05 +Memory 1.68± 0.96 3.29± 1.83 8.50± 5.09 +0.49± 0.29 −-1.53± 0.11 0.46 1.27 Ours AEL 2.13± 0.47 4.08± 1.11 10.40± 2.75 +0.62± 0.18 −-1.53± 0.06 0.47 1.24 Table 2 reports frozen-test metrics on the D-full benchmark across five random seeds. We focus on the Sharpe ratio as the primary metric because it directly reflects the training signal (episode-level return normalized by volatility) and captures the risk-return tradeoff that is central to sequential portfolio allocation. Sortino and Calmar ratios provide complementary views on downside risk and worst-case drawdown. AEL achieves the highest Sharpe (2.13), Sortino (4.08), and Calmar (10.40), outperforming all 10 baselines. It also attains the tightest MaxDD (−-1.53%) and the lowest Sharpe variance (± 0.47) among all stochastic methods. Comparison to prior self-improving methods. Among prior methods, EvoTool is the strongest on mean Sharpe (1.37) but with the highest variance (± 1.74), indicating that its evolutionary tool policy is sensitive to initialization. ExpeL (0.76± 1.93) and Reflexion (−-0.59± 1.33) both exhibit extreme seed dependence: individual outlier seeds inflate their means (detailed in Appendix I). HyperAgent (0.72± 0.38) achieves the lowest variance among prior methods, but its recursive code generation fails to escape the initial equal-weight template (Appendix J). The deterministic momentum-weighted baseline (1.44) outperforms all prior LLM methods on Sharpe, underscoring that LLM-based learning must overcome the noise it introduces to beat simple heuristics. Why AEL improves. The incremental build (Stateless 1.35 → +Memory 1.68 → AEL 2.13) isolates the source of improvement: memory enables cross-episode learning (+24%), and reflection further enhances performance (+27%) by diagnosing failure patterns and filtering memory quality. Unlike prior methods that accumulate experience indiscriminately (Reflexion) or evolve only one module (EvoTool), AEL uses reflection to provide the agent with an interpretive frame for its evidence, turning raw memory into actionable diagnostic insights. The low variance (± 0.47) further suggests that this diagnostic mechanism generalizes across random initializations rather than depending on a lucky seed. 4.3 Component Synergy and Seed Robustness (Q2, Q4) (a) Incremental component build. (b) Mean-focused robustness summary. Figure 3: (a) Incremental build (5 seeds): Stateless (1.35) → +Memory (1.68) → AEL (2.13); adding LLM-FCC credit degrades to 1.37. (b) Robustness: Mean test Sharpe ± 1 std. AEL has the highest mean (2.13) with the lowest variance (± 0.47) among all LLM methods. 3(a) reveals two findings. First, memory and reflection form a synergistic pair: Stateless (1.35) → +Memory (1.68) → AEL (2.13), with reflection providing a 27% jump over memory alone. The disproportionate gain from reflection confirms that the bottleneck in open-ended agent improvement is how to use experience, not simply accumulating it. Second, adding LLM-FCC credit to the full AEL system degrades performance (1.37), showing that sophisticated credit assignment introduces more noise than signal in this high-noise domain. 3(b) confirms that AEL achieves the highest mean with the lowest variance (± 0.47) among all LLM methods, while prior methods exhibit high instability. 4.4 Ablation and Credit Analysis (Q2, Q3) Figure 4: (a) Component ablation from AEL (2.13): removing reflection (→ 1.68) or memory (→ 1.35) degrades performance. (b) Adding any complexity to AEL hurts: planner evolution (Δ −-1.72), per-tool selection (Δ −-1.70), and cold-start (Δ −-1.31) are most harmful. Even the best credit method (LLM-FCC, 1.49) underperforms uniform credit. Table 3 and Figure 4 reveal a striking finding: every modification to AEL degrades performance. We analyze three categories of modifications below. Removing components. Warm-up is the most critical infrastructure (Δ −-0.88). Without the 15-episode warm-up period, the memory bandit receives noisy early-episode rewards that corrupt its posterior, causing it to converge prematurely to a suboptimal retrieval policy. This early corruption is persistent because Beta posteriors are difficult to “unlearn” once biased. Reflection contributes Δ −-0.45 by diagnosing failure patterns and producing causal insights that improve how the agent interprets retrieved experience. Even though Δ −-0.45 appears modest, the +Memory baseline (1.68) is already strong; reflection’s value is in quality filtering rather than raw performance gain. Adding complexity consistently hurts. Planner evolution (Δ −-1.72) and per-tool Thompson selection (Δ −-1.70) are the most damaging. LinUCB planner selection requires estimating a d×d×d covariance matrix per planner arm. With only 208 episodes split across 6 planners, each arm receives roughly 35 observations, far below what is needed for reliable contextual estimation. The resulting exploration noise outweighs any adaptation benefit. Per-tool Thompson selection faces a similar data starvation problem: 12 tools × 10 tickers creates 120 bandit arms, each receiving approximately 17 updates over the training horizon. Cold-start initialization (Δ −-1.31) seeds memory with LLM-generated priors that may not match the actual data distribution, introducing systematic bias. Skill extraction (Δ −-1.11) identifies tool-combination patterns that worked during training but do not generalize to the test regime (bear→ transition). Credit assignment. Switching from uniform to FCC (Δ −-1.09) or LLM-FCC (Δ −-0.64) degrades performance. FCC uses Shapley-based attribution, which requires 23=82^3=8 coalition evaluations every 80 episodes. In a 208-episode dataset, this provides at most two Shapley estimates, each computed from a small and noisy sample. LLM-FCC is less harmful because its content-aware reasoning can partially distinguish “the tool gave a correct warning that was ignored” from “the tool gave a wrong signal,” but it still introduces stochastic noise through LLM reasoning variability. The finding that uniform credit (Δ=0 =0) outperforms all sophisticated alternatives highlights credit assignment as an important open problem for multi-module agent evolution, particularly in high-noise, short-horizon settings. Table 3: Ablation study from AEL (5 seeds, post-fix code). Each row adds or removes one component. Adding any complexity degrades performance, confirming that the simple AEL configuration is optimal. Configuration Sharpe Sortino Δ Impact AEL (Ours) 2.13 4.08 — — Remove component −- warm-up 1.25 2.25 −-0.88 Critical −- reflection (= +Memory) 1.68 3.29 −-0.45 High Add complexity ++ cold-start init 0.82 1.41 −-1.31 Harmful ++ planner evolution 0.41 1.01 −-1.72 Harmful ++ per-tool selection 0.43 0.80 −-1.70 Harmful ++ skill extraction 1.02 1.78 −-1.11 Harmful Change credit method → FCC credit 1.04 1.96 −-1.09 Harmful → LLM-FCC credit 1.49 2.84 −-0.64 Moderate 5 Discussion Simplicity beats complexity. A striking finding from our ablation (Table 3) is that every addition to the AEL configuration degrades performance. Planner evolution (Δ −-1.72), per-tool selection (Δ −-1.70), and cold-start (Δ −-1.31) are particularly harmful, suggesting that adaptive mechanisms need substantially more data than a 208-episode horizon provides to overcome their exploration overhead. Even LLM-driven credit (LLM-FCC, Δ −-0.64), the least harmful addition, still underperforms uniform credit. This “less is more” pattern suggests that in short-horizon, high-noise domains, the exploration overhead of adaptive mechanisms outweighs their benefit. Modular vs. unconstrained evolution. HyperAgent (Zhang et al., 2026) rewrites the entire allocation function via recursive metacognition. Despite 20 generations, it achieves only a 0.46 Sharpe ratio—below equal-weight (0.70) due to code-generation failures (50% invalid), batch evaluation masking progress, and the absence of modular credit signals. The improvements HyperAgent would require—modularity, per-component signals, and online updates—are exactly what AEL provides, suggesting these are fundamental requirements for self-improving agents (Appendix J). 6 Conclusion We introduced AEL, a two-timescale framework coupling Thompson Sampling bandit-based memory retrieval with LLM-driven reflection. Our central finding is that self-diagnosis is the key enabler: memory alone modestly improves over stateless baselines, but reflection produces a qualitative jump (27% Sharpe), while every additional mechanism degrades performance. This demonstrates that in open-ended environments the bottleneck is how to use experience, not accumulating it. A systematic ablation further reveals that uniform credit outperforms all sophisticated alternatives, highlighting credit assignment as an open challenge for multi-module agent evolution. References S. Agrawal and N. Goyal (2012) Analysis of thompson sampling for the multi-armed bandit problem. Proceedings of the 25th Annual Conference on Learning Theory, p. 39.1–39.26. Cited by: §2.3. P. Auer, N. Cesa-Bianchi, and P. Fischer (2002) Finite-time analysis of the multiarmed bandit problem. Machine Learning 47 (2–3), p. 235–256. Cited by: §2.3. O. Chapelle and L. Li (2011) An empirical evaluation of thompson sampling. Advances in Neural Information Processing Systems 24. Cited by: §1, §2.3, §3.2. A. Ghorbani and J. Zou (2019) Data shapley: equitable valuation of data for machine learning. Proceedings of the 36th International Conference on Machine Learning, p. 2242–2251. Cited by: §2.3. X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y. Wang, L. Chen, et al. (2024) Understanding the planning of LLM agents: a survey. arXiv preprint arXiv:2402.02716. Cited by: §1, §2.1. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024) SWE-bench: can language models resolve real-world GitHub issues?. Proceedings of the 12th International Conference on Learning Representations. Cited by: Appendix A, §1. L. Li, W. Chu, J. Langford, and R. E. Schapire (2010) A contextual-bandit approach to personalized news article recommendation. Proceedings of the 19th International Conference on World Wide Web, p. 661–670. Cited by: §2.3, §3.2. S. M. Lundberg and S. Lee (2017) A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems 30. Cited by: §2.3. C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez (2023) MemGPT: towards LLMs as operating systems. arXiv preprint arXiv:2310.08560. Cited by: §2.1. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Advances in neural information processing systems 36, p. 68539–68551. Cited by: §1, §2.1. L. S. Shapley (1953) A value for n-person games. Vol. 2, Princeton University Press. Cited by: §2.3. N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36. Cited by: Appendix P, §1, §2.2, Table 1, §4.1. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023) Voyager: an open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. Cited by: §2.2, Table 1. X. Wang, N. Liao, S. Wei, C. Tang, and F. Xiong (2026a) AutoAgent: evolving cognition and elastic memory orchestration for adaptive agents. arXiv preprint arXiv:2603.09716. Cited by: §2.2. Y. Wang, J. Xu, H. Zhang, S. Huang, D. D. Sun, and X. Zhang (2026b) FactorMiner: a self-evolving agent with skills and experience memory for financial alpha discovery. arXiv preprint arXiv:2602.14670. Cited by: Appendix P, §2.2, Table 1, §4.1. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35. Cited by: §2.1. C. Wu, Y. Luo, Z. Qu, and M. Wang (2025) Meta-policy reflexion: reusable reflective memory and rule admissibility for resource-efficient llm agent. arXiv preprint arXiv:2509.03990. Cited by: Appendix P, §2.2, Table 1, §4.1. B. Xia, M. Hu, S. Wang, J. Jin, W. Jiao, Y. Lu, K. Li, and P. Luo (2026) Tool-genesis: a task-driven tool creation benchmark for self-evolving language agent. arXiv preprint arXiv:2603.05578. Cited by: §2.2. W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang (2025) A-mem: agentic memory for llm agents. arXiv preprint arXiv:2502.12110. Cited by: §1, §2.1. S. Yang, S. C. Han, X. Ma, Y. Li, M. R. G. Madani, and E. Hovy (2026) Evotool: self-evolving tool-use policy optimization in llm agents via blame-aware mutation and diversity-aware selection. arXiv preprint arXiv:2603.04900. Cited by: Appendix P, 1(a), §1, §2.2, Table 1, §4.1. S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan (2024) Tree of thoughts: deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36. Cited by: §2.1. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. Proceedings of the 11th International Conference on Learning Representations. Cited by: §1, §2.1. J. Zhang, B. Zhao, W. Yang, J. Foerster, J. Clune, M. Jiang, S. Devlin, and T. Shavrina (2026) Hyperagents. arXiv. External Links: 2603.19461 Cited by: §4.1, §5. A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang (2024) ExpeL: llm agents are experiential learners. Proceedings of the AAAI Conference on Artificial Intelligence 38. Cited by: Appendix P, 1(a), §1, §2.2, Table 1, §4.1. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig (2024) WebArena: a realistic web environment for building autonomous agents. Proceedings of the 12th International Conference on Learning Representations. Cited by: Appendix A, §1. Appendix A Limitations and Future Work AEL is domain-agnostic (requiring only a context vector and scalar reward), but we evaluate on a single financial domain with a 2-week test period. Cross-domain validation on coding (Jimenez et al., 2024) and web (Zhou et al., 2024) benchmarks is the key next step. All LLM methods share the same backbone, so data contamination is symmetric and does not confound AEL-vs-baseline comparisons. Scaling to larger tool sets and studying backbone model capability are natural extensions. Appendix B Hyperparameters Table 4 summarizes the key hyperparameters used in all AEL experiments. The Thompson Sampling bandit uses uninformative priors (α0=β0=1 _0= _0=1), allowing the posterior to be shaped entirely by observed rewards. LinUCB exploration is set to α=1.0α=1.0 following standard practice for moderate exploration. Memory retrieval uses top-k=5k=5 with a quality threshold of 0.3, balancing recall breadth against noise. Tier boosts (procedural 1.5×, semantic 1.2×, episodic 1.0×) prioritize distilled knowledge over raw episode logs. Reflection uses a lower temperature (0.3) than the main prediction calls to produce more consistent diagnostic outputs. These values were selected based on preliminary runs on the validation split and held fixed across all experiments. Table 4: Key hyperparameters used in AEL. Component Parameter Value LinUCB Exploration α / dim d 1.0 / 7 Thompson Initial α0,β0 _0, _0 1.0, 1.0 Memory Top-k / quality threshold 5 / 0.3 Max entries per tier 500 Retrieval Recency decay λ 0.01 Tier boosts (proc/sem/epi) 1.5/1.2/1.0 Credit Method (main config) uniform Reflection Temperature / fail threshold 0.3 / 3 days Appendix C Evaluation Metric Definitions All metrics are computed on frozen test-phase returns r1,…,rT\r_1,…,r_T\. Sharpe ratio measures risk-adjusted return: Sharpe=Tann⋅r¯/σrSharpe= T_ann· r/ _r, where r¯ r is mean per-bar return, σr _r is return standard deviation, and Tann=1008T_ann=1008 annualizes (252 trading days × 4 bars/day). Sortino ratio replaces total volatility with downside deviation σd=[min(r,0)2] _d= E[ (r,0)^2], penalizing only negative returns: Sortino=Tann⋅r¯/σdSortino= T_ann· r/ _d. Calmar ratio equals annualized return divided by |max drawdown||max drawdown|, measuring return per unit of worst-case loss. Return% is total cumulative test return: ∏t(1+rt)−1 _t(1+r_t)-1. MaxDD% is the largest peak-to-trough portfolio drawdown during test, a direct measure of capital preservation. Win rate (WinR) is the fraction of bars with positive return: |t:rt>0|/T|\t:r_t>0\|/T. Tail ratio (TailR) equals the 95th percentile gain divided by the absolute 5th percentile loss, capturing upside/downside asymmetry (>1>1 is desirable). Appendix D Implementation Protocol All experiments use Claude Haiku 4.5 (us.anthropic.claude-haiku-4-5) as the backbone LLM with temperature 0.3 for predictions and reflection. Each episode invokes all 12 tools (no tool selection in portfolio mode); the planner then synthesizes tool outputs into portfolio weights. A 15-bar warm-up period runs tool-only predictions before learning begins. Memory caps at 500 entries per tier with quality threshold 0.3 for writes. Semantic distillation occurs every 10 episodes. The LinUCB context vector is 7-dimensional (sector encoding, volatility, data richness, recent performance). The main configuration uses uniform credit; full hyperparameters are listed in Appendix B. The matched incremental build uses seeds 42, 123, 456; headline methods extend to 5 seeds (+789, +1024). Cost. Each full AEL run costs approximately $2.80 in LLM calls (~$2 for predictions, ~$0.80 for reflection/evolution across 208 episodes). Test-phase protocol. The primary metric is test-phase Sharpe ratio (annualized); secondary metrics are defined in Appendix C. During test, all learning is disabled: bandit posteriors are frozen, memory is read-only, and no evolution occurs, ensuring that test results reflect the agent’s generalized architecture. Appendix E Bandit Algorithm Details LinUCB for planner selection. For each planner π, LinUCB maintains a d×d× d matrix πA_π, a d-dimensional vector πb_π, and the parameter estimate θ^π=π−1π θ_π=A_π^-1b_π. At episode t, the planner is selected as: πt=argmaxπ(ϕt⊤θ^π+αϕt⊤π−1ϕt), _t= _π ( _t θ_π+α _t A_π^-1 _t ), where ϕt∈ℝ7 _t ^7 is the context vector and α controls exploration. After observing reward r~t(planner) r_t^(planner), the parameters are updated: πt←πt+ϕtϕt⊤A_ _t _ _t+ _t _t and πt←πt+r~t(planner)ϕtb_ _t _ _t+ r_t^(planner) _t. Thompson Sampling for tool and memory selection. For each tool or tool preset a, the controller maintains a Beta posterior μa∼Beta(αa,βa) _a ( _a, _a). At each episode, a reward estimate μ~a μ_a is sampled from each arm’s posterior, and the highest-sampled preset or the top-K sampled tools (in per-tool mode) are selected. After observing the module-specific reward, the posterior is updated: αa←αa+r~t _a← _a+ r_t and βa←βa+(1−r~t) _a← _a+(1- r_t). Memory policies use an identical Thompson Sampling mechanism over a growing pool. Appendix F Credit Assignment Details The main AEL configuration uses uniform credit (gt(m)=1/3g_t^(m)=1/3 for all modules). The methods below are evaluated in the credit comparison study (Table 3). Structural credit extracts deterministic attribution from the execution trace. For tools: gtstruct,tools=(correct−incorrect)/totalg_t^struct,tools=(correct-incorrect)/total, measuring directional accuracy against realized outcomes. For planners: credit is based on step completion and prediction accuracy. For memory: credit measures the fraction of retrieved entries marked as useful. Counterfactual credit compares the actual outcome sts_t with the outcome st−ms_t^-m obtained when module m is replaced by its default: gtcounter,m=st−st−mg_t^counter,m=s_t-s_t^-m. Shapley credit, computed periodically (every N episodes), enumerates all 23=82^3=8 coalitions over the three modules and computes each module’s marginal contribution weighted by the Shapley coefficient |S|!(n−|S|−1)!/n!|S|!\,(n-|S|-1)!/n!. FCC combination: cm=0.2cmstruct+0.3cmcounter+0.5cmshapc_m=0.2\,c^struct_m+0.3\,c^counter_m+0.5\,c^shap_m. Module-specific reward derivation. The episode score st∈[−1,1]s_t∈[-1,1] is first normalized: rt=clip((st+1)/2, 0, 1)r_t=clip((s_t+1)/2,\;0,\,1). The module-specific reward blends the global outcome with per-module credit: r~t(m)=clip(λrt+(1−λ)gt(m), 0, 1), r_t^(m)=clip\! (λ\,r_t+(1-λ)\,g_t^(m),\;0,\,1 ), where λ∈[0,1]λ∈[0,1] controls the blend (we use λ=0.5λ=0.5) and gt(m)g_t^(m) is the credit for module m∈planner,tools,memorym∈\planner,tools,memory\. Appendix G Controlled Single-Variable Experiments Note on code version. These controlled experiments were conducted before credit-system bug fixes (commits b9604f7, f6372a4). After those fixes, both FCC and LLM-FCC collapsed (−-0.07 and −-0.53 Sharpe respectively), while the uniform-credit AEL configuration remained strong (2.13). The results below therefore reflect the pre-fix code state and should be interpreted as an investigation of credit-method sensitivity, not as current production numbers. To isolate individual effects, we ran controlled experiments where each configuration modifies exactly one parameter from the AEL-FCC baseline. All controlled experiments use the full 5-seed evaluation (42, 123, 456, 789, 1024). Figure 5: Pre-fix controlled single-variable ablation from AEL-FCC baseline (dashed line at 1.51). Each bar changes one parameter. Removing per-tool selection or warm-up collapses performance. Uniform credit provides a modest gain; LLM-FCC also improves over FCC. These results predate credit-system bug fixes; post-fix FCC and LLM-FCC both collapsed (see text). Table 5: Controlled single-variable results (5 seeds). Each row changes one parameter from AEL-FCC. Change s42 s123 s456 s789 s1024 Mean AEL full (FCC baseline) 2.77 1.32 1.95 0.97 0.53 1.51 ++ uniform credit only 3.02 1.58 1.88 2.69 0.61 1.96 −- per-tool selection 0.61 −-1.34 0.63 1.63 −-0.21 0.26 −- warm-up episodes −-1.23 −-1.22 2.57 0.50 −-1.32 −-0.14 FCC tuned (intv=40) 2.82 −-1.05 0.65 2.54 1.51 1.29 ++ LLM-FCC credit 3.10 1.11 3.48 −-0.94 2.60 1.87 Removing per-tool Thompson selection (−-1.25) or warm-up (−-1.65) degrades performance substantially, revealing these as essential infrastructure. In this pre-fix setting, uniform credit (1.96) outperformed both FCC (1.51) and LLM-FCC (1.87). However, after credit-system bug fixes, both FCC (−-0.07) and LLM-FCC (−-0.53) collapsed while AEL with uniform credit remained at 2.13, indicating that the pre-fix credit results were partially driven by buggy interactions. Credit assignment for multi-module agent evolution remains an important open problem. Appendix H All Methods Comparison Figure 6: All methods ranked by test-phase Sharpe on D-full (N=5N=5 seeds). Error bars show ± 1 std. AEL achieves the highest Sharpe (2.13± 0.47) with the lowest variance among LLM methods. Appendix I Detailed Results Analysis This section provides extended analysis complementing the main-text results. I.1 Prior Method Fragility (Q1) With 5-seed evaluation, the prior methods show high variance that inflates individual-seed results. Reflexion averages −-0.59± 1.33 Sharpe: 4 of 5 seeds are negative, with a single outlier (s789=+1.54) pulling the mean up. ExpeL (0.76± 1.93) depends almost entirely on one seed (s123=+4.58; the other four average −-0.20). FactorMiner (0.85± 1.15) is the most balanced prior method, with two positive and three near-zero seeds. Meta-Reflexion (0.20± 1.16) and EvoTool (1.37± 1.75) show similarly high variance. The deterministic momentum-weighted baseline (Sharpe 1.44) remains competitive with all prior methods, underscoring that the observed LLM-based gains are fragile. HyperAgent, despite receiving tuned hyperparameters (20 generations, real tool schemas, epsilon acceptance), achieves only 0.46± 0.41, the lowest variance among prior methods but also among the lowest means. The strategies that successfully learn from training data tend to overfit to the training regime (Section 5). I.2 Component Synergy Mechanism (Q2, Q4) With 5-seed evaluation, the incremental build reveals a clear progression: Stateless (1.35) → +Memory (1.68) → AEL (2.13). Memory provides a 24% improvement by enabling cross-episode learning. Reflection then produces a further 27% jump by diagnosing failure patterns and enhancing memory quality. Specifically, the LLM identifies which memories are misleading, which retrieval strategies work for which market conditions, and how to consolidate episodic experiences into reusable semantic knowledge. The mechanism is that reflection acts as a quality filter for the memory system: without reflection, memory accumulates experience indiscriminately; with reflection, it learns to use experience selectively. 3(b) confirms this: AEL has both the highest mean (2.13) and the lowest variance (± 0.47) among all LLM methods. EvoTool achieves competitive mean (1.37) but with much wider spread (± 1.74). I.3 Ablation Analysis (Q2) The component ablation (Table 3) uses post-fix code and ablates directly from the AEL configuration (Ours, 2.13). Removing reflection (Δ −-0.45). Without reflection, performance drops from 2.13 to 1.68 (+Memory). Reflection is the only component that generates new knowledge about the agent’s own performance, converting raw experience into actionable patterns stored in semantic memory. Adding complexity hurts. The ablation (Figure 4b) shows that every addition to AEL degrades performance. Planner evolution (Δ −-1.72) and per-tool Thompson selection (Δ −-1.70) are the most harmful: in a 208-episode horizon, exploration overhead exceeds the adaptation benefit. Cold-start initialization (Δ −-1.31) and skill extraction (Δ −-1.11) similarly overfit learned artifacts to training conditions. Credit methods (FCC Δ −-1.09, LLM-FCC Δ −-0.64) add attribution noise that degrades memory bandit learning. Implications. The optimal configuration is the simplest learning configuration: fixed planner, uniform credit, memory with reflection. This “less is more” pattern suggests that in short-horizon, high-noise domains, the overhead of adaptive mechanisms (bandit exploration, credit estimation) outweighs their benefit. The key enabler is reflection, which provides diagnostic capability without requiring complex infrastructure. I.4 Credit Assignment Details (Q3) The pre-fix controlled study (Appendix G) showed a ranking: uniform credit (1.96) >> LLM-FCC (1.87) >> FCC (1.51) >> FCC tuned (1.29). However, after credit-system bug fixes (negative-credit clamping, z-score normalization, graded tool rewards), both FCC (−-0.07) and LLM-FCC (−-0.53) collapsed entirely, while the uniform-credit AEL configuration remained strong at 2.13. This suggests the pre-fix credit results were partially driven by buggy interactions (e.g., negative credits accidentally regularizing Thompson posteriors), and that principled credit assignment in high-noise domains remains an open challenge. The pre-fix infrastructure experiments showed that per-tool Thompson selection (removal: 1.51 → 0.26, Δ −-1.25) and warm-up episodes (removal: 1.51 → −-0.14, Δ −-1.65) were critical. These infrastructure findings likely generalize beyond the credit-method bugs, as they reflect fundamental learning dynamics rather than credit-specific interactions. Appendix J Detailed Case Study: HyperAgent This section provides the full analysis of HyperAgent’s behavior, complementing the summary in Section 5. J.1 Evolution Log We gave HyperAgent every advantage for a fair comparison: 20 generations (doubled from the default 10), real tool-output key schemas in the meta-prompt, epsilon-acceptance (≥ best −- 0.05) to encourage exploration, and higher temperature (0.7) for code diversity. Despite these improvements, HyperAgent achieves a mean test Sharpe of only 0.46± 0.41 across 5 seeds. Table 6 shows a representative evolution log (seed 42). With the schema fix, generation 0 now improves training Sharpe from −-0.974 to −-0.44 by actually using tool signals. However, most subsequent generations fail code validation (the LLM generates imports that are blocked by the sandbox), and the strategies that do pass tend to overfit the training regime. Table 6: HyperAgent evolution log (seed 42, tuned version with 20 generations). Gen 0 shows initial improvement from tool schema, but most later generations fail validation. Gen Status Train Sharpe 0 Accepted (tool signals used) −-0.438 1 Accepted (epsilon) −-0.670 2 Accepted (epsilon) −-0.464 3–19 Validation failed (17/17) — Test Sharpe (frozen) 0.496 The 5-seed results reveal an overfitting pattern: seeds where the evolved code deviates from equal-weight (s42=0.50, s789=−-0.32) perform worse at test time than seeds that fall back to equal-weight (s123=0.69, s1024=0.69). The best seed (s456=0.83) is the only one where the learned strategy generalizes, but it still underperforms AEL with reflection (2.13). J.2 Root Cause Analysis Even with our tuned configuration, HyperAgent’s failure modes illuminate why AEL’s design choices are necessary. First, code generation quality collapses beyond the initial improvement. While providing real tool-output schemas enables generation 0 to produce working code (train Sharpe improves from −-0.974 to −-0.44), subsequent generations overwhelmingly fail validation (85% failure rate in the tuned version). The LLM generates increasingly complex code that includes blocked imports or access patterns incompatible with the sandbox. AEL avoids this by constraining code evolution to modular components with well-typed interfaces. Second, batch evaluation causes overfitting. The strategies that pass validation and improve training Sharpe often overfit to the training regime: seed 789 achieves improved training performance but collapses to −-0.32 at test time. AEL’s online per-episode learning avoids this: the bandit continuously adapts based on recent feedback rather than optimizing a single batch metric. Third, HyperAgent lacks diagnostic capability. When the allocation function fails, the meta-agent knows only the aggregate Sharpe but not which signals were misleading or which allocation decisions were wrong. AEL’s reflection system diagnoses failure patterns and targets memory improvements, enabling targeted adaptation rather than wholesale code rewriting. J.3 Structural Comparison Table 7 summarizes the structural differences. The key insight is that modular evolution with credit assignment preserves locality: changes to one component do not destroy progress in others. This is analogous to how biological evolution operates through modular gene regulation rather than genome-wide random rewriting. Notably, the improvements HyperAgent would need to succeed (sliding-window evaluation, modular code generation, per-component signals) would make it architecturally similar to AEL, suggesting that AEL’s design reflects fundamental requirements for self-improving agents in complex sequential tasks. Table 7: Structural comparison: why modular evolution with credit beats unconstrained code rewriting. Dimension HyperAgent AEL Modification scope Entire function Individual modules Learning signal 1 Sharpe / generation 1 reward / episode Diagnostic capability None (aggregate) Reflection-based Evolution cost ∼ 20 LLM calls ∼ 400 LLM calls Failure mode Overfits or stuck Gradual improvement Test result (5 seeds) 0.46± 0.41 2.13± 0.47 Appendix K Dataset Details Table 8: Full D-full benchmark statistics. Training includes diverse regimes (bull, bear, flat); the test set contains a bear-to-bull transition. Property Value Task Portfolio weight allocation (10 tickers + cash) Tickers 10 sector-diverse equities Sectors 7 GICS sectors (Tech, Healthcare, Finance, Energy, …) Frequency 1-hour bars (4 per trading day) Train period Jan 6 – Feb 21, 2025 (7 weeks, 140 bars) Train regimes 3 bull + 2 bear + 2 flat weeks Validation Feb 21 – Mar 7, 2025 (2 weeks, 40 bars) Test period Mar 7 – Mar 21, 2025 (2 weeks, 28 bars) Test regimes Bear (W11) → Bull (W12) transition Total episodes 208 bars Action space Weight vector w∈Δ10w∈ ^10 (simplex) Reward Per-bar portfolio return Table 9: D-full: 10 sector-diverse tickers spanning 7 GICS sectors. Ticker Sector Market Cap Train/Val/Test AAPL Technology Mega 60/20/20 NVDA Technology Mega 60/20/20 JNJ Healthcare Mega 60/20/20 UNH Healthcare Mega 60/20/20 JPM Finance Mega 60/20/20 GS Finance Large 60/20/20 XOM Energy Mega 60/20/20 PG Consumer Mega 60/20/20 CAT Industrial Large 60/20/20 NEE Utilities Large 60/20/20 Appendix L Tool Descriptions All methods share the same 12-tool finance registry for fair comparison, so performance differences cannot be attributed to tool access alone. The registry is intentionally heterogeneous: some tools expose raw state from the cached market data, others compute derived signals, and a final layer produces decision-oriented summaries. This separation is important for AEL because different planners may rely on different levels of abstraction, while the credit-assignment mechanism must be able to inspect both low-level evidence and high-level recommendations. The tool set is also deliberately redundant in a useful way. Price-based tools capture short-horizon market structure, fundamentals and DCF provide slower valuation anchors, analyst/options/earnings tools expose event-driven and sentiment information, and correlation/risk tools help the planner reason about portfolio-level diversification rather than single-ticker alpha alone. Table 10 summarizes the role of each tool in the experiments. Table 10: Finance tools used in all experiments. The appendix view here emphasizes not only what each tool returns, but why that signal is useful for sequential portfolio allocation. Tool Main output Why it matters Data retrieval tools get_price_history Recent OHLCV bars, latest close, highs/lows, trading volume Anchors every price-based decision. It gives the planner direct access to recent regime, trend, and liquidity information, and it is the upstream dependency for several derived tools. get_fundamentals Financial statements, profitability, leverage, growth, and valuation ratios Provides a slower-moving view of firm quality and balance-sheet strength. This is important because short-term price moves alone can be noisy, while fundamentals offer a medium-horizon anchor for capital allocation. get_analyst_data Target prices, consensus recommendations, upgrade/downgrade history Adds an external sentiment and expectations channel. Analyst revisions can signal changing market narratives that are not yet obvious from raw prices. get_options_data Implied volatility, put/call ratios, open interest summaries Exposes forward-looking positioning and hedging demand. This is especially useful for distinguishing bullish price action from fragile, high-volatility moves. get_earnings_data Quarterly earnings, revenue, and earnings-calendar metadata Captures event risk and recent fundamental surprises. Around earnings windows, the planner needs to know whether a signal is driven by a durable trend or a transient post-event reaction. Computation tools compute_technicals RSI, MACD, Bollinger bands, moving averages, support/resistance, technical score Converts raw prices into structured timing signals. These indicators help the planner reason about overbought/oversold conditions, trend continuation, and whether current prices are stretched relative to local history. compute_quant_risk Realized volatility, VaR/CVaR, Sharpe, Sortino, max drawdown, beta/alpha Makes downside risk explicit rather than implicit. In portfolio allocation, avoiding bad concentration and tail exposure is as important as finding upside, so this tool supports risk-aware sizing decisions. compute_momentum Multi-horizon returns, trend slope, trend strength, volume trend Measures continuation across several horizons instead of relying on a single lookback. This matters because different assets express momentum at different speeds, and volume confirmation helps separate genuine trends from weak drift. compute_correlations Cross-ticker correlation matrix and rolling correlation to target ticker Moves the planner from single-name prediction to portfolio construction. High correlation means apparently strong single-stock signals may be redundant once existing exposures are considered. Analysis tools run_dcf_model Bull/base/bear DCF scenarios or simplified implied-upside valuation signal Supplies an intrinsic-value estimate that can disagree with recent price action. This is useful for distinguishing momentum-driven trades from opportunities where valuation support exists. score_risk Overall 1–10 risk rating plus valuation/financial/growth/macro/technical sub-scores Compresses several risk dimensions into a planner-friendly summary. This makes it easier to compare heterogeneous tickers and avoid allocations that are attractive on return but unacceptable on fragility. score_composite_signal Weighted BUY/SELL/HOLD style summary using technical, momentum, valuation, analyst, options, and risk inputs Acts as a high-level synthesis layer. It is useful when the planner wants a compact recommendation, but it is also diagnostically important because the credit module can inspect whether the fused signal helped or obscured the true decision. Two design choices are worth noting. First, the registry includes both primitive and aggregated tools rather than forcing a single abstraction level; this gives the planner freedom to use direct evidence when needed and high-level summaries when time is limited. Second, several tools overlap on purpose. For example, momentum, technicals, analyst sentiment, and composite scoring may all point in the same direction during a strong trend, but they diverge during regime shifts; those disagreements are exactly the kind of cross-module evidence that makes credit assignment informative in our setting. Appendix M Planner and Memory Policy Families For the main AEL benchmark configuration, the planner pool is initialized with six built-in planners and the memory-policy registry is initialized with five default retrieval policies before any learned variants are added. These are the concrete families from which the meta-controller selects during training. The simpler incremental ablations intentionally restrict this space, often to a single sequential planner or a reduced memory setup, but the full benchmark uses the richer families summarized below. Table 11: Built-in planner families used by AEL in the main benchmark configuration. Dynamic planners generated by the slow-timescale evolution loop are added on top of this initial pool. Planner Core principle Role in the benchmark sequential Run all available tools in a fixed order, then synthesize once. Serves as the most stable and exhaustive baseline planner. It maximizes coverage and minimizes strategic assumptions, but can be expensive and prone to information overload. decompose Break the task into valuation, momentum, sentiment, and risk sub-problems, then synthesize sub-results. Encourages structured analysis and lets the agent reason about different evidence types separately before combining them into a portfolio decision. adaptive Start with a cheap quick-look tool set, then invoke deeper tools only if the initial signal is ambiguous. Provides an efficiency-oriented planner that trades off speed and depth, which is useful when some episodes are easy while others require broader evidence. cot_reasoning Analyze trend, valuation, sentiment, and risk sequentially with explicit intermediate synthesis. Forces a chain-of-thought-style evidence path rather than a flat aggregation of all tool outputs, which can help when conflicting signals need to be resolved step by step. reflexion Make a quick prediction, self-check confidence, and gather more evidence only if the initial judgment is weak. Adds an intra-episode self-correction behavior: the planner first tests whether existing evidence is sufficient and only expands the search when confidence is low. hypothesis_test Form bull and bear hypotheses, gather targeted evidence for each, then weigh the two cases. Makes the planner explicitly compare competing market narratives instead of only aggregating signals, which is useful in reversal or mixed-regime episodes. These planners are intentionally diverse. Some are exhaustive (sequential), some are decompositional (decompose, cot_reasoning), and some are selective (adaptive, reflexion, hypothesis_test). The contextual planner bandit does not assume any one reasoning style is globally best; instead, it learns which planning style works better under which market context. In addition, procedural and semantic memory can later modify planner behavior by appending learned strategy hints to planner prompts. Table 12: Initial memory-policy families used by AEL in the main benchmark configuration. New retrieval policies may be added later by the slow-timescale evolution loop. Policy Enabled tiers Format Retrieval principle none none none Disable memory entirely. This is the no-retrieval option and is important because some episodes are better solved from current market evidence alone. recent_window episodic sliding_window Retrieve recent episodic memories and keep only a small first-plus-last window. This preserves a few anchors and a few recent cases without flooding the planner with raw logs. full_detailed episodic, semantic, procedural full Return all retrieved memories from all tiers verbatim. This is the highest-information policy, useful when the agent benefits from both concrete cases and abstract rules. compressed semantic, procedural ranked_truncate Focus on abstracted knowledge, rank retrieved memories by relevance, and truncate to a token budget. This is the default “high signal, low clutter” option for using distilled experience. aggressive_learner episodic, semantic, procedural ranked_truncate Retrieve more memories with a larger token budget and a more permissive learning setup. This policy is useful when the system is still exploring and wants to exploit a broader experience base. The key design choice is that memory policies control how experience is exposed to the planner, not just whether memory exists. The policy determines which tiers are visible, how many memories are retrieved, and whether the planner sees raw cases, compact abstractions, or a larger but noisier experience bundle. This is why the memory-policy bandit is necessary: the best retrieval strategy depends on the task context and on the maturity of the memory store itself. Appendix N Inference Workflow Algorithm 2 Inference workflow of AEL 1:Freeze module pools, bandit posteriors, and memory writes. 2:for each test episode ete_t do 3: Extract task features ϕt _t. 4: Select (pt,zt,mt)(p_t,z_t,m_t) using the frozen meta-controller. 5: Retrieve read-only memory under policy mtm_t. 6: Execute planner ptp_t with tools ztz_t to produce y^t y_t. 7: Return prediction (no updates, reflection, or evolution). 8:end for Appendix O Detailed Analysis O.1 LLM Credit Assignment Example Below is a representative LLM credit assignment output from episode 85 (seed 42, bearish regime). The LLM receives each module’s output alongside the ground-truth bar return, and assigns credit scores in [−1,+1][-1,+1]. Input context: Bar 85 (2025-02-10 14:30). Actual portfolio return: −-0.31%. Planner selected adaptive strategy → overweighted tech (NVDA 18%, AAPL 15%). Tools: compute_momentum signaled bullish for NVDA (+0.8); compute_quant_risk flagged elevated VIX (score 7/10); score_composite_signal returned HOLD. Memory: Retrieved semantic memory: “Tech rally stalls when VIX exceeds 20” (relevance 0.72). LLM credit output: planner: -0.6 “The adaptive planner overweighted tech despite conflicting risk signals. It should have reduced concentration when VIX was elevated.” tools: +0.3 “Momentum correctly identified the short-term trend, but quant_risk provided a valid warning that was ignored by the planner. Net positive for tools.” memory: +0.4 “The retrieved memory about VIX and tech was highly relevant and should have been weighted more heavily. Memory retrieval was helpful; the failure was in how the planner used it.” This illustrates how LLM credit can distinguish “the tool gave a correct warning that was ignored” from “the tool gave a wrong signal,” a distinction that Shapley-based methods cannot make because they treat modules as black boxes. O.2 Code Evolution Examples The following are real artifacts produced by AEL’s code evolution mechanism during training. LLM-generated planner class (episode 120, seed 42). After reflection diagnosed that the sequential planner was too slow to react to intraday reversals, the LLM generated a new MomentumReversalPlanner: class MomentumReversalPlanner(BasePlanner): """Reduce position when momentum reverses intraday.""" def plan(self, context): signals = context["tool_outputs"] momentum = t: s["compute_momentum"]["trend_score"] for t, s in signals.items() prev_momentum = context.get("prev_momentum", ) weights = for ticker in context["tickers"]: curr = momentum.get(ticker, 0) prev = prev_momentum.get(ticker, 0) if curr * prev < 0: # reversal weights[ticker] = 0.05 # minimal position else: weights[ticker] = max(0.02, 0.1 * abs(curr)) return self.normalize(weights) LLM-generated memory retrieval policy (episode 95, seed 123). After observing that default retrieval returned too many irrelevant bull-market memories during a bear regime, the LLM designed a regime-filtered retrieval policy: class RegimeFilteredRetrieval(BaseRetrievalPolicy): """Filter memories by current regime before scoring.""" def retrieve(self, query, memories, k=5): regime = query.get("current_regime", "unknown") filtered = [m for m in memories if m.get("regime") == regime or m.get("tier") == "procedural"] if len(filtered) < k: filtered = memories # fallback scored = self.score_by_relevance(query, filtered) return scored[:k] Both artifacts were validated via AST parsing and accepted into the module pool by LinUCB, which subsequently selected them when their context features matched. Appendix P Baseline Adaptation Details All baselines use the same LLM backbone (Claude Haiku 4.5), tool set (12 financial tools), and portfolio allocation interface. Below we describe what was kept from each original method and what was adapted. Reflexion (Shinn et al., 2023). Kept: The core mechanism of accumulating verbal self-critiques after each episode. Reflections are prepended to future prompts as a growing context window. Adapted: Applied to portfolio allocation with a financial-specific reflection prompt (“Write a 1-sentence reflection on what signals to trust or ignore”). Maximum 20 reflections retained (FIFO eviction). No structured memory tiers; only flat string accumulation. ExpeL (Zhao et al., 2024). Kept: The experience extraction mechanism where the LLM distills episodes into reusable “lessons” stored in a flat lesson store, retrieved by keyword similarity. Adapted: Lessons are keyed by ticker and sector for retrieval (same-ticker: +2.0, same-sector: +1.0). Maximum 100 lessons. Extraction prompt asks for generalized rules prefixed with “RULE:”. No bandit-based policy selection. FactorMiner (Wang et al., 2026b). Kept: Dual-tier memory with skill extraction from successful tool combinations and experience memory with outcome tracking. Adapted: Skills are named by sorted tool sequences from correct predictions (≥ 2 tools). Success rates use exponential moving average. Maximum 15 skills and 200 experiences. No joint evolution or credit assignment across modules. Meta-Reflexion (Wu et al., 2025). Kept: Rule distillation from accumulated reflections, with admissibility checking that prunes contradicted or low-success rules. Adapted: Distillation runs every 5 episodes, extracting “RULE:” lines from the 10 most recent reflections. Rules with <<0.3 success rate and ≥ 5 applications are pruned. Maximum 10 active rules. No tool or planner evolution. EvoTool (Yang et al., 2026). Kept: Population-based evolutionary optimization of tool-selection policies with blame-aware mutation. Fitness-proportional selection across a population of 5 policies. Adapted: Blame attribution uses signal-level analysis (e.g., if compute_momentum signaled bullish but the actual direction was bearish, that tool is blamed). Mutations remove blamed tools, add random tools (50% chance), or swap tools (30% chance). Minimum 3 tools per policy. No memory system or planner evolution. Appendix Q Transaction Cost Sensitivity A practical concern for portfolio allocation systems is whether the reported Sharpe ratios survive realistic transaction costs. Since the D-full benchmark uses hourly bars (4 per trading day), frequent rebalancing can generate substantial turnover. To assess this, we retroactively apply proportional transaction costs to the recorded portfolio weight changes at each bar. Specifically, for a cost level of c basis points per unit of turnover, the cost-adjusted return at bar t is rtadj=rt−c⋅∑i|wi,t−wi,t−1|r_t^adj=r_t-c· _i|w_i,t-w_i,t-1|, where wi,tw_i,t is the portfolio weight of ticker i at bar t. Table 13 reports cost-adjusted Sharpe ratios at four cost levels spanning the range from zero-cost (0 bp, the main benchmark setting) to institutional-level costs (20 bp, typical for large-cap equity rebalancing with market orders). Table 13: Cost-adjusted Sharpe ratio at varying transaction cost levels (5-seed means). Costs are applied retroactively based on recorded turnover at each bar. Method 0 bp 5 bp 10 bp 20 bp AEL 2.13 ∼ 1.9 ∼ 1.7 ∼ 1.3 momentum_weighted 1.44 ∼ 1.3 ∼ 1.2 ∼ 1.0 EvoTool 1.37 ∼ 1.2 ∼ 1.0 ∼ 0.7 Stateless 1.35 ∼ 1.2 ∼ 1.0 ∼ 0.7 Several observations are worth noting. First, AEL remains the top-performing method at all cost levels tested, maintaining a Sharpe above 1.3 even at 20 bp. Second, the cost degradation is moderate (∼ 0.8 Sharpe from 0 to 20 bp) because AEL’s reflection mechanism tends to produce stable allocation strategies that avoid excessive turnover; the agent learns through reflection that frequent large weight shifts are penalized by the market. Third, the momentum-weighted baseline is relatively cost-resilient (Sharpe 1.0 at 20 bp) because its allocation weights change smoothly by construction, while EvoTool’s evolutionary mutations can produce abrupt policy changes that incur higher turnover costs. The cost estimates at 5, 10, and 20 bp are based on average turnover statistics from logged portfolio histories and should be interpreted as approximate; the cost_adjusted_sharpe function in the codebase enables exact computation from individual run logs.