Paper deep dive
Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces
Fangfei Li, Chenyang Zhao, Long Wang, Feng Tian, Lv Guo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-evolving agents are usually scored by endpoint Gain: held-out success after an update minus success before. That difference can credit a frozen rescore as evolution. We introduce Trace Evolution Bench (TEB), a protocol that separates that rescore from a context or weight update, together with three frozen 80/20/40 packs. An update is accepted only when validation does not fall, held-out identifiers never enter the update, and a text update is not ranked with a weight update. The primary scoreboard is coding, graded by hidden unit tests in a closed workspace. On that pack (40 held-out tasks, one family), DeepSeek v4 Pro under no_evolve never changes its policy yet moves 7.5, 5.0, and 12.5 percentage points across three seeds. Of nine text-update cells inspired by GEPA, ACE, and ExpeL, seven have positive excess above that seed's own band, and one gepa_llm seed moves down by the width of its band (-12.5, regression 0.48). GLM and MiniMax do not reproduce that seven-of-nine pattern. No method stays above its own band on every seed of all three models: GLM has two paired cells above their own bands, and MiniMax expel_llm seed 44 is a third (regression 0.30). The methods were fixed on a validation screen before held-out was read. Office, graded by rule-bounded delivery, and manufacturing table QA are separate packs: office DeepSeek does not exceed its own bands (widest 17.5 points), and manufacturing has no paired positive excess beyond its seed band.
Tags
Links
- Source: https://arxiv.org/abs/2610.15055v1
- Canonical: https://arxiv.org/abs/2610.15055v1
Trouble viewing inline? Open PDF directly →
Full Text
117,010 characters extracted from source content.
Expand or collapse full text
Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces Fangfei Li, 1 Chenyang Zhao, 1 Long Wang, 1 Feng Tian, 1 Lv Guo 1,∗ 1 MatrixOrigin, China ∗ Corresponding author. lifangfei@matrixorigin.cn, zhaochenyang@matrixorigin.cn, wanglong@matrixorigin.io, tianfeng@matrixorigin.cn, guolv@matrixorigin.cn ∗ Abstract Self-evolving agents are often judged by endpoint Gain (final held-out success minus round-0 suc- cess), which can conflate true evolution with frozen rescoring variance. We present Trace Evolution Bench (TEB), an evaluation protocol that isolates attributable improvement through four controls: locked held-out identifiers, validation-gated accept/rollback, repeated frozen noevolve rescoring, and seed- matched paired excess against each seed’s own noise band. TEB includes three frozen 80/20/40 packs, with coding as the primary scoreboard because success is determined by hidden unit tests in a closed workspace. On coding (40 held-out tasks), DeepSeek v4 Pro under noevolve keeps version M 0 yet shifts by 7.5, 5.0, and 12.5 percentage points across seeds, showing that endpoint Gain alone overstates progress. Under this control, seven of nine DeepSeek text-update cells exceed their seed bands, but the pattern does not reproduce on GLM and MiniMax; no method remains above its own band on every seed of all three models. On separate packs, office DeepSeek does not exceed its seed bands (widest 17.5 points), and manufacturing has no paired positive excess beyond its seed band. These results position TEB as a reproducible attribution contract for self-evolving agents. 1 Introduction Most agent benchmarks freeze a policy and score static capability [Liu et al., 2024, Jimenez et al., 2024, Yao et al., 2023, Zhou et al., 2024, Mialon et al., 2024, Shridhar et al., 2021]. Self-evolving systems pose a different measurement problem: whether post-update improvement is attributable to the evolution pro- cedure rather than to evaluation noise or protocol mismatch [Silver et al., 2017]. In practice, attribution is confounded when teacher traces substitute for self-generated traces [Ross et al., 2011], when evaluation targets a checkpoint different from the artifact returned by evolve(), when held-out ids leak into the update [Magar and Schwartz, 2022], or when text updates and weight updates are compared under one budget axis [Henderson et al., 2018]. Reporting Success % alone can hide weak round-0 baselines [Liang et al., 2023]; reporting Gain from one stochastic round-0 pass can credit frozen rescoring variance as evolu- tion [Dodge et al., 2019]. These issues motivate an attribution-first contract, not only a final-score report. Trace Evolution Bench (TEB) operationalizes this contract with locked held-out ids, validation-gated accept/rollback, frozen no evolve rescoring, and seed-matched paired excess. We report coding as the primary scoreboard because success is determined by hidden unit tests in a closed workspace: tests, rather than phrasing, decide each pass. This setting is harder to game than office’s rule-bounded delivery and broader than manufacturing table QA. The three scored methods were fixed before held-out reading, via an office validation screen whose mean gains were not positive (Section 4). On coding, DeepSeek v4 1 arXiv:2610.15055v1 [cs.AI] 10 Oct 2026 Pro [DeepSeek-AI, 2026] noevolve keeps version M 0 yet moves 7.5, 5.0, and 12.5 percentage points across four passes of the same 40 held-out tasks (Table 6), showing that endpoint Gain still contains frozen rescore. Under paired excess, seven of nine DeepSeek text-update cells lie above their own seed bands, while gepa llm seed 44 moves down by the width of its band (Table 8, Figure 4). GLM and MiniMax do not reproduce that seven-of-nine pattern (Figure 6, Table 9); no method stays above its own band on every seed of all three models. GLM has two paired cells above their own bands, and MiniMax expel llm seed 44 is a third, with final regression 0.30. Office and manufacturing are interpreted as separate control packs on their own bands: office DeepSeek does not exceed its bands (widest 17.5 p; Section 6), and manufacturing has no paired positive excess beyond its seed band (Section 5). Contributions. (1) An attribution-first evaluation contract for self-evolving agents: locked held-out ids, validation-gated accept/rollback, frozen noevolve rescoring, and separate budget classes for B0 text updates versus B2 weight updates. (2) A reproducible diagnostic layer for gain claims: per-seed frozen noise bands, seed-matched paired excess, round-0 pairing checks, and explicit regression/eligibility markers that separate scoring evidence from ranking policy. (3) Cross-model evidence on the coding primary scoreboard: DeepSeek shows seven of nine cells above their own bands, but this pattern does not reproduce on GLM and MiniMax; no method stays above its own band on every seed of all three models. (4) Three frozen 80/20/40 packs with separate interpretation lanes: coding as primary evidence, office and manufacturing as independent controls on their own seed bands. 2 Related work ReAct-style loops [Yao et al., 2023] and agent suites [Liu et al., 2024, Jimenez et al., 2024] grade a frozen policy. Reflexion [Shinn et al., 2023], ExpeL [Zhao et al., 2024], self-refinement [Madaan et al., 2023], workflow memory [Wang et al., 2025], prompt evolution [Agrawal et al., 2026], skill libraries [Wang et al., 2023], and open-ended mutation [Zhang et al., 2026a] are systems TEB can host. The adapters are method- inspired text updates on one frozen actor, and the same three methods are scored on coding, office, and manufacturing. The quantity compared with those systems is whether an update beats a frozen rescore, not their published scores. Three protocol choices separate TEB from that set. The held-out ids never enter evolve(). Neighbors that mix the test ids into the update, or that do not say which split is locked, are not the same contract [Magar and Schwartz, 2022]. A validation gate accepts a proposal only when validation success does not fall. Held- out success is scored after that decision, on the accepted artifact, and is not the accept test. Weight updates are a separate budget class. A B0 text update is not ranked against a B2 weight update, including LoRA-style training [Hu et al., 2022]. Table 1 is the evaluation question. Table 2 is the feature contrast under binary coding, using a stated-evidence review.✓ marks explicit support in the paper’s main protocol, and × marks all other cases (including not stated). Nguyen et al. [2026] commit a candidate only when held-out performance does not regress, so the validation-gate and held-out-lock cells are crosses. Zheng et al. [2026] separate training, validation, and held-out views but do not state which split accepts an update. Shi et al. [2026] separate exploration from strict held-out evaluation and compare history, summary-memory, and training pathways in one framework. The freeze-weights row is a cross where the benchmark trains: Meng et al. [2026], Kang et al. [2026], Yang et al. [2026], Yuan et al. [2026], Rank et al. [2026], and Shi et al. [2026]. PostTrainBench also forbids training on the test set and scores the submitted checkpoint. Those papers also discuss transfer, business tasks, frozen artifacts, retained experience, and whether one financial case helps the next; none of those properties is a row of Table 2 [Gao et al., 2026, Zhou et al., 2026, Lin et al., 2026, Xue et al., 2026, Deng et al., 2026]. Zheng et al. [2025] and 2 Table 1: Evaluation target of each benchmark. A frozen rescore band is the range of repeated held-out passes of one unchanged policy. BenchmarkScored unitFrozen rescore band SWE-bench, AgentBench [Jimenez et al., 2024, Liu et al., 2024] One episodeNo LifelongAgentBench [Zheng et al., 2025] A task sequenceNo SEA-Eval [Jiang et al., 2026]A task sequenceNo EvoAgentBench [Gao et al., 2026]Transfer on an ability-supported splitNo; one vanilla condition EvoPathBench [Lin et al., 2026]Capability checkpoints over ordered streamsNo S 3 Gym [Shi et al., 2026]Strict held-out evaluationNo TEBThe same locked held-out idsYes; four no evolve passes Jiang et al. [2026] evaluate agents over task sequences rather than single episodes; Liu et al. [2026] studies skill feedback over rounds, and Shang et al. [2026] gates skill updates before commit. Table 2: Protocol mechanics contrast (binary coding).✓ means the property is explicitly supported in the paper’s main protocol; × means otherwise (including not stated). In “Carrier flex. in this paper (m/p/c/w)”, m/p/c/w = memory/prompt/code/weight. Abbreviations: SEA (SEAGym), Evo (EvoAgentBench), GDP (GDPevo), RSI (RSIBench-Data), PTB (PostTrainBench), TR (TRACE), MLE (OpenMLE), EPB (EvoPath- Bench), S3 (S 3 Gym), SEB (SE-Bench), PAST (PAST-Bench), Fin (FinEvo-Bench). PropertyTEB RSEA SEA Evo GDP RSI PTB TR MLE EPB S3 SEB PAST Fin Own traces✓×✓×✓ Multi-round✓×✓×✓×✓ Carrier flex. in this paper (m/p/c/w) × Explore/eval split✓×✓×✓×✓× Validation gate✓× Held-out lock✓×✓×✓×✓× Weight freeze✓×✓×✓ Weight slot✓× 3 Protocol Figure 1 is the measurement. Figure 2 is one round of it. Round 0 scores M 0 on held-out and freezes Gain’s baseline. Each later round collects a disjoint slice of evolution episodes, calls evolve() once, accepts the proposal iff validation success is at least the current agent’s (min delta=0), otherwise rolls back, then scores the accepted agent on the same held-out ids. Held-out success never enters evolve(). Let Succ(M) = #held-out pass/n be a rate on the locked held-out ids. Percentage points are 100 times a rate difference: Gain(M r ) = 100× Succ(M r )− Succ(M 0 ) ,(1) with this-run M 0 . In these equations M 0 is that round-0 score. A version label M 0 , M 1 , . . . names the accepted artifact, and no evolve keeps version M 0 while the score still moves. A proposal is accepted when validation success is at least the current agent’s (mindelta = 0); otherwise it is rolled back. Held-out ids do not enter evolve(). For actor a, pack e, and seed s, the noise band is the range of the four frozen 3 Stage 1 Global evaluation contract Stage 2 Per-round mechanics (r>=1) Stage 3 Seed-wise attribution rule 1 Freeze the pack held-out never enters evolve() 2 Score the packs apart 3 Round 0 Score M0. No evolve(). 1 own traces 2 evolve() once 3 accept / back 4 same held-out step 2 writes gepa instruction ace playbook expel notes weights frozen four no_evolve passes band = max − min above: count it inside: not evolution Before the band 1 this seed's band 2 round 0 < 10 p, else omit 802040 evolutionval.held-out codingofficemfg. each round r1 r2 r3 Figure 1: TEB global evaluation contract across rounds. Round 0 scores M 0 and does not update. Rounds 1–3 each accept or roll back one text update, then rescore the same held-out. Excess is read against that seed’s own four-pass band. no evolve held-out rates k ∈0, 1, 2, 3: B(a, e, s) = 100× max k Succ(M (0) a,e,s,k )− min k Succ(M (0) a,e,s,k ) .(2) Paired excess uses the same actor, pack, and seed: Excess = Gain method (a, e, s)− Gain no evolve (a, e, s).(3) An excess inside that seed’s own B(a, e, s) is not reported as evolution. A method whose round-0 held-out rate differs from the same-seed frozen round 0 by 10 p or more is not read as the same starting draw. Each seed is compared with its own B(a, e, s). Office DeepSeek’s widest seed band is 17.5 p, and seed 43’s own band is 7.5 p. A value inside the widest band can still sit on seed 43’s band. Manufacturing keeps its own bands. Regression is, among tasks passed on the previous held-out pass, the fraction that fail on the final pass. A rate above 0.20 is stored on the cell. The excess comparison keeps the cell either way. Coding is the primary scoreboard. The evolution-token cap is 500,000 per round, summed from per-step usage on evolution tasks only. Validation, held-out, and calls inside evolve() are not added. Crossing the cap skips evolve() for that round, marks over budget, and skips later rounds when the violation is recorded before the next round. no evolve returns the same agent. Its four held-out passes are repeated measurements of a frozen policy [Henderson et al., 2018, Madaan et al., 2024]. The noise band is the max-minus-min of those four scores. Paired excess is a method’s Gain minus the same model and seed’s no evolve Gain. Excess inside the band is not reported as evolution. Runner eligibility (main tableeligible) is a separate flag: a frozen no evolve cell can pass it while agentversion stays M 0 . Paper-min, unless labeled otherwise: R=3; at least three seeds [Bouthillier et al., 2021]; at least 40 evolution ids, 10 validation, 20 held-out; M 0 inside [0.10, 0.80] or the run is flagged; humanedits=0. 4 Identity: environment · budget class · trace origin · actorB0 is not ranked with B2 500,000 evolution-token cap skips evolve() and later rounds Round 0 Held-out idsScore M0Freeze Gain same ids on later rounds Rounds 1–R Disjoint self traces evolve() once Validation gate min_delta = 0 accept rollback New artifact Prior artifact Locked held-out accepted agent same ids next round held-out never enters evolve() Figure 2: Per-round accept/rollback mechanics (this loop is rounds r ≥ 1 in Figure 1). Round 0 freezes Gain on the locked held-out ids. Later rounds call evolve() once; accept and rollback both score that same held-out. Held-out never enters evolve(). B0 is not ranked against B2. The runner also flags a pack with no OOD slice. Coding, the primary scoreboard, is one family and has no OOD slice; that flag does not move the scoreboard. Each pack uses 80/20/40, so n=40 held-out (2.5 p per task). Default mix all, schedule accumulate chunks, curriculum none [Bengio et al., 2009]. Rank only inside (env,budgetclass,traceorigin,actor): B0 does not train weights and is not ranked with B2, a weight update. This paper reports B0 and B2 only; B1 is out of scope. Table 3 is the pack contract. Table 3: Three frozen packs. Each has 140 tasks, split 80/20/40, so held-out n=40. The observation omits gold files, expected SQL, and unit-test oracles. OfficeCodingManufacturing SourceProduct-usage workflows142 merged pull requests, rewritten as functions 12 real tables; 31 real questions, 109 generated Items10 familiesOne family, fixbug13 families Outside evolution12/40: accessdenied, do notpublish None on held-out12/40: real questions on 4 unseen table families Tools listfiles, readfile, writefile, publishartifact, grant access, submit listdir, readfile, writefile, runtests, submit;onefile, src/solution.py list tables, describetable, executesql, submit answer SuccessEvery terminal check passesHidden unit tests passScore≥ 80: SQL 40, result 40, an- swer 20 Step cap201212 4 Coding measurement Coding is the primary scoreboard. It uses the same 80/20/40 split, R=3, and B0 Standard Tool Actor. The episode cap is 12 steps. Held-out is one family, fixbug (n=40). A task is one file, src/solution.py. The tools are list dir, readfile, writefile, runtests, and submit. Success is the hidden unit tests; the gold patch is not in the observation. Every miss is the 12-step cap, so a gain here does not open a second task type. The three methods, gepallm, acellm, and expelllm, were fixed on an office validation screen before held-out was read (Appendix A.3). That screen is a separate smoke split: one round, six tasks, ten steps. Five methods passed the artifact check. The screen kept the three with the highest mean validation gain, then lower screen regression, then fewer selection tokens. Their mean screen gains were 0,−5.6, and−11.1 p, none positive (Table 4). Those gains do not predict the held-out excess 5 on this board. Each method reads that round’s own traces and injects the text in Table 5 into the same frozen model. Table 4: Screen ranking across three models at seed 42. Mean gain is in percentage points. Mean reg. is a rate. Valid is the artifact check. MethodValid Gain (p) Mean reg.Tokens gepallmtrue0.00.056 469231 ace llmtrue −5.60.111 513033 expelllmtrue −11.10.056 448950 dgmtrue −16.70.111 396406 clin llmtrue −16.70.222 411690 awmllmfalse0.00.000 447296 reasoningbankfalse −11.10.056 457843 Table 5: Text injected by the three scored methods. Each reads that round’s own traces. The kept text is the TEB mechanism. MethodInjected text gepallmOne revised instruction. The validation gate accepts it or rolls it back. acellmA grow-and-refine playbook from a generator, reflector, and curator. expelllm A deduplicated pool of insights from success and failure. DeepSeek noevolve keeps version M 0 with four-pass ranges B = 7.5, 5.0, and 12.5 p (Table 6, Figure 3). Frozen Gain and band for all three models are Table 7. 0123 round 60 80 held-out % no evolve 0123 round gepa llm 0123 round ace llm 0123 round expel llm 424344 Figure 3: DeepSeek coding held-out success by round. noevolve keeps version M 0 and the score still moves. Seeds are 42, 43, and 44. Figure 3 is why excess, not the raw curve, is the coding result. noevolve keeps version M 0 while each seed’s curve still moves, and its Gain is negative on all three seeds. A method panel can rise while part of that rise is the frozen score falling. Table 7 is the cross-model control. DeepSeek is the only model whose three frozen Gains are all neg- ative, so a rising method panel on that model includes the frozen score falling. GLM seeds 42 and 44 are also negative. GLM seed 44’s band is 15.0 p, so an excess of +15.0 equals that band. The same +15.0 on DeepSeek seed 42 exceeds its 7.5 p band. 6 Table 6: Coding, DeepSeek v4 Pro, noevolve only. Success is held-out percent. Version is M 0 at every round. Range is that seed’s band. Seedr0r1r2r3Gain Range 42 67.5 67.5 65.0 60.0 −7.57.5 43 62.5 65.0 62.5 60.0 −2.55.0 44 62.5 62.5 65.0 52.5 −10.012.5 Table 7: Coding noevolve by model. Frozen Gain is the four-pass Gain. Band is that seed’s four-pass range. DeepSeek pass scores are Table 6. ModelSeed Frozen Gain Band DeepSeek 42−7.57.5 DeepSeek 43−2.55.0 DeepSeek 44 −10.012.5 GLM42−2.512.5 GLM43+7.57.5 GLM44−7.515.0 MiniMax42+5.010.0 MiniMax430.02.5 MiniMax440.012.5 Table 8 is DeepSeek paired excess. Seven of the nine cells are positive and larger than that seed’s own B. All seven have final regression at or below 0.20: gepallm seeds 42 and 43 (+15.0 against bands 7.5 and 5.0), ace llm seeds 42 and 43 (+20.0 and +10.0), and expelllm seeds 42, 43, and 44 (+15.0, +7.5, +20.0). Round-0 success on these seven cells is within 5 p of the same-seed frozen round 0. gepallm seed 44 moves the other way (−12.5, equal to its 12.5 band) and its final regression is 0.48. The cell stays in the excess comparison. One seed of the same method moves down by the width of its band. The seven cells above their bands leave version M 0 . ace llm seed 44, the positive cell inside its band, keeps version M 0 . Leaving version M 0 is not the excess: gepallm seed 44 also finishes at version M 2 , and its excess is −12.5. Table 8: Coding-v1, DeepSeek v4 Pro. Excess is method Gain minus the same-seed no evolve Gain, in percentage points. Band is that seed’s four-pass range. Reg. is the final regression rate. None of these nine cells is over budget. Method SeedGain Frozen Excess BandReg. Against that band gepa42+7.5 −7.5 +15.07.5 0.167 outside gepa43 +12.5 −2.5 +15.05.0 0.069 outside gepa44 −22.5 −10.0 −12.512.5 0.481 equal, down ace42 +12.5 −7.5 +20.07.5 0.065 outside ace43+7.5 −2.5 +10.05.0 0.071 outside ace440.0 −10.0 +10.012.5 0.130 inside expel42+7.5 −7.5 +15.07.5 0.062 outside expel43+5.0 −2.5 +7.55.0 0.097 outside expel44 +10.0 −10.0 +20.012.5 0.103 outside GLM coding is the same comparison on a wider frozen range (Table 9). gepallm excesses are +7.5, 7 gepa 42 4344 ace 42 4344 expel 42 4344 −10 0 10 20 excess (p) Figure 4: DeepSeek coding paired excess. Black ticks are that seed’s own band. Dark bars lie strictly above it. The light positive bar is acellm seed 44, inside its 12.5 p band. gepallm seed 44 equals its band on the way down. 0, and +15.0 against bands 12.5, 7.5, and 15.0: the first two sit inside, and seed 44 equals its band. ace llm seed 42 is−12.5 on a 12.5 band and is overbudget; seeds 43 and 44 are +5.0 (inside 7.5) and +15.0 (equal to 15.0), and seed 43 is also over budget. expelllm seeds 43 and 44 are +10.0 and +22.5 against bands 7.5 and 15.0, with final regression 0.03 and 0.13. Seed 44’s round 0 is 60.0 against the frozen round 0 of 67.5, a 7.5 p gap, below the 10 p cutoff, so that excess stays in the comparison. expelllm seed 42 is not in this pairing: its recorded M 0 is 5%, while noevolve seed 42 opens at 72.5%. The round-0 gap is 67.5 p. The cell is outside the paper-min M 0 window [0.10, 0.80]. MiniMax coding is finished for all three methods on seeds 42, 43, and 44. On seed 42 the excesses are −7.5,−5.0, and +7.5 for gepa llm, acellm, and expelllm, all inside the 10.0 band; expelllm has final regression 0.22. Seed 43 gepa llm and acellm open at 27.5% against a frozen round 0 of 37.5%, so their excesses of +45.0 and +17.5 are not the same starting draw. expel llm seed 43 equals the 2.5 band and has regression 0.36. On seed 44, gepa llm Gain is +10.0 and its excess is +10.0, inside the 12.5 band, with regression 0.14; acellm excess is 0. expelllm excess is +15.0, above that 12.5 band. Its round 0 is 37.5 against the frozen round 0 of 40.0, and its final regression is 0.30. Table 9: Coding status for GLM and MiniMax. Excess and band are defined as in Table 8. Unpaired means the round-0 gap is at least 10 p. Reg. is the final regression rate. Cap marks over budget. A regression above 0.20 is shown and does not remove the cell. ModelMethod Seed Excess BandReg. Cap Against that band GLMgepa42 +7.5 12.5 0.207inside GLMgepa43 0.07.5 0.069inside GLMgepa44 +15.0 15.0 0.094equal GLMace42 −12.5 12.5 0.130 yes equal, down GLMace43 +5.07.5 0.065 yes inside GLMace44 +15.0 15.0 0.091equal GLMexpel42 +70.0 12.5 0.000unpaired GLMexpel43 +10.07.5 0.031outside GLMexpel44 +22.5 15.0 0.125outside MiniMax gepa42 −7.5 10.0 0.353inside MiniMax gepa43 +45.02.5 0.000unpaired MiniMax gepa44 +10.0 12.5 0.136inside MiniMax ace42 −5.0 10.0 0.214inside MiniMax ace43 +17.52.5 0.158unpaired MiniMax ace44 0.0 12.5 0.188inside MiniMax expel42 +7.5 10.0 0.222inside MiniMax expel43 +2.52.5 0.364equal MiniMax expel44 +15.0 12.5 0.304outside 8 0.02.55.07.510.012.515.017.5 band (p) −20 −10 0 10 20 excess (p) DeepSeek GLM MiniMax Figure 5: Paired cells. The wedge is inside the band. A point above the upper line clears that seed’s band. Unpaired cells are omitted. 0 25 50 GLM gepa 42 4344 ace 42 4344 expel 42 4344 0 20 40 MiniMax Figure 6: GLM and MiniMax coding paired excess. Black ticks are that seed’s own band. Dark bars lie strictly above it: GLM expel llm seeds 43 and 44, and MiniMax expelllm seed 44. Hatched bars are not the same starting draw: GLM expelllm seed 42, and MiniMax gepallm and acellm seed 43. Figure 6 is why the seven-of-nine pattern stays a DeepSeek result. Three dark bars clear their own bands, not seven of nine. A hatched bar is a different starting draw and is not one of those seven. Figure 5 is the same paired cells. DeepSeek’s seven cells lie above the upper boundary. GLM’s +15 at seed 44 lies on the boundary of a 15 p band. DeepSeek’s +15 at seed 42 lies above the boundary of a 7.5 p band. 5 Manufacturing measurement Manufacturing asks whether the coding seven-of-nine pattern appears for the same three methods. It does not. A pass lists tables, describes one, executes SQL, and submits an answer. Success is a deterministic score of at least 80: SQL 40, result 40, answer 20. The expected SQL is not in the observation. The pack uses the same 80/20/40 split and the 12-step cap. DeepSeek no evolve ranges are 7.5, 5.0, and 2.5 p. GLM ranges are 2.5, 5.0, and 2.5. MiniMax ranges are 12.5, 10.0, and 12.5. One task is still 2.5 p. 9 No finished DeepSeek manufacturing cell has a positive excess larger than its seed band (Table 10). ace llm excess is−2.5 on every seed; Gain on those three cells is 0. expelllm seed 44 is +2.5, equal to the 2.5 band. gepallm seeds 42 and 43 have excesses−10.0 and−12.5. Seed 42 gepallm and seed 42 expelllm both open at 45.0 against a frozen round 0 of 35.0, so those two excesses are not the same starting draw. gepallm seed 43 opens at 47.5 against 40.0, a 7.5 p gap, and stays in the comparison. ace llm seed 44 and expelllm seed 43 have final regression 0.21, above 0.20. GLM’s largest positive excess is +5.0 (acellm seed 43), equal to that seed’s 5.0 band. expelllm seed 42 is −7.5 against a 2.5 band. The other GLM excesses are 0 or +2.5 on bands of 2.5 or 5.0. Table 10: Manufacturing-v1, DeepSeek v4 Pro. Excess and band are defined as in Table 8. Method SeedGain Frozen Excess Band Against that band gepa42 −7.5 +2.5 −10.07.5 outside, negative; r0 gap 10 p gepa43 −10.0 +2.5 −12.55.0 outside, negative; r0 gap 7.5 p gepa440.0 +2.5 −2.52.5 equal ace420.0 +2.5 −2.57.5 inside ace430.0 +2.5 −2.55.0 inside ace440.0 +2.5 −2.52.5 equal; reg. 0.21 expel42 −7.5 +2.5 −10.07.5 outside, negative; r0 gap 10 p expel43+2.5 +2.50.05.0 inside; reg. 0.21 expel44+5.0 +2.5 +2.52.5 equal MiniMax manufacturing is finished for all three methods on seeds 42, 43, and 44. On seed 42, acellm excess is−15.0 and expelllm excess is−7.5 against a 12.5 band, so only acellm falls outside, and it falls below. gepa llm opens at 27.5 against a frozen round 0 of 17.5, a 10 p gap, so its −7.5 excess is not the same starting draw. Seed 43 gepa llm excess is +5.0 and expelllm excess is −5.0, both inside the 10.0 band. ace llm seed 43 excess is +20.0 against that band, but round 0 is 12.5 against a frozen round 0 of 22.5. On seed 44, acellm excess is +2.5 and expelllm excess is−7.5, both inside the 12.5 band; expel llm keeps version M 0 . gepallm seed 44 Gain is +25.0 and its excess is +20.0 against that band, but round 0 is 12.5 against a frozen round 0 of 22.5. Final regression on all nine MiniMax method cells is above 0.20. 6 Office measurement Office-v1 uses the same 80/20/40 split and the B0 Standard Tool Actor. The episode cap is 20 steps. The tools are list files, readfile, writefile, publishartifact, grantaccess, and submit. Success is every terminal check passing. A failed submit returns the failed check names and leaves the episode open. Gold files are not in the observation. It is scored on its own band and is not aver- aged with coding. Actor, prompt, and held-out ids stay fixed, and the request temperature is 0. The hosted endpoint is still not deterministic, so repeated scoring of the same version-M 0 policy is a band [Atil et al., 2024]. DeepSeek v4 Pro no evolve keeps version M 0 . The three seed ranges are 17.5, 7.5, and 17.5 p, so the widest frozen swing is 17.5 p; one task is 2.5 points. GLM seed 42 spans 17.5 p. MiniMax seed 42 spans 5.0 p. GLM and MiniMax are one seed on this pack. No DeepSeek office cell has a positive excess larger than its own seed band (Table 11). Mean excess is −2.5 p (gepa llm), −3.3 p (acellm), and +4.2 p (expelllm), all inside the widest 17.5 p range. Seed 43’s own band is 7.5 p: gepallm is−7.5 and acellm is +7.5, so both sit on that band rather than beyond it. incidenttrack, postmortem, and onboarding stay at zero solved tasks on every DeepSeek pass. The solved-count changes that remain are in families the frozen policy already solves on some pass. 10 Table 11: Office-v1, DeepSeek v4 Pro. Excess and band are defined as in Table 8. Reg. is the final regression rate. Method SeedGain Frozen Excess BandReg. Against that band gepa42+7.5 +10.0 −2.517.5 0.250 inside gepa430.0 +7.5 −7.57.5 0.375 equal gepa44 +17.5 +15.0 +2.517.5 0.143 inside ace42+5.0 +10.0 −5.017.5 0.267 inside ace43 +15.0 +7.5 +7.57.5 0.077 equal ace44+2.5 +15.0 −12.517.5 0.385 inside expel42 +17.5 +10.0 +7.517.5 0.154 inside expel43 +10.0 +7.5 +2.57.5 0.000 inside expel44 +17.5 +15.0 +2.517.5 0.091 inside GLM excesses sit inside 17.5 p, and the three adapter cells are overbudget. MiniMax gepallm excess is +2.5 p, inside the 5.0 p band. acellm excess is−7.5 p and stopped after round 2. expelllm excess is +7.5 p, 2.5 p past that 5.0 p band, on one seed, at final version M 1 . That is not a confirmed evolution result. 7 Limitations Coding held-out is one family, fix bug (n=40, 2.5 p per task). The seven cells above their own bands do not include gepallm seed 44, which moves down by 12.5 p with final regression 0.48. GLM and MiniMax do not reproduce that seven-of-nine pattern. GLM expelllm seeds 43 and 44 still lie above their own bands; seed 42 is unpaired (M 0 5% against a frozen round 0 of 72.5%). MiniMax expelllm seed 44 also lies above its 12.5 band, with final regression 0.30. Frozen coding seeds 42 and 44 have regression 0.231 and 0.308 with no policy change. Office-v1 held-out has n=40. Effects under the 17.5 p DeepSeek band are unresolved by this de- sign [Dror et al., 2018, Card et al., 2020], and the 0.20 regression cutoff already raises the runner’s flag on frozen seed 43 (0.231, M 0 ). Validation has n=20 (5 p per task). On office-v1, MiniMax and GLM are one seed. GLM adapters and MiniMax ace llm crossed the evolution-token cap. awmllm, reasoningbank, clin llm, and dgm were screened and are not on the office scoreboard. Closed hosted models were not weight-updated; TRACE and a B2 open-weight card [Hu et al., 2022] are a different budget class. Archived ticket-gym, inbox, and NL2SQL cards use another tool set and n=20 held-out and are not office results (Appendix A.18). Manufacturing is not a coding ranking (Section 5). Office uses 20 steps per task; coding and manu- facturing use 12. B0 token costs outside the evolution-token cap are estimates. MiniMax manufacturing is complete on three seeds. The two MiniMax excesses outside their seed bands (ace llm seed 43 and gepallm seed 44) are not the same starting draw, and all nine MiniMax method cells have final regression above 0.20. 8 Conclusion TEB’s main result is attribution, not a single global winner. On coding, endpoint Gain includes frozen rescoring variance: DeepSeek no evolve keeps version M 0 while the three seeds span 7.5, 5.0, and 12.5 p (Section 4). After seed-matched paired excess, seven of nine DeepSeek text-update cells lie above their own bands, but this pattern does not reproduce on GLM and MiniMax. Across all three models, no method 11 stays above its own band on every seed (Table 9). This separation between observed Gain and attributable excess is the core contribution: TEB turns self- evolution claims into a falsifiable protocol with explicit controls, rather than a single endpoint delta. The same contract keeps office and manufacturing as independent control packs on their own bands. Office DeepSeek does not exceed its own bands (widest 17.5 p; Section 6), and manufacturing has no paired positive excess beyond its seed band. AI use statement Generative AI coding assistants participated in the early execution of the implementation, namely the TEB runner and the method adapters. They also participated in polishing some prose, in retrieval, and in drafting some sections. The experimental design was set by the authors. We have not used generative AI to generate synthetic evaluation packs, formulate mathematical claims, write proofs, or invent held-out scores. All Success %, Gain, family counts, and gate decisions are read from stored run artifacts. The authors reviewed that implementation and take responsibility for the final content of this work. Ethics statement Manufacturing-v1 derives from internal business tables provided by a manufacturing partner. The bench- mark replaces counterparty, company, department, cost-center, and document identifiers with pseudonyms and withholds the private mapping. The tables contain no personal names, phone numbers, e-mail addresses, or identity numbers. Office and coding tasks are rewritten from product workflows and pull requests; contact details in office files use reserved example domains. No human subjects were involved. Reproducibility statement Section 3 defines every scored quantity. Appendix A.2 lists what each adapter keeps, Appendix A.3 the development screen, Appendix A.9 the eligibility flags, and Appendix A.17 one task from each suite. The release contains the runner and table contract (teb/, teb/table_contract.py), the three packs with source metadata removed (tasks/office, tasks/coding, tasks/manufacturing), the configs of the reported runs, and the sanitized per-run summaries used by scripts/plot_office_v1.py. Appendix A.1 provides the dataset card and licenses (data C BY 4.0, code MIT). Hosted endpoints require the reader’s own API key. References Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexan- dros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representa- tions, 2026. Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. Non- determinism of “deterministic” LLM settings. arXiv preprint arXiv:2408.04667, 2024. 12 Yoshua Bengio, J ́ er ˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Interna- tional Conference on Machine Learning, 2009. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems, volume 3, 2021. Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020. DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. Bo Deng, Kang Zhou, Lifan Guo, Chongyang Tao, Xuanren Chen, Chenggang Xie, Renzhao Liang, Feng Chen, and Chi Zhang. FinEvo-Bench: A longitudinal benchmark for self-evolving agents in professional financial workflows. arXiv preprint arXiv:2608.06144, 2026. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018. Aparna Elangovan, Jiayuan He, and Karin Verspoor. Memorization vs. generalization: Quantifying data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021. Xingze Gao, Chuanrui Hu, Hongda Chen, Pengfei Yao, Zhao Wang, Yi Bai, Zhengwei Wu, Yunyun Han, Xiaofeng Cong, Jie Gui, Yafeng Deng, and Teng Li. EvoAgentBench: Benchmarking agent self-evolution via ability transfer. arXiv preprint arXiv:2607.05202, 2026. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daum ́ e I, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2018. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Tengfei Wang, Shisong Chen, Jinghao Zhang, Tianjun Pan, Weijia Li, Jiaqing Liang, and Yanghua Xiao. SEA-Eval: A benchmark for evaluating self-evolving agents beyond episodic assessment. arXiv preprint arXiv:2604.08988, 2026. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. 13 Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, and Azalia Mirhoseini. TRACE: Capability-targeted agentic training. In International Conference on Machine Learning, 2026. Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, 2021. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? a BIRD bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems, 2023. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transac- tions on Machine Learning Research, 2023. Hongqiang Lin, Chao Liu, Xiaofan Bai, Xuan Jin, Yuhong Li, Nenggan Zheng, and Xipeng Cao. Beyond endpoint performance: Process-level evaluation of self-evolving agents.arXiv preprint arXiv:2609.24663, 2026. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024. Yuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, and Yangqiu Song. Rethinking self-evolving agent skills: Feedback dynamics over multiple rounds. arXiv preprint arXiv:2608.02636, 2026. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallman, Dongchan Gao, Christopher Re, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, 2023. Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. arXiv preprint arXiv:2406.10229, 2024. Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 157–165, 2022. Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization. In COLM, 2024. Fanqing Meng, Lingxiao Du, Qiguang Chen, Ziqi Zhao, Haocheng Lu, Mengkang Hu, and Michael Qizhe Shieh. RSIBench-Data: Benchmarking data-centric research for recursive self-improvement. arXiv preprint arXiv:2607.25886, 2026. Gr ́ egoire Mialon, Cl ́ ementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A bench- mark for general AI assistants. In International Conference on Learning Representations, 2024. MiniMax. The MiniMax-M2 series: Mini activations unleashing max real-world intelligence. arXiv preprint arXiv:2605.26494, 2026. 14 Michael Nguyen, Quoc Nguyen, and Paul Vuong. Recursive self-evolving agents via held-out selection. arXiv preprint arXiv:2606.28374, 2026. Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. ReasoningBank: Scaling agent self-evolving with reasoning memory. In International Conference on Learning Representations, 2026. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonza- lez. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. PostTrainBench: Can LLM agents automate LLM post-training? In Inter- national Conference on Machine Learning, 2026. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, 2020. St ́ ephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 627–635, 2011. Linfang Shang, Ming Xu, Yiding Sun, Tianle Xia, Lingxiang Hu, Lan Xu, and Ning Zheng. When self-evolution backfires: Pre-commit gating against skill contamination in LLM agents. arXiv preprint arXiv:2608.05810, 2026. Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, et al. S3gym: Can llms turn self-testing and self-judging into self-improvement? arXiv preprint arXiv:2608.31100, 2026. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Lan- guage agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆ ot ́ e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In Inter- national Conference on Learning Representations, 2021. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550:354–359, 2017. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Aniket Mandlekar, Chaowei Xiao, Yuke Zhu, Song An, et al. Voyager: An open-ended embodied agent with large language models. In Advances in Neural Information Processing Systems, 2023. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. In Proceed- ings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 63897–63911, 2025. 15 Shuhan Xue, Zixin Ding, Yichen Shen, Yinjie Wang, Zhenfei Yin, Yingcheng Wu, Yuxin Chen, Mengdi Wang, and Ling Yang. PAST-Bench: Benchmarking the foundations of recursive self-improvement in personal agents. arXiv preprint arXiv:2608.04003, 2026. Junlin Yang, Che Jiang, and Kaiyan Zhang. Frontis-MA1: Training an AI4AI model towards recursive self-improvement in machine learning engineering. arXiv preprint arXiv:2607.28568, 2026. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- Act: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. Jiarui Yuan, Tailin Jin, Weize Chen, and Zeyuan Liu. SE-Bench: Benchmarking self-evolution with knowl- edge internalization. arXiv preprint arXiv:2602.04811, 2026. Yunpeng Zhai, Shuchang Tao, Cheng Chen, Anni Zou, Ziqian Chen, Qingxu Fu, Shinji Mai, Li Yu, Jiaji Deng, Zouying Cao, Zhaoyang Liu, Bolin Ding, and Jingren Zhou. AgentEvolver: Towards efficient self-evolving agent system. arXiv preprint arXiv:2511.10395, 2025. Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin G ̈ odel machine: Open-ended evolution of self-improving agents. In International Conference on Learning Representations, 2026a. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agen- tic context engineering: Evolving contexts for self-improving language models. In International Confer- ence on Learning Representations, 2026b. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. Congjie Zheng, Chuanyi Xue, Bin Liang, Jun Yang, and Changshui Zhang. SEAGym: An evaluation environment for self-evolving LLM agents. arXiv preprint arXiv:2606.17546, 2026. Junhao Zheng, Xidi Cai, Qiuke Li, Duzhen Zhang, Zhong-Zhi Li, Yingying Zhang, Le Song, and Qianli Ma. LifelongAgentBench: Evaluating LLM agents as lifelong learners. arXiv preprint arXiv:2505.11942, 2025. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM- as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023. Ruiqi Zhong, Tao Yu, and Dan Klein. Semantic evaluation for text-to-SQL with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 16 Leijun Zhou, Zhihao Liu, Xiang Qu, Chenxu Liu, Yifei Liu, Yanke Yu, Jingzhe Xu, Xuejun Wu, Buyue Qian, Xi Chen, Yaowei Zheng, and Junhao Hu. GDPevo: Evaluating agent self-evolution on real business tasks. arXiv preprint arXiv:2608.03764, 2026. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024. Appendix contents A.1Dataset card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.2What each office adapter keeps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.3Development screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.4Office validation gates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.5Per-round held-out record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.6DeepSeek family counts on every pass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 A.7Seed-42 family counts outside DeepSeek . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 A.8Failure reasons outside the DeepSeek office cells . . . . . . . . . . . . . . . . . . . . . . . . 24 A.9Eligibility flags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 A.10Office grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 A.10.1Frozen band . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 A.10.2Paired excess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 A.10.3Same agent, different score . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 A.10.4Full grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 A.10.5Seed-42 transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 A.10.6Where the 40 tasks move . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 A.10.7Failure reasons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 A.11Office artifact versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 A.12Manufacturing frozen repeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 A.13Manufacturing family counts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 A.14Coding frozen repeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 A.15Extra office frozen passes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 A.16Coding grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 A.17Worked examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 A.17.1One task from each current suite . . . . . . . . . . . . . . . . . . . . . . . . . 32 A.17.2What evolve() actually returns . . . . . . . . . . . . . . . . . . . . . . . . 33 A.17.3One held-out task, same M 0 , two scores . . . . . . . . . . . . . . . . . . . . . 34 A.18Archived ticket-gym mock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 A.18.1Why the protocol constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 A.18.2Success, Gain, and actors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 A.18.3Setup of the archived mock . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 A.18.4Summary boards and expansion families . . . . . . . . . . . . . . . . . . . . . 36 A.18.5Round curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 A.18.6Mechanism: what heuristic failures teach . . . . . . . . . . . . . . . . . . . . . 37 A.18.7Why R=3 is enough on ticket-gym . . . . . . . . . . . . . . . . . . . . . . . . 38 A.18.8Inbox mock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.18.9Seed, wrapper, inbox, and hygiene tables . . . . . . . . . . . . . . . . . . . . . 38 17 A.18.10Weight-track provenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.18.11Ticket-gym B0 ablations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.18.12Ticket-gym gate decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.18.13Ticket-gym native pairing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 A.18.14NL2SQL teacher contrast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 A.18.15Ticket-gym expansion pack . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 A.18.16Cross-environment summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 A.1 Dataset card Following datasheet practice [Gebru et al., 2021], TEB ships three frozen packs with the same 80/20/40 split and fixed held-out membership (Table 12). Each pack is adapted from a real source: product-usage workflows for office, merged pull requests of one production repository for coding, and one manufacturer’s internal business tables for manufacturing. Manufacturing contains 31 real questions; the other 109 are generated over the same real tables with recomputed gold, so “real-derived” refers to the tables and those 31 questions, not every item. The generated questions come from deterministic templates whose gold is computed by executing SQL on the released tables; a candidate is rejected if it repeats an original question, gold SQL, or answer value, names a real entity, or touches a held-out-only table. Coding tasks restate each pull request’s behavior as a rewritten Python function; no repository code is copied. Two held-out sets contain items outside the evolution distribution: office adds two held-out-only families, and manufacturing places its 12 held-out real questions on four table families that never appear in evolution or validation. A frozen DeepSeek v4 Pro policy leaves headroom on every pack, and five of the 13 manufacturing families are unsolved on every pass (Appendix A.13). Gold files, expected SQL, and unit-test oracles never enter the observation. The packs are not claimed to measure production performance or open-ended NL2SQL generalization [Yu et al., 2018, Li et al., 2023]; manufacturing scoring checks execution rather than string match [Zhong et al., 2020]. Verifiers are deterministic. The reference solution of every one of the 420 items (gold files, gold patch, or gold SQL) passes its verifier, and the released tests check this. We did not conduct a separate annotator audit. Benchmark data are licensed under C BY 4.0 and code under MIT. Table 12: TEB pack card. Frozen range is held-out Success over the four passes of DeepSeek v4 Pro no evolve, seeds 42–44. Language counts instructions containing Chinese. office-v1coding-v1manufacturing-v1 SourceProduct-usage workflows in a closed catalog workspace 142 merged pull requests of one production repository 12 pseudonymized real tables (32,290 rows) and 31 real questions Items140; 10 families140; fixbug140; 13 families; 31 real, 109 generated Held-out OOD12/40: accessdenied, donotpublish none12/40: real questions on 4 unseen table families Language33/140 Chinese, rest EnglishChineseChinese ObservationCatalog files and six toolsOne file, src/solution.pyFour tools: list tables, describe a table, execute SQL, submit an answer SuccessEvery terminal check passesClosed-workspace unit tests passScore≥ 80 of 100: SQL 40, result 40, answer 20; a partial SQL or result check caps the score below 80 Step cap201212 Frozen range22.5–45.0%52.5–67.5%35.0–45.0% 18 A.2 What each office adapter keeps All seven adapters read the current round’s own traces, update once before the validation gate, and inject bounded text into the same frozen model. None updates weights. noevolve returns the same agent and still runs the round schedule, so the baseline is round-matched. The slots are method-inspired by Ex- peL [Zhao et al., 2024], workflow memory [Wang et al., 2025], ACE [Zhang et al., 2026b], GEPA [Agrawal et al., 2026], ReasoningBank [Ouyang et al., 2026], CLIN [Majumder et al., 2024], and DGM [Zhang et al., 2026a]. Upstream benchmark numbers are not reused. Table 13: Method-inspired slots on the Standard Tool Actor. The kept column is the TEB mechanism. The omitted column is the upstream system this paper does not claim to reproduce. SlotKeptOmitted expelllmInsights from success and failure; deduplicated poolUpstream harness and retrieval awm llmWorkflows from successful traces onlyUpstream induction stack ace llmGenerator, reflector, curator; grow-and-refine playbookUpstream application runtime gepallmOne revised instruction; the TEB gate accepts or rolls it back Population search and Pareto compile reasoning bankTitle, when, and content memories; lexical top-kUpstream agent and test-time search clinllmCondition, action, and effect rulesDomain gold memory dgmArchive, parent score, text mutationRepository self-modification dgmnativecode is a separate board (Appendix A.18.13). AgentEvolver [Zhai et al., 2025], the SEED context probe, and TRACE [Kang et al., 2026] are not part of Table 28. TRACE remains a B2 card: contrast traces, synthesize environments, train LoRA experts [Hu et al., 2022]. A closed hosted model cannot take that update. A.3 Development screen The screen file records heldoutread=false. Gain in this section is validation success after one up- date minus the model-matched no evolve validation success on the same screen pass. Screen regression is max(0, before− after) on that validation pass. It is not the held-out regression rate in Table 11. In- valid rows failed the artifact check: the evolve artifact had to carry a hash, a non-empty context state, and implementation fidelity=method-inspired. The three valid methods with the highest mean gain, then lower screen regression, then fewer selection tokens, were extended to seeds 43 and 44. Those means are 0, −0.056, and −0.111 for gepa llm, acellm, and expelllm: 0, −5.6, and −11.1 p. None is a positive validation gain. The screen budget is one round, 10 steps per task, and 6 tasks per round, with packrole=smoke on a separate split. Formal office-v1 is R=3, 20 steps, and 80/20/40. Selection used the shorter episode. This screen is an office validation screen. The three methods it fixed are the ones scored on the coding primary scoreboard; that board was not a separate coding pre-registration (Section 4). The ranking is Table 4. Table 14: Screen cells. Validation columns are success rates. Gain is after minus the model-matched no evolve rate. ModelMethodBeforeAfterGainReg.Artifact deepseek-v4-proexpelllm0.1670.167-0.3330.000true deepseek-v4-proawm llm0.3330.333-0.1670.000true 19 ModelMethodBeforeAfterGainReg.Artifact deepseek-v4-proacellm0.1670.333-0.1670.000true deepseek-v4-progepallm0.5000.333-0.1670.167true deepseek-v4-proreasoningbank0.3330.167-0.3330.167false deepseek-v4-proclinllm0.5000.333-0.1670.167true deepseek-v4-prodgm0.1670.167-0.3330.000true minimax-m2-5expel llm0.1670.167+0.0000.000true minimax-m2-5awmllm0.0000.167+0.0000.000false minimax-m2-5acellm0.3330.167+0.0000.167true minimax-m2-5gepa llm0.0000.167+0.0000.000true minimax-m2-5reasoningbank0.0000.000-0.1670.000true minimax-m2-5clinllm0.0000.167+0.0000.000true minimax-m2-5dgm0.1670.000-0.1670.167true glm-5-3expel llm0.5000.333+0.0000.167true glm-5-3awmllm0.5000.500+0.1670.000false glm-5-3acellm0.5000.333+0.0000.167true glm-5-3gepallm0.5000.500+0.1670.000true glm-5-3reasoning bank0.3330.500+0.1670.000true glm-5-3clinllm0.5000.000-0.3330.500true glm-5-3dgm0.5000.333+0.0000.167true A.4 Office validation gates The gate compares validation success of the current agent with the proposal. Both numbers are one pass on n=20 validation tasks. A drop rolls the proposal back. Table 15 is every non-zero round in the 20- cell matrix. Round 0 has no proposal. One validation task is 5 p. Several accepts in Table 15 are 0 p changes, which min delta=0 keeps. On noevolve the artifact stays M 0 whether the row says accept or rollback: the two validation numbers are two passes of one frozen policy. GLM noevolve validation moves 45→40, 60→45, and 55→30. A later held-out move on that same version is a rescore, as in Table 27. A.5 Per-round held-out record Table 16 is every held-out pass in the 20-cell matrix. Regression is against the previous pass; round 0 is 0. Tokens are total tokens summed on that pass’s held-out traces, not the 500,000 evolution-token cap. Validation is the stored val success. Main-text regression rates are these values rounded to two decimals. Table 16: Office-v1 held-out record. Success and validation are percent. Regression is a rate. ModelMethodSeedRoundSucc.Val.Reg.Tokens DSace42032.5—0.000592020 DSace42130.040.00.231690436 DSace42237.550.00.083667912 DSace42337.535.00.267808663 20 ModelMethodSeedRoundSucc.Val.Reg.Tokens DSace43032.5—0.000576054 DSace43125.030.00.308634816 DSace43232.525.00.100585464 DSace43347.535.00.077958826 DSace44030.0—0.000579534 DSace44127.530.00.167594075 DSace44232.535.00.364679860 DSace44332.535.00.385967211 DSexpel42025.0—0.000576918 DSexpel42132.520.00.200704342 DSexpel42232.535.00.308786634 DSexpel42342.535.00.154776706 DSexpel43032.5—0.000562761 DSexpel43130.035.00.077671670 DSexpel43220.020.00.417773513 DSexpel43342.525.00.000673400 DSexpel44032.5—0.000568396 DSexpel44127.525.00.231680503 DSexpel44227.515.00.364718343 DSexpel44350.030.00.091694999 DSgepa42035.0—0.000580628 DSgepa42125.025.00.357590063 DSgepa42240.030.00.200685235 DSgepa42342.535.00.250635933 DSgepa43030.0—0.000571489 DSgepa43130.030.00.167555353 DSgepa43240.035.00.167692592 DSgepa43330.020.00.375736969 DSgepa44025.0—0.000608887 DSgepa44135.030.00.100556708 DSgepa44235.020.00.071578150 DSgepa44342.525.00.143609770 DSno evolve42035.0—0.000563887 DSnoevolve42132.520.00.286577280 DSnoevolve42227.520.00.308619302 DSno evolve42345.025.00.091511676 DSnoevolve43027.5—0.000611006 DSnoevolve43135.030.00.182575530 DSnoevolve43232.530.00.143598822 DSno evolve43335.020.00.231564660 DSnoevolve44025.0—0.000612008 DSnoevolve44130.030.00.200597854 DSno evolve44222.525.00.333616784 DSno evolve44340.030.00.000534010 GLMace42047.5—0.000457328 GLMace42147.550.00.105431379 21 ModelMethodSeedRoundSucc.Val.Reg.Tokens GLMace42235.045.00.316695475 GLMace42337.5—0.071689266 GLMexpel42037.5—0.000466603 GLMexpel42150.050.00.133415846 GLMexpel42235.060.00.400677535 GLMexpel42337.5—0.071698951 GLMgepa42047.5—0.000433726 GLMgepa42145.050.00.211582084 GLMgepa42237.550.00.4441142477 GLMgepa42332.5—0.2001229120 GLMno evolve42042.5—0.000443087 GLMnoevolve42135.045.00.235464064 GLMno evolve42252.560.00.071441988 GLMnoevolve42335.055.00.381659535 MMace42032.5—0.000510135 MMace42120.015.00.538775021 MMace42227.5—0.000765450 MMexpel42022.5—0.000523694 MMexpel42122.510.00.333540754 MMexpel42220.015.00.222533679 MMexpel42332.55.00.125770089 MMgepa42017.5—0.000524370 MMgepa42120.05.00.286565039 MMgepa42227.55.00.375537457 MMgepa42322.510.00.273533844 MMno evolve42020.0—0.000527353 MMnoevolve42125.010.00.250523358 MMno evolve42220.010.00.300525339 MMnoevolve42322.55.00.250533712 A.6 DeepSeek family counts on every pass Counts are solved held-out tasks. Column order is catalog discover (n=8), briefreport (5), checklist (5), csv audit (4), accessdenied (6), donotpublish (6). incidenttrack, postmortem, and onboarding are omitted: they are 0 on every DeepSeek pass. A.7 Seed-42 family counts outside DeepSeek DeepSeek family counts are Table 30 and Appendix A.6. The cells below are the seed-42 transfer. Entries are solved counts at round 0 and at the last recorded round. 22 Table 21: GLM 5.3 and MiniMax M2.5 solved counts, round 0 to last round. CellFamilyRound 0Last GLM noevolveaccessdenied2/63/6 GLM noevolvebriefreport3/51/5 GLM no evolvecatalogdiscover4/85/8 GLM noevolvechecklist2/50/5 GLM noevolvecsvaudit1/41/4 GLM noevolvedonotpublish5/64/6 GLM gepaaccess denied2/62/6 GLM gepabriefreport3/52/5 GLM gepacatalogdiscover6/82/8 GLM gepachecklist3/52/5 GLM gepacsvaudit1/41/4 GLM gepadonotpublish4/64/6 GLM aceaccess denied2/61/6 GLM acebriefreport3/52/5 GLM acecatalog discover6/86/8 GLM acechecklist2/51/5 GLM acecsvaudit1/41/4 GLM acedonotpublish5/64/6 GLM expelaccessdenied1/62/6 GLM expelbrief report3/52/5 GLM expelcatalogdiscover3/86/8 GLM expelchecklist1/50/5 GLM expelcsvaudit2/41/4 GLM expeldo notpublish5/64/6 M noevolveaccessdenied1/61/6 M no evolvebriefreport0/51/5 M noevolvecatalogdiscover5/84/8 M noevolvechecklist1/50/5 M noevolvedonotpublish1/63/6 M gepaaccess denied1/61/6 M gepabrief report1/51/5 M gepacatalogdiscover2/83/8 M gepachecklist2/51/5 M gepado notpublish1/63/6 M aceaccessdenied1/61/6 M acebriefreport1/51/5 M acecatalog discover5/84/8 M acechecklist2/51/5 M acecsvaudit0/41/4 M acedo notpublish4/63/6 M expelaccessdenied1/63/6 M expelbriefreport1/51/5 M expelcatalog discover4/84/8 23 CellFamilyRound 0Last M expelchecklist1/52/5 M expelcsvaudit0/41/4 M expeldonotpublish2/62/6 Rows that stay at zero solved tasks on both ends are omitted. incidenttrack, postmortem, and onboarding are zero on every round of every seed-42 cell, matching the DeepSeek passes. A.8 Failure reasons outside the DeepSeek office cells Table 31 is DeepSeek office-v1. Table 22 is the seed-42 transfer on the same pack. Counts are maxsteps / verifierfail. A dash is a round the runner did not start. Table 22: Seed-42 office-v1 transfer. Counts are maxsteps / verifier fail. CellSeedr0r1r2r3 GLM noevolve4217/622/414/522/4 GLM gepa4214/712/1020/522/5 GLM ace4218/316/521/520/5 GLM expel4220/518/223/323/2 M no evolve4227/529/130/228/3 M gepa4231/229/325/429/2 M ace4223/429/329/0— M expel4228/328/331/127/0 Every recorded failure in Table 22 is maxsteps or verifierfail. Table 23: Manufacturing-v1 noevolve. Counts are belowthreshold / maxsteps. These passes have no verifier fail. Seedr0r1r2r3 4217/915/813/1016/9 4316/817/714/815/8 4415/916/815/915/8 A.9 Eligibility flags gated is false when final held-out regression exceeds 0.20, the run is over the evolution-token cap, the trace origin is not self, the anchor flag is not ok, the artifact does not match, contamination is set, M 0 is unbound, or the gate is off. paper min repeats those gated reasons and also fails a pack that is under 40/10/20, has no OOD slice, overlaps split ids, or leaves the default mix and schedule. Coding has no OOD slice, so this runner reason fires there; the paper still reports that board and states the single family in Section 4. On these office-v1 cells the pack size is 80/20/40, so a papermin mark that travels with gated 24 is the copied regression or budget reason, not a short pack. maintableeligible can still be true for a frozen no evolve cell whose regression is at most 0.20. That flag does not mean the policy changed. The Main column is the runner’s office-lane flag. It is not this paper’s ranking; coding is the primary scoreboard (Section 4). Table 24: Stored eligibility reasons for the 20 office-v1 cells. Main is the runner’s office-lane flag, not the coding scoreboard. CellSeedMainReasons DS ace42falsegated, papermin DS ace43trueeligible DS ace44falsegated, papermin DS expel42trueeligible DS expel43trueeligible DS expel44trueeligible DS gepa42falsegated, paper min DS gepa43falsegated, paper min DS gepa44trueeligible DS noevolve42trueeligible DS noevolve43falsegated, papermin DS no evolve44trueeligible GLM ace42falsegated, papermin, overbudget GLM expel42falsegated, papermin, overbudget GLM gepa42falsegated, paper min, overbudget GLM noevolve42falsegated, papermin M ace42falsegated, papermin, overbudget M expel42trueeligible M gepa42falsegated, paper min M noevolve42falsegated, papermin A.10 Office grids Setup. One hosted model is both the tool actor and the evolver. The actor is the TEB Standard Tool Actor: temperature 0, one user message, neutral prompt, at most 20 steps. Budget class is B0. The pack is frozen office-v1: catalog-workspace jobs, deterministic checks, no gold fields in the obser- vation (all three packs: Appendix A.1). Evolution chunks are 27/27/26 of the 80 evolution tasks. Seeds 42, 43, 44 shuffle evolution order, not held-out membership. The three-seed methods were fixed from a development screen on validation gain versus model-matched no evolve, then lower validation regres- sion, then lower tokens, before formal held-out output was read: gepa llm, acellm [Zhang et al., 2026b], expelllm. Their mean screen gains were 0, −5.6, and −11.1 p (Appendix A.3). awmllm and reasoningbank [Ouyang et al., 2026] failed the screen’s artifact check. dgm and clinllm [Ma- jumder et al., 2024] ranked below and were not given this matrix. Every finished cell has contamination=false and adapter honesty=ok (a labeled text update, not the upstream implementation). What a pass is. The actor’s tools are listfiles, readfile, writefile, publishartifact, grant access, and submit. A failed submit returns the failed check names and leaves the episode 25 open. Success is every terminal check passing. Gold files are not in the observation [Elangovan et al., 2021]. Four grade classes add checks on top of that contract: discover (names and list only), extract (bound phrases, length cap), report (facts under the required heading), and structured (strict JSON, no extra keys). Eight of the forty held-out tasks are structured. They stay inside the same Success denominator. Office Success is instruction following and rule-bounded delivery [Zhou et al., 2023, Ribeiro et al., 2020]. It is not a writing-quality score. A.10.1 Frozen band Table 25 and the first panel of Figure 7 are the control. DeepSeek v4 Pro and GLM 5.3 each span 17.5 p with no context update. MiniMax M2.5 [MiniMax, 2026] spans 5.0 p. DeepSeek seed 42’s path is 35.0, 32.5, 27.5, 45.0, all agent version=M 0 . Actor, prompt, and held-out ids are identical across these passes; the hosted endpoint is not deterministic at temperature 0 [Atil et al., 2024], and seeds do not control it. Round 0 itself is 35.0, 27.5, and 25.0 across the three seeds, so Gain’s baseline is one draw from this band. Seed 43’s regression is 0.231, so the runner marks that frozen cell ineligible: the regression gate fires on rescore noise. On seed 42, every frozen MiniMax pass (20–25%) is below every frozen GLM pass (35–52.5%), so the pack separates models even where it cannot resolve updates. A.10.2 Paired excess Figure 7 shows that no evolve moves with the adapters. Figure 8 shades the widest DeepSeek band, ±17.5 p, which is wider than seed 43’s own 7.5 p band. Means and the per-seed reading are in Section 6. Table 11 is the cell record. Its Gain column still contains the frozen movement in Table 25. 0123 round 20 40 held-out % no evolve 0123 round gepa llm 0123 round ace llm 0123 round expel llm DS 42DS 43DS 44GLM 42M 42 Figure 7: Held-out success by round. The first panel is frozen noevolve, including GLM 5.3 and Min- iMax M2.5 at seed 42. DeepSeek seed 42 goes from 27.5% at round 2 to 45% at round 3 with no context update. The other panels are DeepSeek only. noevolve moves as much as the adapters. A.10.3 Same agent, different score A version change is an evolve() accept, including 0 p ties under min delta=0: DeepSeek acellm seed 42 (35→35 to M 3 ), expelllm seed 44 (15→15, 30→30), and MiniMax gepallm (three ties to M 3 ; Appendix A.4). A score change without a version change is a rescore [Atil et al., 2024]. expelllm seed 43 is M 2 at both round 2 (20.0%) and round 3 (42.5%): +22.5 p on one artifact. gepa llm seed 43 rescores M 1 from 40.0% to 30.0%. GLM ace llm and expelllm stay M 0 : rounds 1 and 2 roll back, and round 3 hits the evolution-token cap. Their held-out paths still span 12.5 p and 15.0 p. Table 27 lists 26 gepa 42 4344 ace 42 4344 expel 42 4344 −20 −10 0 10 20 excess (p) Figure 8: DeepSeek office paired excess. Shading is the widest seed band,±17.5 p, not seed 43’s own 7.5 p band. Each seed’s range is the Range column of Table 25. same-version gaps of at least 10 p. Table 26 is the accepted version after every round. Figure 9 is every finished cell, next to the full grid. MiniMax ace llm has no round 3. A.10.4 Full grid Table 28 reports all 20 office-v1 cells; Figure 9 shows their round-wise scores. r0r1r2r3 DS no 42 DS no 43 DS no 44 DS gepa 42 DS gepa 43 DS gepa 44 DS ace 42 DS ace 43 DS ace 44 DS expel 42 DS expel 43 DS expel 44 GLM no 42 GLM gepa 42 GLM ace 42 GLM expel 42 M no 42 M gepa 42 M ace 42 M expel 42 3532.527.545 27.53532.535 253022.540 35254042.5 30304030 25353542.5 32.53037.537.5 32.52532.547.5 3027.532.532.5 2532.532.542.5 32.5302042.5 32.527.527.550 42.53552.535 47.54537.532.5 47.547.53537.5 37.5503537.5 20252022.5 17.52027.522.5 32.52027.5— 22.522.52032.5 15 20 25 30 35 40 45 50 55 held-out % Figure 9: Held-out success (%) at each round for all 20 cells. A dash is a round the runner did not start (MiniMax acellm round 3). 27 Flags: reg. means final regression > 0.20; budget means overbudget. GLM rows that show a round-3 success still recorded that held-out pass and were then marked over the evolution-token cap. GLM gepa llm regression is 0.200, which does not by itself exceed 0.20; the cell is ineligible because round 3 crossed 500,000 evolution tokens. DeepSeek expelllm seeds 42 and 44 reach M 3 . Their excesses are +7.5 and +2.5 p. Eligible DeepSeek excesses do not exceed the seed’s own range in Table 25. ace llm seed 43 is +7.5 p, equal to that seed’s 7.5 p range. The other positive eligible excesses are expelllm +7.5, +2.5, +2.5 and gepallm seed 44 +2.5, against the 17.5 p ranges of seeds 42 and 44 (Appendix A.9). A.10.5 Seed-42 transfer MiniMax M2.5 and GLM 5.3 are one seed. The reading is in Section 6. Figure 10 and Table 29 are the cell record. 0123 25 50 held-out % GLM no evolve 0123 GLM gepa 0123 GLM ace 0123 GLM expel 0123 round 25 50 held-out % M no evolve 0123 round M gepa 0123 round — M ace 0123 round M expel Figure 10: Seed-42 transfer. Top row: GLM 5.3. Bottom row: MiniMax M2.5. A dash is a round the runner did not start (MiniMax acellm round 3). These curves are not a second three-seed confirmation. A.10.6 Where the 40 tasks move Held-out n=40 splits into nine families (Table 30). Eight of the 80 evolution tasks are contentdraft; this held-out fingerprint contains none, so Gain cannot show that family. accessdenied and donotpublish are held-out only (n=6 each) [Koh et al., 2021]. The other seven are not marked OOD on this held-out finger- print. Figure 11 is the round-0 to final change in solved count for every DeepSeek cell. incidenttrack (n=3), postmortem (n=2), and onboarding (n=1) stay at zero solved tasks on every round of all twelve DeepSeek cells, including the three expelllm runs that reach a new agentversion. The movement that produces raw Gain is among families the frozen policy already solves on some passes: catalog discover, briefreport, checklist, csvaudit, accessdenied, and donotpublish. On noevolve seed 42 the final pass gains two briefreport, two checklist, one catalogdiscover, and one csvaudit, and loses one accessdenied and one donotpublish. expelllm seed 44, the highest final Success in the matrix (50%), moves checklist from 1/5 to 4/5 and catalog discover from 5/8 to 7/8. Those are the same families that move when the agent string does not change. A fam- ily rate on n≤8 flips when one or two tasks flip. That is the 2.5 p task grain inside the 17.5 p band (Appendix A.17). 28 catalog_discover brief_report checklist csv_audit access_denied do_not_publish incident_track postmortem onboarding no_evolve 42 no_evolve 43 no_evolve 44 gepa 42 gepa 43 gepa 44 ace 42 ace 43 ace 44 expel 42 expel 43 expel 44 +1+2+2+1-1-1000 0+1+100+1000 +1+1+300+1000 +1+1+1+10-1000 -1+2-1000000 +40+10+1+1000 -1+10+10+1000 +20+3+100000 0+100-1+1000 +2+3+100+1000 +1+1+20-1+1000 +2+1+30+2-1000 −4 −2 0 2 4 tasks Figure 11: DeepSeek family change from round 0 to the final held-out pass, in solved tasks. Zero-change families incidenttrack, postmortem, and onboarding are included. Color is the task delta. A.10.7 Failure reasons Raw Gain ranks no evolve seed 44 (+15 p, M 0 ) with gepallm seed 44 and two expelllm seeds at +17.5 p. Paired excess ranks gepa llm seed 43 and acellm seed 44 lowest (−7.5,−12.5). Table 31 is the movement: misses are the 20-step cap, and no evolve seed 42 goes 23, 25, 25, 20 maxsteps with no context update. A.11 Office artifact versions Table 28 in the main text is the score grid. Table 26 is the accepted artifact after each round. no evolve stays at M 0 on every seed. A repeated version means the gate rolled back or the runner rescored the same artifact. GLM acellm and expelllm stay at M 0 because the evolution-token cap skipped evolve(). GLM gepallm accepted M 1 and then rescored it. MiniMax acellm has no round 3. The evolution-token cap is 500,000 per round. After each evolution task the runner adds that task’s token total, the sum of its steps. A step uses usage.total tokens when the server sends it, otherwise prompt tokens plus completion tokens. Validation tasks, held-out tasks, and model calls inside evolve() are not added. Crossing the cap skips evolve() for that round, marks the cell overbudget, and skips later rounds when the violation is recorded before the next round. Screen scores are not results. The screen ranked gepa llm, acellm, and expelllm ahead of dgm and clinllm by mean validation gain against model-matched noevolve, then lower validation regression, then lower token cost. awm llm and reasoningbank failed the screen’s artifact check. Those four methods were not given this held-out matrix. Running them on the same Gain definition would repeat the remeasurement in Table 25. Every finished cell reports contamination=false, adapterhonesty=ok, traceorigin=self, trace mix=all, curriculum=none, budgetclass=B0, humanedits=0, and m0source=measured. The held-out fingerprint is identical across the DeepSeek cells that share the frozen pack. Temperature is 0. The client sends one user message. Inference timeout is 60 seconds, with retries on transport errors and HTTP 429/502/503/504. HTTP 402 is not retried. Held-out failures on the DeepSeek no evolve passes 29 are maxsteps and verifier failures. A response cache of identical prompts would stabilize a rescore; it would not create the version changes in Table 26. A.12 Manufacturing frozen repeat Manufacturing-v1 uses the same 80/20/40 shape, R=3, and DeepSeek v4 Pro noevolve on the Standard Tool Actor. The runner flags these cells envnotoffice. This is the manufacturing frozen band, scored separately from coding. All three seeds stay at M 0 , contamination=false, adapter honesty=ok, overbudget=false. The four-pass ranges are 7.5, 5.0, and 2.5 p. Failures on these passes are belowthreshold and maxsteps; verifierfail does not appear (Table 23). These ranges are not the office DeepSeek band and not the coding band (Appendix A.14). Office allows 20 steps per task; this repeat allows 12. A.13 Manufacturing family counts These are the finished DeepSeek v4 Pro no evolve cells in Table 32. Each entry is solved tasks on that family at rounds 0/1/2/3. agentversion stays M 0 . The office lane rejects the cells (envnotoffice). Families with zero solved tasks on every pass are accountspayable, advancepayments, advancereceipts, bpc report, receivablemonthly. pcrevenue is zero at both ends; seed 44 round 1 solves one task. Table 33: Manufacturing-v1 solved counts by family at rounds 0– 3, written r0/r1/r2/r3. FamilynSeed 42Seed 43Seed 44 accountspayable30/0/0/00/0/0/00/0/0/0 advancepayments30/0/0/00/0/0/00/0/0/0 advancereceipts10/0/0/00/0/0/00/0/0/0 bpcreport70/0/0/00/0/0/00/0/0/0 commodity31/2/2/22/2/2/22/2/2/2 electricity33/3/3/33/3/3/33/3/3/3 expense detail41/2/1/11/1/2/22/1/1/2 pcrevenue30/0/0/00/0/0/00/1/0/0 purchasereceipt33/3/3/33/3/3/33/3/3/3 receivablemonthly10/0/0/00/0/0/00/0/0/0 revenue cost31/1/2/01/1/2/20/1/1/1 tax32/3/3/33/3/3/33/3/3/3 utility33/3/3/33/3/3/23/2/3/3 A.14 Coding frozen repeat DeepSeek v4 Pro no evolve on the coding pack uses the same 80/20/40 shape and R=3. The runner flags these cells env notoffice. The four passes are Table 6. All three seeds stay at M 0 , contamination=false, adapterhonesty=ok, overbudget=false. The four-pass ranges are 7.5, 5.0, and 12.5 p, with Gains of−7.5, −2.5, and−10.0 p. The episode cap is 12 steps, not Office’s 20. Every miss is that cap. 30 Across the four passes the miss counts sum to 56, 60, and 63 on seeds 42, 43, and 44. Coding is a single family (fix bug, n=40), so there is no family split to tabulate. Final regression is 0.231 and 0.308 on seeds 42 and 44: the 0.20 cutoff fires on frozen rescore noise here too, as on office seed 43. A.15 Extra office frozen passes Six extra DeepSeek no evolve passes on the office-v1 held-out fingerprint (afd748c6, two per seed, R=0, no evolve() call) score 30.0 and 27.5 on seed 42, 27.5 and 22.5 on seed 43, and 30.0 and 27.5 on seed 44, all at M 0 with contamination=false. Merged with the original four passes, the per-seed ranges are 17.5, 12.5, and 17.5 p. The widest stays 17.5 p. Seed 43’s own range widens from the 7.5 p reported in Section 6 to 12.5 p, because one extra pass scores 22.5. These are the extra repeats the Limitations ask for. A.16 Coding grids These displays use the same 36 coding cells as Section 4. Held-out is one family, so there is no family-delta figure. Every recorded miss is max steps; verifierfail does not appear. r0r1r2r3 DS no 42 DS no 43 DS no 44 DS gepa 42 DS gepa 43 DS gepa 44 DS ace 42 DS ace 43 DS ace 44 DS expel 42 DS expel 43 DS expel 44 67.567.56560 62.56562.560 62.562.56552.5 62.577.57570 62.57572.575 657567.542.5 67.552.577.580 6572.57072.5 606557.560 70758077.5 6577.577.570 6567.572.575 0 20 40 60 80 100 held-out % Figure 12: DeepSeek coding held-out success (%) at each round. Twelve cells: four methods by three seeds. Paired excess is Figure 4. A.17 Worked examples The three task cards are evolution-split tasks. They are not held-out items, and they omit gold files, ex- pected SQL, and unit-test oracles. The last card is one stored DeepSeek no evolve held-out trace pair; its instruction and written files are not reproduced. 31 0123 20 40 60 80 GLM no evolve 0123 gepa llm 0123 ace llm 0123 expel llm 0123 20 40 60 80 MiniMax 012301230123 Figure 13: GLM (top row) and MiniMax (bottom row) coding held-out success by round. Both models have seeds 42, 43, and 44. Line color is the seed. A.17.1 One task from each current suite Office-v1 evolution task Task ofcevodiscoveropentasks Family catalogdiscover; verification class contract Instruction“List iteration-12 files and publish catalogindex.md.” ObservationTwo catalog files: catalog/iter12.md, catalog/owners.md. The gold output/catalog index.md is not in the observation. Tools listfiles, readfile, writefile, publishartifact, grant access, submit Step cap20 SuccessEvery terminal check passes. A failed submit returns the failed check names and leaves the episode open. Coding-v1 evolution task Task codingfixhideloadinggate Family fix bug; verification class executable InstructionChinese in the pack; English paraphrase: change show realnamegate so a query that is still loading does not open the gate; only required or unavailable opens it. Workspace src/solution.py Step cap12 SuccessUnit tests in the closed workspace pass. The gold patch is not shown. 32 r0r1r2r3 GLM no 42 GLM no 43 GLM no 44 GLM gepa 42 GLM gepa 43 GLM gepa 44 GLM ace 42 GLM ace 43 GLM ace 44 GLM expel 42 GLM expel 43 GLM expel 44 M no 42 M no 43 M no 44 M gepa 42 M gepa 43 M gepa 44 M ace 42 M ace 43 M ace 44 M expel 42 M expel 43 M expel 44 72.572.56070 606567.567.5 67.5707560 67.572.572.572.5 6582.572.572.5 65708072.5 67.557.552.5— 6577.577.5— 707082.577.5 552.56572.5 62.5758080 6082.58075 3037.54035 37.53537.537.5 4037.527.540 32.53042.530 27.537.552.572.5 4557.55555 32.5353532.5 27.542.547.545 3537.54035 32.5354545 4047.55542.5 37.54057.552.5 0 20 40 60 80 100 held-out % Figure 14: GLM and MiniMax coding held-out success (%) at each round. GLM acellm seeds 42 and 43 have no round 3. Paired excess for these cells is Figure 6. Manufacturing-v1 evolution task Task manufacturingxtax2025personal Family tax QuestionChinese in the pack; English paraphrase: the group’s 2025 personal-income-tax total. DataReal-derived tables; counterparty, company, and department names pseudonymized. Step cap12 on this repeat SuccessDeterministic score at least 80. A miss is below threshold or maxsteps; verifierfail is not a manufacturing failure. The expected number and SQL are not shown. A.17.2 What evolve() actually returns Frozen control: DeepSeek noevolve, office seed 42 UpdateNone. evolve() returns the same agent. Stored filesAn evolve input for rounds 1–3; no evolve-artifact file. VersionM 0 after every round. 33 0 p accept: DeepSeek acellm, office seed 42, round 3 GateValidation 0.350→ 0.350 under mindelta=0; accept (Table 15). VersionM 2 → M 3 CarrierText playbook. Round 3 keeps the four round-2 bullets and adds a fifth. Added bulletBefore the final publish artifact that precedes submission, cross-check the draft against every task requirement and output rule, and edit the file to prevent submit constraintsfailed. A.17.3 One held-out task, sameM 0 , two scores Rescore: DeepSeek noevolve, office seed 42 Task ofcholddiscoverrfqarchive (catalogdiscover) Round 2M 0 , 20 steps, fail (maxsteps) Round 3M 0 , 20 steps, pass EffectOne task is 2.5 p. This pair is part of the seed-42 path from 27.5% at round 2 to 45% at round 3. A.18 Archived ticket-gym mock This section is not office-v1. The hosted matrix in Section A.10 uses catalog tools (listfiles, read file, writefile, publishartifact, grantaccess, submit) and held-out n=40. The numbers below are the earlier in-process ticket mock (held-out n=20). They are not ranked against Table 28. Ticket-gym is a first-party in-process tool world: search, read, write file, updateticket, authrefresh, listtickets, grantaccess, submit. The verifier is deterministic (files, ticket state, submit args), and the same task id never crosses evolution / validation / held-out. Figure 15 visu- alizes the split; Table 38 lists its counts together with the inbox mock and the officework expansion. Evolution and validation are in-distribution only; held-out adds six access denied items (ood=true). Dual is therefore learnable from self failures; the OOD recover tool cannot be practiced on evolution. evolution (n=40) validation (n=10) held-out (n=20) 0 10 20 30 40 Tasks (a) Office paper-min skill families ticket_close dual_ticket auth_recovery file_report access_denied evolutionvalidationheld-out 0 2 4 6 8 OOD tasks (ood=true) 00 6 (b) OOD only in locked held-out Figure 15: Archived ticket-gym, not office-v1. Split is 40/10/20 and held-out n=20. OOD is locked in held-out. In-distribution families. ticket close: find a policy, write a summary, close one ticket. dualticket: write a bundle covering two topics and close two tickets with distinct notes; an incomplete submit does 34 not end the episode. authrecovery: auth can expire again after a successful refresh. filereport: report path plus required substrings. OOD family. accessdenied (n=6 on held-out only): a tool returns permissiondenied until grant access, which is not authrefresh. Evolution never contains this family. Transfer here is “new recover tool,” not a new domain. Canonical heuristic M 0 is ticket-close only, including authrefresh on expiry. It does not know dual-ticket close or grantaccess. On this pack that floor is 50% (10/20): the four ticket-close, three auth, and three file items, and none of dual or OOD. A.18.1 Why the protocol constraints Each check maps a provenance or evaluation failure to a stored eligibility flag. A teacher dump on the same tasks can look like a large Gain with no successful self rollout. The NL2SQL paper-min pack is the control: teacher schema memory reaches 100%, self traces stay at 0% (anchorfloor; Appendix A.18.14). Teacher and self traces therefore occupy separate labeled boards. If evolve() sees held-out ids or gold fields, Success is leakage: the runner hashes the sorted held-out taskid list and scans dumps. Training on held-out is not a TEB measurement. Scoring a live server whose artifactid differs from the value returned by evolve() breaks ar- tifact binding. The discarded DGM-inspired run listed grantaccessondenied in the mutation ISA and reached 80% / OOD 6/6 without OOD traces, so it is excluded (Appendix A.18.13). A B2 fullft and a B0 tip are different objects; the same evolver on a heuristic actor, frozen-3B wrapper, and native runtime is three cards. Without rollback, wrapper GEPA [Agrawal et al., 2026] / Letta [Packer et al., 2023] / DGM [Zhang et al., 2026a] would look evolved; they reject every round, and dgm native accepts with validation 70% while held-out stays 50% (Appendix A.18.12). Turning the gate off on B0 does not move those cards. R=1 cannot separate a jump from a loop; on this pack the heuristic jump is at r1, so R=3 is the eligibility floor. M 0 outside [0.10, 0.80] cannot show Gain and is flagged. The default mix is all (successes and failures) because failure traces carry information that success traces alone do not encode—on ticket-gym, dual-ticket learning relies on recoverable failed submits that expose the missing second ticket. Historical weight success only artifacts are excluded with their parent card because their trace actor and evaluated actor differ; they cannot support a mix claim. The default curriculum is none [Bengio et al., 2009] (static chunks) because static ordering keeps rounds comparable: every seed sees the same evolution ids in the same round, so round-level differences are attributable to the evolver, not to task selection. The failurefamily ablation reorders remaining evolution ids toward families that failed on prior traces; it does not raise heuristic held-out because dual is already saturated and OOD never enters evolution. It is a labeled ablation, not a candidate default. The default schedule is accumulate chunks: each evolution id is collected in at most one round, and evolve() sees traces from rounds 1. . .r. replay reuses the first 14 evolution ids across all rounds instead; it is marked ineligible (roundschedule) because a second pass over the same tasks cannot introduce new failure modes. accumulate chunks ensures that each round contributes new traces to the evolving payload, so a system that learns from novel failures is not starved of them. The archived ticket-gym uses a 40/10/20 paper-min floor. Office-v1 and manufacturing-v1 are 80/20/40. That floor (R=3,≥ 3 seeds) is a statistical minimum, not an optimum. Held-out n=20 gives each family at least 3–4 items so a family rate is not a single-task coin flip; evolution n=40 gives R=3 at least 13 new ids per round under accumulate chunks, so a round is not a single-batch update. Fewer than these and a card’s Success % and Gain are noisy enough to obscure the family split that the protocol exists to expose. 35 TEB does not require weight updates or emit a scalar “evolution IQ.” Seeds do not alter held-out membership, failed self rollouts are not replaced with teacher traces, held-out inference is outside the up- date budget class, and main-table verification is deterministic rather than LLM-as-judge [Zheng et al., 2023]. A.18.2 Success, Gain, and actors Success % = 100×#held-out pass/#held-out. Gain (p) = 100×(Succ(M r )−Succ(M 0 )), with M 0 this-run unless the card is eval-only with a frozen floor. Rank only inside (env,budgetclass,traceorigin,actor). An adapter name is a retrofit; the actor is who emits the tool call. A.18.3 Setup of the archived mock All ticket-gym numbers use tasks 40/10/20, R=3, traceorigin=self, mix all, schedule accumulatechunks, gate mindelta =0, unless labeled otherwise. Evolution collection is chunked at 14 tasks per round. Episode cap is 20 env steps. Seeds42, 43, 44 shuffle evolution order, not held-out membership. Heuristic B0. Actor: SandboxAgent. Evolvers: noevolve, promptmemory, successmemory, stand-in reflexion [Shinn et al., 2023] / expel [Zhao et al., 2024] / awm [Wang et al., 2025]. No train GPU. Stand-in Reflexion maps failure traces onto executable tips (e.g. workflow:dual ticketclose); there is no LLM judge. Weight B2. Actor: Qwen2.5-3B-Instruct [Qwen Team, 2024] after self-trace full fine-tune. Eval binds evolve’s artifact. Gain vs. frozen heuristic M 0 =0.50 (eval-only of a trained checkpoint is not a re-measure of untrained 3B M 0 ). Verbal B0 wrappers. Same frozen 3B, TEB-stuffed tips, seed 42. Gain vs. this-run M 0 (measured 0.45/0.50/0.55 across cards; not the heuristic 0.50). Wrapper M 0 is unbound relative to the heuristic floor, so these rows are not ranked into the B0 heuristic card. These are not native runs of Packer et al. [2023], Agrawal et al. [2026], Zhang et al. [2026a], Zhai et al. [2025]. Native pairing (separate board): Letta [Packer et al., 2023] messages API as actor; DGM [Zhang et al., 2026a] genome compiled onto SandboxAgent without leaking OOD tool names in the mutation ISA; the same Reflexion-style [Shinn et al., 2023] wrapper loop on a larger chat model (qwen-plus). inbox is a second first-party mock with the same 40/10/20 shape (Section A.18.8). The NL2SQL paper-min pack is a teacher-contrast negative control, not a third headline world (Appendix A.18.14). A.18.4 Summary boards and expansion families Table 39 separates the eligible heuristic ticket-gym card from one-seed wrapper contrasts and the incomplete weight attempt. Rows are boards, not methods to sort. The expansion pack is not another board: it has M 0 =0.20 and different families, so 75% is not a denser 70%. Table 40 is that pack’s family counts. Table 48 is the same pack’s Success summary. A.18.5 Round curves Heuristic stand-in Reflexion jumps at r1 and saturates (70%). Other B0 methods stay at the M 0 floor. Among one-seed wrappers, only reflexion llm moves (45→55); GEPA/Letta/DGM wrappers roll back to this- run M 0 ; agentevolver 45→55 occurs under rollback and is M 0 jitter, not a second +10p. Wrapper file report stays 0/3. 36 A.18.6 Mechanism: what heuristic failures teach At heuristic M 0 , ticket-close, auth, and file are already solved. Stand-in Reflexion’s +20p is dual-ticket going 0/4→4/4 at r1 from self incomplete-submit traces; OOD stays 0/6 because that family is absent from evolution. Figure 16 is a live SandboxAgent rollout on two locked held-out ids, not a 3B completion log. On dual (ofc holddu001), M 0 searches once then repeatedly submits a single ticket; the verifier returns submit constraintsfailed and the episode continues to maxsteps. After the stand-in writes workflow:dualticketclose, the same id is solved by two updatetickets with distinct notes. On OOD (ofcholdad001), M 0 never calls grantaccess and dies in permissiondenied; the recover tool is a one-step unlock, after which the ticket-close loop succeeds. 1. search 2. search → auth expired 3. auth_refresh() 4. submit ×4 → constraints fail 5. auth_refresh() 6. submit ×4 → constraints fail 7. auth_refresh() 8. submit ×4 → constraints fail 9. auth_refresh() 10. submit ×2 → constraints fail FAIL · one-ticket submit · continues to cap (a) Dual M 0 · ofc_hold_du_001 1. search 2. read(refund) → auth expired 3. auth_refresh() 4. search 5. read(refund) 6. write_file 7. update(refund) → auth expired 8. auth_refresh() 9. update(refund) 10. update(billing) 11. submit → pass PASS · two distinct notes, then submit (b) Dual after workflow:dual_ticket_close 1. search ×2 → denied 2. auth_refresh() 3. search ×13 → denied FAIL · never grant_access (c) OOD M 0 · ofc_hold_ad_001 1. grant_access() 2. search → auth expired 3. auth_refresh() 4. search 5. read(refund) 6. write_file 7. update(refund) 8. submit → pass PASS · recover tool, then ticket-close (d) OOD after grant_access_on_denied Figure 16: Archived ticket-gym tool calls, not office-v1.Locked held-out ids (SandboxAgent, not a 3B log).(a) M 0 dual: recoverable failed submit, second ticket never closed.(b) Same id after workflow:dual ticketclose.(c) M 0 OOD: no grantaccess.(d) OOD after grantaccessondenied. Consecutive identical calls are collapsed (×n). A hygiene run that drops grantaccess from the serving prompt puts no-hint 3B M 0 at 30%, OOD 0/6 (Table 44). The hinted wrapper’s M 0 OOD came from the prompt, not a learned recover skill. Frozen wrapper cards keep the hinted prompt so they reproduce; the no-hint M 0 is reported separately. 37 A.18.7 WhyR=3 is enough on ticket-gym Heuristic Reflexion jumps at r1 and saturates (70% / dual 4/4). Extra dual practice (failurefamily) and replay of the first 14 evolution ids do not raise held-out, because dual is already solved for the stand-in and OOD access denied is still absent from evolution. We therefore keep R=3 as the replication floor rather than grinding more rounds. A.18.8 Inbox mock To check that the protocol is not specific to ticket-gym, inbox provides a second in-process mock (list, read, label, archive, reply, quarantine) with the same 40/10/20 paper-min shape, R=3, and three seeds (Table 43). Canonical M 0 is triage-only (40%). B0 stand-in Reflexion reaches 70% / +30p by unlocking reply 3/3 and thread 3/3, and never OOD quarantine (0/6). Other B0 methods stay at 40%. As in ticket-gym, the stand-in learns an in-distribution family exposed by failed submits. Historical inbox weight grids are excluded because first-round traces came from InboxAgent, not the evaluated 3B. Inbox and ticket-gym remain separate boards; inbox is not a production mail client. A.18.9 Seed, wrapper, inbox, and hygiene tables These tables record the archived heuristic seeds, the seed-42 3B wrappers, the inbox board, and the prompt- hygiene check. They are not a ranking and not extra models. Step-token columns in the runner are estimates (Limitations); we do not treat them as a serving benchmark. Wall-clock is process time of the seed-42 3B loops only. Wrapper round paths below are held-out Success % of the accepted agent (roll- back keeps M r−1 ; re-eval jitter is labeled). Native pairing is Table 46. A.18.10 Weight-track provenance Historical ticket-gym, inbox, and office-work weight grids are excluded. Their first round collected “self” traces with local heuristic actors, then trained and evaluated a 3B checkpoint; a scalar frozen heuristic M 0 did not bind that change of actor. Artifact matching alone cannot repair the provenance mismatch. The ticket-gym native attempt collects traces with the served 3B actor, but its prepare submit round did not submit training and was rolled back; the resulting 50% / +0 card is an ineligible infrastructure outcome, not a weight-method result. We therefore report no weight-track headline. A.18.11 Ticket-gym B0 ablations Gate-off, replay, and failure-family runs keep heuristic Reflexion at 70% and other B0 methods at 50% (three seeds). Replay and failure-family remain labeled contrasts. Historical weight-mix and hybrid artifacts inherit the trace-actor provenance failure and are not ablation results. A.18.12 Ticket-gym gate decisions Figure 17 is the accept/rollback heatmap. Table 45 adds the held-out consequence. Green in the figure is accept; orange is rollback. Office seed 42 unless labeled B0. Validation Success is the gate input; held-out Success is scored on the accepted agent after that decision. A.18.13 Ticket-gym native pairing Not ranked vs. Table 39 (Table 46). letta native: this-run M 0 60%, final 65% / +5p, which is M2 re-eval jitter after r3 rollback (r1 held-out was 10%); dual 2/4 is the first 3B-backed dual on this pack, while 38 r1r2r3 B0 reflexion reflexion_llm gepa_llm letta wrapper dgm wrapper agent_evolver letta_native dgm_native qwen-plus acceptacceptaccept acceptaccept rollback rollbackrollbackrollback rollbackrollbackrollback rollbackrollbackrollback rollbackrollbackrollback acceptaccept rollback acceptacceptaccept acceptacceptaccept Validation gate decisions · accept is not held-out Gain Figure 17: Archived ticket-gym gates, not office-v1. Accept and rollback are not held-out Gain. ticketclose stays 0/4. dgmnative (ISA does not name OOD tools): 50% / +0p. A discarded run that listed grant accessondenied in the mutation ISA hit 80% / OOD 6/6 without OOD traces— wrong mechanism, not a result. Historical weight grids are audit artifacts, not results. The scale probe in this freeze is the same Reflexion-style wrapper loop on qwen-plus (a larger chat model than the bound 3B; not a 7B/14B fullft). This-run M 0 is 5%; peak M2 is 15%; the card is final 5% / +0p / dual 0/4 (r3 undid M2). A bigger decoder did not unlock dual the way the heuristic stand-in does. We do not headline the M2 peak. M0r1r2r3 Protocol round 0 20 40 60 80 100 Held-out Success (%) (a) Native pairing / larger LLM letta_native dgm_native qwen-plus + reflexion_llm dualfile_reportOOD 0 20 40 60 80 100 Family Success (%) (b) Final-round families letta_native dgm_native qwen-plus Figure 18: Archived ticket-gym native pairing board, not office-v1. Not ranked vs. Table 39. A.18.14 NL2SQL teacher contrast NL2SQL paper-min 40/10/20 [Yu et al., 2018, Li et al., 2023], not the 12/3/9 snapshot. Table 47: teacher schemamemory 0→100% by r1 (three seeds), OOD 4/4; self all 0% (anchorfloor). This is a labeled teacher contrast and a protocol floor, not an office-v1 result. Teacher 100% is not mixed with any self board. 39 M0r1r2r3 0 20 40 60 80 100 Held-out Success (%) (a) Teacher traces schema_memory prompt_memory no_evolve M0r1r2r3 (b) Self traces schema_memory prompt_memory no_evolve Figure 19: NL2SQL teacher contrast, not office-v1. (a) Teacher traces. (b) Self traces, which contain no successful SQL, so every evolver stays at 0%. A.18.15 Ticket-gym expansion pack Same ticket-gym tools, new families mapped from public product-usage cases, 40/10/20. Family counts are Table 40. Env id office work, lane contrast (packnotv1). Not a fourth frozen actor board and not mixed with ticket-gym or inbox. Heuristic B0 (three seeds): measured M 0 20%; stand-in Reflexion 75% / +55p (in-dist 15/15); other B0 stay 20%; OOD accessdenied 0/3 and checklistbound 0/2 (Figure 20, Table 48). No valid weight result is reported. 40 Table 15: Validation gate on office-v1. Columns are validation percent before and after the proposal. n=20, so one task is 5 p. Model MethodSeed Round Version Before AfterDecision DSace421M120.040.0accept DSace422M235.050.0accept DSace423M335.035.0accept DSace431M030.020.0rollback DSace432M025.05.0rollback DSace433M115.035.0accept DSace441M125.030.0accept DSace442M225.035.0accept DSace443M235.025.0rollback DSexpel421M115.020.0accept DSexpel422M220.035.0accept DSexpel423M325.035.0accept DSexpel431M125.035.0accept DSexpel432M220.020.0accept DSexpel433M225.020.0rollback DSexpel441M115.025.0accept DSexpel442M215.015.0accept DSexpel443M330.030.0accept DSgepa421M025.015.0rollback DSgepa422M120.030.0accept DSgepa423M135.025.0rollback DSgepa431M030.025.0rollback DSgepa432M120.035.0accept DSgepa433M120.05.0rollback DSgepa441M030.025.0rollback DSgepa442M020.010.0rollback DSgepa443M120.025.0accept DSno evolve421M020.015.0rollback DSnoevolve422M020.020.0accept DSno evolve423M025.025.0accept DSnoevolve431M030.015.0rollback DSnoevolve432M030.015.0rollback DSnoevolve433M020.020.0accept DSno evolve441M030.015.0rollback DSnoevolve442M020.025.0accept DSnoevolve443M030.025.0rollback GLMace421M050.040.0rollback GLMace422M045.035.0rollback GLMace423M0— over budget GLMexpel421M050.035.0rollback GLMexpel422M060.030.0rollback GLMexpel423M0— over budget GLMgepa421M150.050.0accept GLMgepa422M150.045.0rollback GLMgepa423M1— over budget GLMno evolve421M045.040.0rollback GLMnoevolve422M060.045.0rollback GLMnoevolve423M055.030.0rollback MMace421M110.015.0accept MMace422M1— over budget MMexpel421M010.05.0rollback MMexpel422M015.010.0rollback MMexpel423M10.05.0accept MMgepa421M15.05.0accept MMgepa422M25.05.0accept MMgepa423M310.010.0accept MMnoevolve421M010.05.0rollback MMno evolve422M010.010.0accept MMnoevolve423M05.00.0rollback 41 Table 17: DeepSeek noevolve solved counts by family and round. incidenttrack, postmortem, and onboarding are omitted (0 on every pass). Seed Roundcatalog discover brief report checklistcsv audit access denied do not publish 420600125 421411124 422400115 423722214 430510113 431601124 432501124 433521114 440500113 441411123 442400113 443613114 Table 18: DeepSeek gepallm solved counts by family and round. incidenttrack, postmortem, and onboarding are omitted (0 on every pass). Seed Roundcatalog discover brief report checklistcsv audit access denied do not publish 420511115 421501112 422511126 423622214 430501114 431511113 432521224 433420114 440320113 441520124 442520115 443721124 42 Table 19: DeepSeek acellm solved counts by family and round. incidenttrack, postmortem, and onboarding are omitted (0 on every pass). Seed Roundcatalog discover brief report checklistcsv audit access denied do not publish 420502114 421510114 422620223 423412215 430520114 431401113 432520114 433723214 440510123 441600113 442312115 443520114 Table 20: DeepSeek expelllm solved counts by family and round. incidenttrack, postmortem, and onboarding are omitted (0 on every pass). Seed Roundcatalog discover brief report checklistcsv audit access denied do not publish 420500113 421520114 422411115 423731114 430510124 431510024 432300113 433622115 440511114 441510113 442411113 443724133 Table 25: Frozen-policy band. Range is max minus min of the four held-out passes, in percentage points. ModelSeedr0r1r2r3 Range DeepSeek v4 Pro 4235.0 32.5 27.5 45.017.5 DeepSeek v4 Pro 4327.5 35.0 32.5 35.07.5 DeepSeek v4 Pro 4425.0 30.0 22.5 40.017.5 GLM 5.34242.5 35.0 52.5 35.017.5 MiniMax M2.54220.0 25.0 20.0 22.55.0 43 Table 26: Accepted agentversion after each office-v1 round. A repeated version means the artifact did not change. On noevolve that is every round. Model MethodSeed r0r1r2r3 DSnoevolve42 M 0 M 0 M 0 M 0 DSno evolve43 M 0 M 0 M 0 M 0 DSno evolve44 M 0 M 0 M 0 M 0 DSgepa42 M 0 M 0 M 1 M 1 DSgepa43 M 0 M 0 M 1 M 1 DSgepa44 M 0 M 0 M 0 M 1 DSace42 M 0 M 1 M 2 M 3 DSace43 M 0 M 0 M 0 M 1 DSace44 M 0 M 1 M 2 M 2 DSexpel42 M 0 M 1 M 2 M 3 DSexpel43 M 0 M 1 M 2 M 2 DSexpel44 M 0 M 1 M 2 M 3 GLMno evolve42 M 0 M 0 M 0 M 0 GLMgepa42 M 0 M 1 M 1 M 1 GLMace42 M 0 M 0 M 0 M 0 GLMexpel42 M 0 M 0 M 0 M 0 MMno evolve42 M 0 M 0 M 0 M 0 MMgepa42 M 0 M 1 M 2 M 3 MMace42 M 0 M 1 M 1 — MMexpel42 M 0 M 0 M 0 M 1 Table 27: Consecutive held-out passes with the same agentversion and an absolute gap of at least 10 p. The accepted artifact did not change between the two scores. CellVersion FromTo Gap DS noevolve 42M 0 27.5 45.0 17.5 DS no evolve 44M 0 22.5 40.0 17.5 DS gepa 42M 0 35.0 25.0 10.0 DS gepa 43M 1 40.0 30.0 10.0 DS gepa 44M 0 25.0 35.0 10.0 DS expel 43M 2 20.0 42.5 22.5 GLM noevolve 42 M 0 35.0 52.5 17.5 GLM no evolve 42 M 0 52.5 35.0 17.5 GLM ace 42M 0 47.5 35.0 12.5 GLM expel 42M 0 37.5 50.0 12.5 GLM expel 42M 0 50.0 35.0 15.0 44 Table 28: All 20 office-v1 cells. Success columns are held-out percent. Gain and excess are percentage points. Excess is blank on noevolve. DS / GLM / M are DeepSeek v4 Pro, GLM 5.3, and MiniMax M2.5. Model MethodSeedr0r1r2r3Gain Excess Flag DSnoevolve42 35.0 32.5 27.5 45.0 +10.0— DSnoevolve43 27.5 35.0 32.5 35.0 +7.5— reg. DSno evolve44 25.0 30.0 22.5 40.0 +15.0— DSgepa42 35.0 25.0 40.0 42.5 +7.5−2.5 reg. DSgepa43 30.0 30.0 40.0 30.0 0.0−7.5 reg. DSgepa44 25.0 35.0 35.0 42.5 +17.5 +2.5 DSace42 32.5 30.0 37.5 37.5 +5.0−5.0 reg. DSace43 32.5 25.0 32.5 47.5 +15.0 +7.5 DSace44 30.0 27.5 32.5 32.5 +2.5−12.5 reg. DSexpel42 25.0 32.5 32.5 42.5 +17.5 +7.5 DSexpel43 32.5 30.0 20.0 42.5 +10.0 +2.5 DSexpel44 32.5 27.5 27.5 50.0 +17.5 +2.5 GLMno evolve42 42.5 35.0 52.5 35.0−7.5— reg. GLMgepa42 47.5 45.0 37.5 32.5−15.0−7.5 budget GLMace42 47.5 47.5 35.0 37.5−10.0−2.5 budget GLMexpel42 37.5 50.0 35.0 37.5 0.0 +7.5 budget MMnoevolve42 20.0 25.0 20.0 22.5 +2.5— reg. MMgepa42 17.5 20.0 27.5 22.5 +5.0 +2.5 reg. MMace42 32.5 20.0 27.5—−5.0−7.5 budget MMexpel42 22.5 22.5 20.0 32.5 +10.0 +7.5 Table 29: Seed-42 transfer. Excess is against that model’s noevolve. Not ranked with DeepSeek. ModelMethodSucc.Gain Excess Note MiniMax noevolve22.5 +2.5— reg. 0.25 MiniMax gepa22.5 +5.0 +2.5 inside 5.0; reg. 0.273 MiniMax ace27.5 −5.0 −7.5 stopped at round 2 MiniMax expel32.5 +10.0 +7.5 2.5 p past a 5.0 band GLMno evolve35.0 −7.5— reg. 0.381 GLMgepa32.5 −15.0 −7.5 budget; reg. 0.20 GLMace37.5 −10.0 −2.5 budget; version stays M 0 GLMexpel37.5 0.0 +7.5 budget; version stays M 0 Table 30: Office-v1 held-out families. n sums to 40. OOD families do not appear in evolution. Familyn OOD What the check is catalogdiscover80 List /catalog, write and publish catalogindex.md briefreport50 Bounded summary; invented owners, dates, or claims fail checklist50 Required sections only csvaudit40 Recompute stated counts or clean a table incidenttrack30 Timeline, risk, and triage; unknowns stay unknown postmortem20 Facts and actions split; blame fails onboarding10 Numbered first-run steps accessdenied66 Tools return permissiondenied until grantaccess do notpublish66 Write the draft and submit without publishartifact 45 Table 31: DeepSeek failure counts by round, maxsteps / verifierfail. The two counts sum to the number of held-out failures. No other failure reason appears. MethodSeedr0r1r2r3 noevolve 4223/3 25/2 25/4 20/2 noevolve 4324/5 21/5 24/3 25/1 noevolve 4423/7 26/2 25/6 21/3 gepa4224/2 26/4 22/2 22/1 gepa4327/1 26/2 19/5 26/2 gepa4425/5 22/4 24/2 18/5 ace4221/6 24/4 21/4 22/3 ace4324/3 25/5 25/2 20/1 ace4423/5 25/4 26/1 22/5 expel4227/3 23/4 24/3 22/1 expel4324/3 24/4 24/8 20/3 expel4423/4 25/4 23/6 14/6 Table 32: Manufacturing-v1, DeepSeek v4 Pro, noevolve only. Success is held-out percent. Version is M 0 at every round. Seedr0r1r2r3Range 4235.042.542.537.57.5 4340.040.045.042.55.0 4440.040.040.042.52.5 Table 34: Coding frozen-policy passes for GLM and MiniMax. Gain and Range for all three models are Table 7. DeepSeek passes are Table 6. Range is max minus min of the four held-out passes, in percentage points. ModelSeedr0r1r2r3 Gain Range GLM4272.5 72.5 60.0 70.0-2.512.5 GLM4360.0 65.0 67.5 67.5+7.57.5 GLM4467.5 70.0 75.0 60.0-7.515.0 MiniMax 4230.0 37.5 40.0 35.0+5.010.0 MiniMax 4337.5 35.0 37.5 37.5+0.02.5 MiniMax 4440.0 37.5 27.5 40.0+0.012.5 46 Table 35: Accepted agentversion after each coding round. A repeated version means the artifact did not change. On noevolve that is every round. ModelMethodSeed r0r1r2r3 DeepSeek noevolve42 M 0 M 0 M 0 M 0 DeepSeek no evolve43 M 0 M 0 M 0 M 0 DeepSeek no evolve44 M 0 M 0 M 0 M 0 DeepSeek gepa42 M 0 M 1 M 1 M 1 DeepSeek gepa43 M 0 M 1 M 1 M 1 DeepSeek gepa44 M 0 M 1 M 2 M 2 DeepSeek ace42 M 0 M 0 M 1 M 1 DeepSeek ace43 M 0 M 0 M 1 M 1 DeepSeek ace44 M 0 M 0 M 0 M 0 DeepSeek expel42 M 0 M 1 M 1 M 1 DeepSeek expel43 M 0 M 1 M 1 M 1 DeepSeek expel44 M 0 M 0 M 1 M 1 GLMno evolve42 M 0 M 0 M 0 M 0 GLMno evolve43 M 0 M 0 M 0 M 0 GLMnoevolve44 M 0 M 0 M 0 M 0 GLMgepa42 M 0 M 1 M 1 M 2 GLMgepa43 M 0 M 1 M 2 M 2 GLMgepa44 M 0 M 1 M 2 M 3 GLMace42 M 0 M 1 M 1 — GLMace43 M 0 M 1 M 1 — GLMace44 M 0 M 1 M 1 M 2 GLMexpel42 M 0 M 1 M 2 M 3 GLMexpel43 M 0 M 1 M 2 M 2 GLMexpel44 M 0 M 0 M 1 M 2 MiniMaxno evolve42 M 0 M 0 M 0 M 0 MiniMaxnoevolve43 M 0 M 0 M 0 M 0 MiniMaxno evolve44 M 0 M 0 M 0 M 0 MiniMaxgepa42 M 0 M 0 M 0 M 0 MiniMaxgepa43 M 0 M 1 M 2 M 3 MiniMaxgepa44 M 0 M 1 M 1 M 1 MiniMaxace42 M 0 M 1 M 2 M 3 MiniMaxace43 M 0 M 1 M 2 M 3 MiniMaxace44 M 0 M 1 M 2 M 3 MiniMaxexpel42 M 0 M 1 M 1 M 2 MiniMaxexpel43 M 0 M 1 M 1 M 1 MiniMaxexpel44 M 0 M 1 M 2 M 3 Table 36: Consecutive coding held-out passes with the same agentversion and an absolute gap of at least 10 p. The accepted artifact did not change between the two scores. ModelMethodSeed Version Rounds FromToGap DeepSeek noevolve44 M 0 2–365.0 52.5 -12.5 DeepSeek gepa44 M 2 2–367.5 42.5 -25.0 DeepSeek ace42 M 0 0–167.5 52.5 -15.0 GLMno evolve42 M 0 1–272.5 60.0 -12.5 GLMnoevolve42 M 0 2–360.0 70.0 +10.0 GLMnoevolve44 M 0 2–375.0 60.0 -15.0 GLMace44 M 1 1–270.0 82.5 +12.5 GLMexpel44 M 0 0–160.0 82.5 +22.5 MiniMaxnoevolve44 M 0 1–237.5 27.5 -10.0 MiniMaxnoevolve44 M 0 2–327.5 40.0 +12.5 MiniMaxgepa42 M 0 1–230.0 42.5 +12.5 MiniMaxgepa42 M 0 2–342.5 30.0 -12.5 MiniMaxexpel42 M 1 1–235.0 45.0 +10.0 MiniMaxexpel43 M 1 2–355.0 42.5 -12.5 47 Table 37: DeepSeek coding failure counts by round. The count is maxsteps. verifierfail is absent on every pass. MethodSeedr0r1r2r3 noevolve 4213 13 14 16 noevolve 4315 14 15 16 noevolve 4415 15 14 19 gepa42159 10 12 gepa4315 10 11 10 gepa4414 10 13 23 ace4213 1998 ace4314 11 12 11 ace4416 14 17 16 expel4212 1089 expel431499 12 expel4414 13 11 10 Table 38: Archived ticket-gym family counts, held-out n=20, not office-v1. OOD families are held-out only. The same table lists the inbox mock and the office work expansion. EnvFamilyEvoValHeld-out ticket-gym ticketclose1234 ticket-gym dual ticket1034 ticket-gym authrecovery923 ticket-gym filereport923 ticket-gym accessdenied (OOD)006 inbox triage1648 inbox replyneeded1233 inbox thread pair1233 inbox quarantine (OOD)006 officework ticketclose624 office work issuetriage823 officework releasenotes722 officework postmortem722 officework diffreview612 office work tablebrief612 officework accessdenied (OOD)003 officework checklistbound (OOD)002 Table 39: Archived ticket-gym mock, held-out n=20. Not an office-v1 score. Heuristic rows are three seeds. Wrapper rows are one seed. The weight attempt submitted no training. BoardSucc.GaindualfileOODSeeds B0 reflexion (heuristic)70%+20 vs 0.50 4/4 3/3 0/63 B0 other (heuristic)50%+00/4 3/3 0/63 Weight attempt (ineligible)50%+0 vs 0.50 0/4 3/3 0/61 3B reflexion llm55%+10 vs 0.45 0/4 0/3 4/61 3B other wrappers45–55% +0 this-run 0/4 0/3 3–4/61 48 Table 40: Archived ticket-gym expansion pack officework, not office-v1. Held-out n=20, three identi- cal seeds. Heuristic B0 only. Not ranked vs. Table 39. FamilynM 0 / other B0reflexion ticketclose44/44/4 issuetriage30/33/3 releasenotes20/22/2 postmortem20/22/2 diff review20/22/2 tablebrief20/22/2 access denied (OOD)30/30/3 checklistbound (OOD)20/20/2 Held-out Success2020%75% (+55p) Table 41: Archived ticket-gym heuristic B0, not office-v1. Held-out Success % by seed. Other B0 methods are noevolve, promptmemory, successmemory, expel, awm. Evolvers42s43s44dualOOD reflexion (stand-in)707070 4/4 0/6 no evolve505050 0/4 0/6 promptmemory505050 0/4 0/6 successmemory505050 0/4 0/6 expel505050 0/4 0/6 awm505050 0/4 0/6 Table 42: Archived ticket-gym 3B tip-wrappers, not office-v1. Seed 42. Columns r0–r3 are held-out Suc- cess % of the accepted agent. reflexion llm r3 rolls back (card is M2 55%). agentevolver 45→50→55 occurs under three rollbacks (jitter), not a second +10p. Dual 0/4 and file 0/3 on every row. Wall is loop process time, not time-to-first-token. Loopr0r1r2r3GainOODWall (h) reflexionllm45555555 +10 4/64.0 gepallm50505050 +0 3/63.8 letta wrapper55555555 +0 4/65.9 dgm wrapper45454545 +0 3/65.7 agent evolver45505555jitter 4/65.4 Table 43: Archived inbox mock, not office-v1. Held-out n=20. Not ranked vs. Table 39. Reply n=3, thread n=3, OOD quarantine n=6. BoardSucc.GainreplythreadOODSeeds B0 reflexion70% +30 vs 0.40 3/33/30/63 B0 other40%+00/30/30/63 49 Table 44: Archived ticket-gym hygiene, not office-v1. Same frozen 3B, serving prompt with vs. without grantaccess hint. Eval-only (R=0). Dual 0/4 on both rows. Table 42 keeps the hinted prompt. PromptSucc.tcauthfileOOD Hinted wrapper M 0 45% 4/4 2/3 0/3 3/6 No hint M 0 30% 4/4 2/3 0/3 0/6 Table 45: Archived ticket-gym gates, not office-v1 (Table 15). A = accept, R = rollback. dgmnative can accept a validation 70% proposal while held-out stays 50%. qwen-plus r3 accepts 0→0; the card is final 5%, not the M2 peak. Cardr1r2r3Held-out note B0 reflexionAAA70% from r1 reflexionllmAARstays M2 55% gepa/letta/dgm wrapRRRthis-run M 0 agent evolverRRR45→55 jitter lettanativeAARr1 is 10%; final M2 jitter 65% dgm nativeAAAval 70%, held-out 50% qwen-plusAAAr0–r3: 5, 5, 15, 5 Table 46: Archived ticket-gym native pairing, not office-v1. Seed 42. Columns r0–r3 are held-out Success % of the accepted agent. Dual/OOD are the frozen (final) card. Not ranked vs. Table 39. letta native r3 rolls back: final 65% is M2 re-eval jitter vs. r2 55%. qwen-plus peak r2 15% is not the card. dgmnative actor is SandboxAgent (genome only). File: Letta/DGM 3/3, qwen-plus 0/3. Ticket- close: Letta 0/4, DGM 4/4, qwen-plus 0/4. Cardr0r1r2r3GaindualOOD lettanative60105565 +5 jitter 2/4 6/6 dgmnative50505050+00/4 0/6 qwen-plus55155+00/4 1/6 Table 47: NL2SQL teacher contrast, not office-v1. Held-out Success %, mean of seeds 42–44 (cells identical across seeds). Self has no successful SQL in traces. EvolverTeacher tracesSelf traces schemamemory100%0% promptmemory0%0% noevolve0%0% 50 Table 48: Archived ticket-gym officework heuristic B0, not office-v1. Held-out Success %. Identical across seeds 42–44. Other B0: noevolve, promptmemory, successmemory, expel, awm. EvolverSuccessGainin-distOOD reflexion (stand-in)75%+55p15/150/5 other B020%+0p 4/15 (tc only) 0/5 M0r1r2r3 0 20 40 60 80 100 Held-out Success (%) (a) Expansion-pack B0 reflexion other B0 mean tc triage rel pm di f qa OOD-ad OOD-cb 0 25 50 75 100 125 Family Success (%) (b) Families (not office 80%) reflexion other B0 Figure 20: Archived ticket-gym expansion pack, not office-v1. Not ranked vs. Table 39. A.18.16 Cross-environment summary Table 49 consolidates the frozen ticket-gym, inbox, NL2SQL, and officework cards. Rows are actor boards. Each row uses the 40/10/20 pack, R=3, the validation gate, and artifact binding. Seed counts are the last column: the 3B wrapper row is one seed, and the other rows are three. No single row is ranked above another across budget classes or environments. Table 49: Archived cards across ticket-gym, inbox, NL2SQL, and office work. Not an office-v1 rank- ing. “—” = family not present in that environment. EnvBoardBudget Succ.GainIn-dist familyOOD family Seeds ticket-gym B0 reflexion (heur.)B070% +20 vs 0.50dual 4/40/63 3B reflexionllm wrapB055% +10 vs 0.45dual 0/44/61 inbox B0 reflexion (heur.)B070% +30 vs 0.40 reply 3/3, thread 3/30/63 B0 other (heur.)B040%+00/30/63 NL2SQL Teacher schemamem—100% +100 vs 0OOD 4/4—3 (contrast)Self (all evolvers)—0%—0/4—3 officework B0 reflexion (heur.)B075% +55 vs 0.20in-dist 15/150/53 51