Paper deep dive
Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds
Yuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, Yangqiu Song
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/5/2026, 3:37:42 AM
Summary
This paper evaluates self-evolving agent skills by controlling feedback conditions (Normal, Fail-only, Success-only) across multiple models and benchmarks. It finds that evolution is sparse and heavily dependent on including failed trajectories (Normal or Fail-only views), with Success-only never selected. Validation-based selection improves test, robustness, and transfer performance in most settings, challenging the notion of steady improvement. Test-time scaling controls (Parallel Sampling, Sequential Refinement) generally fail to match the gains of persistent skill evolution.
Entities (17)
Relation Signals (13)
Normal → includes → successes and failures
confidence 95% · successes and failures (Normal)
Fail-only → includes → Failed Trajectories
confidence 95% · failures only
Success-only → includes → successful trajectories
confidence 95% · successes only
Success-only → selectedbyvalidation → 0 settings
confidence 95% · Success-only is never selected in the primary study
Sequential Refinement → failstorecover → gains
confidence 90% · Sequential Refinement recovers neither gain.
Normal → selectedbyvalidation → 9 settings
confidence 90% · Normal is selected in nine settings and Fail-only in two
Fail-only → selectedbyvalidation → 2 settings
confidence 90% · Normal is selected in nine settings and Fail-only in two
SpreadsheetBench → shows → largest gains
confidence 90% · SpreadsheetBench shows the largest and most consistent gains
Self-evolving skill systems → uses → execution feedback
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model. Yet it remains unclear when further evolution helps, how successful and failed trajectories shape revision, and whether extra test-time computation can recover the same gains. To address these questions, we present a controlled evaluation framework across five benchmarks and three models. Our primary study contains 42 feedback runs across 14 supported model-benchmark settings. Within each setting, we hold the executor and optimizer configuration, revision procedure, validation rule, and round budget fixed, while varying only the feedback shown to the optimizer: successes and failures (Normal), failures only, or successes only. Evolution is sparse: only 55 of 388 candidates establish byte-distinct validation bests. Validation-based selection chooses an evolved skill in 11 of 14 settings, nine of which improve released-test performance. All 11 selections come from feedback conditions that include failed trajectories, although the relative ranking of Normal and Fail-only varies across settings. Validation and downstream evaluations on test, robustness, and transfer sometimes favor different feedback views. A broader SearchQA analysis covering eight models shows similarly sparse, feedback-dependent dynamics. In the GPT-5.5 test-time-scaling controls, oracle Parallel Sampling comes within 0.43 points of the evolved SearchQA skill but remains 30.96 points behind on SpreadsheetBench; Sequential Refinement recovers neither gain. Overall, persistent skill self-evolution is better understood as sparse, validation-filtered search with model- and benchmark-dependent returns, rather than steady improvement from additional rounds. The implementation is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.02636v1
- Canonical: https://arxiv.org/abs/2608.02636v1
Trouble viewing inline? Open PDF directly →
Full Text
111,786 characters extracted from source content.
Expand or collapse full text
Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds Yuxuan Liu1, Zhaochen Su1, Yuhao Zhang3, Jiahe Guo2, Zhongwei Xie1, Huihao Jing1, Lingyun Xie1, Qing Zong1, Yauwai Yim1, Zhixiong Zhang4, Haoran Li1 , Yangqiu Song1 Abstract Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model. Yet it remains unclear when further evolution helps, how successful and failed trajectories shape revision, and whether extra test-time computation can recover the same gains. To address these questions, we present a controlled evaluation framework across five benchmarks and three models. Our primary study contains 42 feedback runs across 14 supported model–benchmark settings. Within each setting, we hold the executor and optimizer configuration, revision procedure, validation rule, and round budget fixed, while varying only the feedback shown to the optimizer: successes and failures (Normal), failures only, or successes only. Evolution is sparse: only 55 of 388 candidates establish byte-distinct validation bests. Validation-based selection chooses an evolved skill in 11 of 14 settings, nine of which improve released-test performance. All 11 selections come from feedback conditions that include failed trajectories, although the relative ranking of Normal and Fail-only varies across settings. Validation and downstream evaluations on test, robustness, and transfer sometimes favor different feedback views. A broader SearchQA analysis covering eight models shows similarly sparse, feedback-dependent dynamics. In the GPT-5.5 test-time-scaling controls, oracle Parallel Sampling comes within 0.43 points of the evolved SearchQA skill but remains 30.96 points behind on SpreadsheetBench; Sequential Refinement recovers neither gain. Overall, persistent skill self-evolution is better understood as sparse, validation-filtered search with model- and benchmark-dependent returns, rather than steady improvement from additional rounds. The implementation is available at https://github.com/HKUST-KnowComp/rethinkskill. Introduction Agent skills encode reusable behavior outside the underlying model and guide later executions without updating model parameters. Self-evolving skill systems extend this idea by using execution traces and evaluation feedback to generate, repair, and retain new skill versions. Unlike extra inference for a single task instance, a revised skill persists across future executions. Prior work has reported substantial gains over skill-free baselines and, in some settings, human-authored skills (Li and Hu 2026; Liu et al. 2026e; Yang et al. 2026c). A central design choice is which execution evidence should shape the next skill version. Success-gated methods retain useful discoveries from successful trajectories (Liu et al. 2026a). Diagnosis-oriented methods revise skills based on observed failures (Liu et al. 2026e, b). Mixed-trajectory methods use both sources (Yang et al. 2026c; Ni et al. 2026; Yang et al. 2026a). These choices reflect different intuitions: successes show behavior worth preserving, failures expose skill defects, and mixed feedback provides broader but potentially conflicting evidence. However, existing results do not isolate the effect of feedback composition. Each system is usually evaluated under its own revision and selection procedure, leaving three questions unresolved. First, before-and-after scores hide accepted revisions, rejected candidates, rollbacks, and stopping decisions. They therefore do not show when additional rounds help or when the search has stopped making progress. Second, comparisons across systems cannot determine how successful trajectories, failed trajectories, or both affect evolution under the same procedure and across different models. Third, it remains unclear whether additional per-instance inference can achieve similar gains without changing the persistent skill. Run-to-run variation and verifier choice further complicate these comparisons because score changes do not always reflect lasting skill changes. To address these questions, we present a controlled evaluation framework for feedback-conditioned skill evolution. Our primary study contains 42 feedback runs across 14 supported model–benchmark settings involving GPT-5.5, Gemini 3.1 Pro, DeepSeek V4-Pro, and five benchmarks. Within each setting, Normal, Fail-only, and Success-only start from the same parent skill. The task executor and revision optimizer use the same model configuration, while the revision procedure, validation rule, and round budget remain fixed. At each round, the revision operator proposes a candidate skill. A candidate becomes the next-round skill when validation does not decrease; otherwise, the current skill is retained. Only strict improvement updates the best checkpoint. For artifact-level analysis, we count a new validation best only when the candidate is also byte-distinct from the incoming skill. We record all candidates, acceptance and rollback decisions, best-so-far updates, and skill identities, exposing each view’s full search trajectory. We retain the validation-best skill from each run and select one skill for each model–benchmark setting using validation alone; test and diagnostic results do not affect this selection. After evolution, we evaluate the frozen skills on released test, same-task robustness, and transfer. For the five GPT-5.5 settings, we also examine test-time scaling, repeated-deployment variability, and verifier sensitivity. A broader SearchQA analysis expands the model coverage to eight models in total. Figure 1 summarizes the framework. The results show that skill evolution is sparse and depends on both the model and benchmark. Only 55 of 388 candidates establish byte-distinct validation bests. The observed trajectories range from late discovery to early saturation and complete stagnation. Validation selects an evolved skill in 11 of 14 model–benchmark settings, nine of which improve released-test performance. Nine also improve robustness and nine improve transfer, but only seven improve both. All 11 selected evolved skills come from Normal or Fail-only: Normal is selected in nine settings and Fail-only in two, while Success-only is never selected in the primary study. The broader SearchQA analysis shows similar sparse, feedback-dependent dynamics across eight models. Test-time scaling is also uneven: oracle Parallel Sampling nearly recovers the SearchQA gain but remains far behind on SpreadsheetBench, while Sequential Refinement reproduces neither. Together, these results characterize persistent skill self-evolution as validation-filtered search rather than steady improvement from additional rounds. This paper makes three contributions: • Cross-model evolution dynamics. We trace 388 candidates across 42 feedback runs and 14 model–benchmark settings, revealing late improvement, early saturation, rejected regressions, and stagnation. A broader eight-model SearchQA analysis shows similarly sparse, feedback-dependent dynamics across model families. • Controlled feedback comparison. We compare Normal, Fail-only, and Success-only while holding the evolution procedure fixed. All 11 evolved selections in the primary study include failed trajectories, while artifact-level case studies show how different feedback views produce different retained guidance. • Evolution, generalization, and test-time computation. We evaluate whether selected skills improve test, robustness, and transfer, and compare the GPT-5.5 skills with parallel and sequential test-time scaling. The results separate persistent skill gains from per-instance inference and validation-specific improvement. Related Work Self-Evolving Agent Skills Self-evolving skill systems make different choices about which execution evidence becomes persistent skill change (Jiang et al. 2026a; Zhang et al. 2026a; Tian et al. 2026; Xia et al. 2026; Shen, Li, and Zhang 2026). Success-gated systems such as SkillsVote admit only successful reusable discoveries to evidence-gated skill-library updates (Liu et al. 2026a), whereas diagnosis-oriented systems such as SkillRevise and SkillForge identify execution failures, localize skill defects, and revise the affected skills (Liu et al. 2026e, b). Mixed-trajectory systems, including SkillOpt, Trace2Skill, SkillGen, and OptSkills, consolidate successful procedures and failure-derived lessons into subsequent skill revisions (Yang et al. 2026c; Ni et al. 2026; Ma et al. 2026; Yang et al. 2026b). These systems evaluate their respective native feedback policies as a whole, rather than holding the update and validation procedure fixed while comparing successful trajectories only, failed trajectories only, and both together across complete multi-round evolution runs. Consequently, the comparative roles of successful and failed execution evidence in adaptive persistent-skill search remain unclear. Agent skill evaluation and benchmarks. Agent-skill benchmarks evaluate marginal utility, continual learning, and process quality at several levels. SkillsBench and SWE-Skills-Bench use paired conditions to isolate skill utility, while SkillLearnBench evaluates skill quality, execution trajectories, and task outcomes (Li et al. 2026a; Han et al. 2026; Zhong et al. 2026). OpenClawBench complements outcome-based evaluation by aligning task-oracle outcomes with localized process-anomaly evidence in real-world execution trajectories (Liu et al. 2026d). Other work analyzes experience extraction and consumption, process-level skill use, or paired behavioral influence (Huang et al. 2026; Zhu et al. 2026; Zhou et al. 2026). SkillsWild evaluates skill retrieval and use under realistic library conditions, while SkillJuror measures behavioral changes under controlled skill organizations (Liu et al. 2026c; Chen et al. 2026). SEA-Eval studies sequential cross-task evolution, and metric co-evolution examines improvement when the evaluator itself changes (Jiang et al. 2026b; Zhang et al. 2026c). These benchmarks characterize skill utility and evaluation scope, but they do not isolate how alternative feedback sources shape a fixed-model skill-evolution procedure over multiple rounds. Feedback exposure Success-only SkillsVote (Liu et al. 2026a) Failure-driven SkillRevise (Liu et al. 2026e) SkillForge (Liu et al. 2026b) EvoSkill (Alzubi et al. 2026) MemSkill (Zhang et al. 2026b) SkillAdaptor (Yu et al. 2026) Normal SkillOpt (Yang et al. 2026c) Trace2Skill (Ni et al. 2026) GeoSkill (Yang et al. 2026a) SkillGen (Ma et al. 2026) OptSkills (Yang et al. 2026b) Table 1: Representative self-evolving skill systems grouped by feedback evidence. Figure 1: Feedback-conditioned skill evolution and controlled evaluation. Each round executes the current skill, constructs a feedback view, and proposes a candidate for validation. The validation gate determines the next-round skill and updates the best checkpoint only after strict improvement. The best checkpoint is then frozen for controlled evaluation. Framework Overview Self-evolution process. Let M denote the fixed underlying agent model and sr(c)s_r^(c) the persistent skill at evolution round r under feedback condition c. Executing the round tasks with M and sr(c)s_r^(c) produces trajectories τr(c) _r^(c), while fr(c)f_r^(c) denotes the feedback exposed for revision. The same model instantiates the revision operator O, which proposes a candidate from the current skill and condition-specific evidence: s^r+1(c)=O(sr(c),τr(c),fr(c)). s_r+1^(c)=O(s_r^(c), _r^(c),f_r^(c)). (1) Here, s^r+1(c) s_r+1^(c) is a candidate rather than the next-round skill. If the candidate is not rejected, sr+1(c)←s^r+1(c).s_r+1^(c)← s_r+1^(c). (2) Otherwise, sr+1(c)←sr(c).s_r+1^(c)← s_r^(c). (3) We separately track the validation-best skill sbest(c)s_best^(c), initialized as s0(c)s_0^(c). For artifact-level analysis, a new best requires both a strict validation improvement and a byte-distinct candidate. Score changes from byte-identical reruns are treated as execution variability and do not update the reported best round. After the final round, each view returns its best validation skill. For each benchmark, we choose among these three skills using validation before evaluating released test, robustness, and transfer. Framework components. Figure 1 instantiates this process through three components. Execution and feedback construction. The execution component applies the current skill to the round tasks and records the resulting trajectories, which the benchmark verifier labels as successful or failed. The feedback component then constructs one of three views: Normal includes both successful and failed trajectories, Fail-only includes failed trajectories, and Success-only includes successful trajectories. Validation-gated skill revision. The revision component generates a candidate through O. The validation gate advances a candidate whose validation score does not decrease and carries the current skill forward after a regression. A separate checkpoint updates only after a strict validation improvement and is returned after evolution. Controlled evaluation. After evolution, the resulting skills are frozen and passed through a common evaluation path. For a task x and skill s, execution and verification are o=A(x,s,z),y=V(x,o),o=A(x,s,z), y=V(x,o), (4) where z denotes the controlled execution conditions. Changes to s, z, x, and V respectively organize the parent–evolved comparison, test-time scaling and repeated execution, task variation, and verifier replacement. Experiments Benchmark Condition GPT-5.5 Gemini 3.1 Pro DeepSeek V4-Pro Validation Test R T Validation Test R T Validation Test R T SearchQA Parent (r=0r=0) 80.0 (–) 75.6 (–) 75.9 (–) 72.1 (–) 79.0 (–) 76.9 (–) 76.3 (–) 73.7 (–) 74.5 (–) 73.0 (–) 72.4 (–) 72.6 (–) Normal 82.0 (+2.0) 77.9 (+2.3) 76.1 (+0.2) 77.6 (+5.5) 82.0 (+3.0) 76.6 (-0.4) 76.0 (-0.3) 75.2 (+1.5) 76.5 (+2.0) 73.9 (+0.9) 72.2 (-0.2) 74.4 (+1.8) Fail-only 81.5 (+1.5) 77.3 (+1.7) 74.9 (-1.0) 76.2 (+4.2) 81.5 (+2.5) 77.6 (+0.6) 76.0 (-0.3) 74.6 (+0.8) 76.0 (+1.5) 74.2 (+1.2) 72.6 (+0.2) 74.1 (+1.5) Success-only 81.5 (+1.5) 77.5 (+1.9) 77.8 (+1.9) 75.2 (+3.2) 81.0 (+2.0) 77.8 (+0.9) 75.3 (-1.0) 74.9 (+1.2) 74.5 (+0.0) 72.0 (-1.0) 72.5 (+0.1) 73.4 (+0.8) OfficeQA Parent (r=0r=0) 77.6 (–) 62.8 (–) 59.2 (–) 59.4 (–) 77.6 (–) 65.5 (–) 68.8 (–) 73.1 (–) 73.5 (–) 56.8 (–) 52.5 (–) 58.6 (–) Normal 87.8 (+10.2) 69.6 (+6.8) 70.5 (+11.3) 71.1 (+11.7) 77.6 (+0.0) 65.5 (+0.0) 69.9 (+1.1) 71.0 (-2.1) 73.5 (+0.0) 54.7 (-2.0) 48.4 (-4.2) 52.7 (-5.9) Fail-only 85.7 (+8.2) 69.6 (+6.8) 67.2 (+8.0) 67.7 (+8.4) 77.6 (+0.0) 64.9 (-0.7) 69.1 (+0.2) 70.8 (-2.2) 73.5 (+0.0) 52.7 (-4.1) 50.4 (-2.2) 53.7 (-4.9) Success-only 85.7 (+8.2) 66.9 (+4.1) 64.1 (+4.9) 63.9 (+4.5) 77.6 (+0.0) 67.6 (+2.0) 68.1 (-0.8) 67.9 (-5.1) 73.5 (+0.0) 56.8 (+0.0) 49.1 (-3.5) 55.3 (-3.3) SpreadsheetBench Parent (r=0r=0) 41.0 (–) 50.2 (–) 44.5 (–) 49.8 (–) 38.5 (–) 42.0 (–) 41.2 (–) 51.3 (–) 38.5 (–) 39.9 (–) 39.9 (–) 44.4 (–) Normal 82.1 (+41.0) 82.2 (+32.0) 79.8 (+35.3) 77.6 (+27.8) 82.1 (+43.6) 79.7 (+37.7) 85.1 (+43.8) 76.8 (+25.5) 76.9 (+38.5) 68.7 (+28.8) 72.2 (+32.3) 65.5 (+21.1) Fail-only 82.1 (+41.0) 85.8 (+35.6) 80.0 (+35.5) 77.9 (+28.1) 76.9 (+38.5) 81.5 (+39.5) 83.7 (+42.5) 78.5 (+27.1) 69.2 (+30.8) 64.8 (+24.9) 66.4 (+26.5) 65.5 (+21.1) Success-only 41.0 (+0.0) 47.7 (-2.5) 41.6 (-2.9) 49.9 (+0.1) 48.7 (+10.3) 50.2 (+8.2) 48.4 (+7.1) 54.7 (+3.3) 48.7 (+10.3) 39.1 (-0.7) 41.0 (+1.1) 46.0 (+1.6) LiveMath Parent (r=0r=0) 51.4 (–) 49.1 (–) 49.6 (–) 62.5 (–) 42.9 (–) 41.5 (–) 47.8 (–) 62.5 (–) 40.0 (–) 10.4 (–) 12.4 (–) 16.7 (–) Normal 51.4 (+0.0) 40.6 (-8.5) 47.9 (-1.7) 54.2 (-8.3) 74.3 (+31.4) 64.2 (+22.6) 66.4 (+18.5) 79.2 (+16.7) 54.3 (+14.3) 20.8 (+10.4) 20.6 (+8.2) 50.0 (+33.3) Fail-only 57.1 (+5.7) 42.5 (-6.6) 57.8 (+8.3) 45.8 (-16.7) 74.3 (+31.4) 56.6 (+15.1) 63.0 (+15.1) 62.5 (+0.0) 48.6 (+8.6) 28.3 (+17.9) 20.0 (+7.6) 41.7 (+25.0) Success-only 51.4 (+0.0) 43.4 (-5.7) 48.1 (-1.4) 45.8 (-16.7) 62.9 (+20.0) 61.3 (+19.8) 61.9 (+14.0) 66.7 (+4.2) 40.0 (+0.0) 23.6 (+13.2) 14.4 (+2.0) 25.0 (+8.3) DocVQA Parent (r=0r=0) 96.2 (–) 92.0 (–) 87.9 (–) 93.7 (–) 92.5 (–) 95.5 (–) 94.8 (–) 96.9 (–) N/A N/A N/A N/A Normal 96.2 (+0.0) 91.7 (-0.3) 75.4 (-12.4) 91.2 (-2.5) 94.3 (+1.9) 96.0 (+0.5) 95.1 (+0.3) 96.6 (-0.3) N/A N/A N/A N/A Fail-only 96.2 (+0.0) 91.7 (-0.3) 76.8 (-11.1) 89.5 (-4.3) 94.3 (+1.9) 95.2 (-0.3) 95.0 (+0.2) 97.0 (+0.1) N/A N/A N/A N/A Success-only 96.2 (+0.0) 91.2 (-0.8) 75.3 (-12.6) 91.0 (-2.7) 92.5 (+0.0) 95.7 (+0.3) 94.8 (+0.0) 96.8 (-0.1) N/A N/A N/A N/A Table 2: Feedback-conditioned results across models and benchmarks. Cells report score (change from the corresponding model–benchmark parent, p). Blue shading and boldface mark validation-selected views; teal, orange, and gray indicate positive, negative, and zero, parent-identical, or unavailable comparisons; underlining marks within-setting metric bests. DeepSeek–DocVQA is unsupported. Scores use benchmark-specific evaluators; R/T are equal-weight averages over available probes. Experimental Setup Benchmarks and evolution protocol. We evaluate three underlying models across five benchmarks. GPT-5.5 and Gemini 3.1 Pro are evaluated on SearchQA, OfficeQA, SpreadsheetBench, LiveMath, and DocVQA. We exclude DeepSeek–DocVQA because its endpoint does not accept the benchmark’s native page images, and adding an OCR stage would alter the task interface. This yields 14 model–benchmark settings and 42 matched feedback runs. ALFWorld is reported in Appendix A7. For each model–benchmark setting, Normal, Fail-only, and Success-only start from the same parent skill and use the same executor and optimizer configuration, revision procedure, validation rule, and ten-round budget. They differ only in the trajectories shown to the optimizer: both successful and failed trajectories (Normal), failed trajectories only, or successful trajectories only. Each round executes 40 training trajectories (36 for LiveMath); a nonempty feedback view triggers one optimizer call proposing at most four minimal, task-general edits, followed by full-split validation. A candidate becomes the next-round skill when validation does not decrease; otherwise, the current skill is retained. Only a strict validation improvement updates the best checkpoint. Runs stop after ten rounds, five consecutive regressions or no-ops, or an empty feedback pool; across rounds, only the incumbent skill carries forward. Appendices A1, A2, A3, and A9 provide the benchmark definitions, common protocol, execution controls, and per-run records. Figure 2: Evolution events across 14 model–benchmark settings. Gray dots mark other evaluated candidates, blue dots mark byte-distinct validation bests, stars mark the best byte-distinct round within each feedback view, and black rings mark benchmark-level selections among the three views. Rows denote models, columns denote benchmarks, and N, F, and S denote Normal, Fail-only, and Success-only. DeepSeek–DocVQA is unavailable because its endpoint does not accept native page images. Robustness and transfer probes. We assess same-task robustness with three tiers: R1 changes surface form or input representation while preserving the task and gold answer, R2 adds irrelevant context or distractors, and R3 changes the output or interface contract. Transfer also has three tiers: T1 uses a new official task from the same source artifact, T2 uses a new source within a task subtype represented in the original evaluation panel, and T3 uses a new source with a subtype, difficulty, or distribution shift. Each probe score averages three deployments on a fixed panel. R and T are equal-weight macro-averages over the available tiers. Appendix A6 lists the benchmark-specific probes and eligible sample sizes. Test-time-scaling controls. Prior work studies parallel and sequential test-time scaling for agents (Zhu et al. 2025; Kim et al. 2026; Li et al. 2026b), including comparisons with harness evolution (Wang et al. 2026). Across five GPT-5.5 benchmarks, we compare one-call deployment of the validation-selected skill with two controls initialized from the frozen parent skill for each task. At budget K, which includes the parent attempt, Parallel Sampling reports oracle any-success over that attempt and K−1K-1 independent attempts, while Sequential Refinement conditions each additional attempt only on the task and preceding response and reports the final response. The controls share the executor, tool interface, test pool, verifier, and call budget; curves compare items receiving the full budget. SearchQA and SpreadsheetBench are the primary contrasts. Appendices A25, A26, A27, and A2 provide complete definitions, allocations, scores, and the call accounting for the selected SearchQA run. Main Results All selected evolved skills come from feedback containing failed trajectories; Success-only is never selected. Across the 14 model–benchmark settings in Table 2, validation selects 11 evolved skills. Normal accounts for nine selections and Fail-only for two, while the parent is retained in three settings: Gemini–OfficeQA, DeepSeek–OfficeQA, and GPT-5.5–DocVQA. On the released test, nine of the 11 selected evolved skills improve over their parents, with positive gains ranging from 0.50.5 to 37.737.7 points. SpreadsheetBench shows the largest and most consistent gains: 35.635.6 points for GPT-5.5, 37.737.7 for Gemini, and 28.828.8 for DeepSeek. LiveMath is more variable: GPT-5.5 declines by 6.66.6 points, whereas Gemini and DeepSeek improve by 22.622.6 and 10.410.4 points, respectively. SearchQA changes are smaller: +2.3+2.3, −0.4-0.4, and +0.9+0.9 points for GPT-5.5, Gemini, and DeepSeek, respectively. OfficeQA selects an evolved skill only for GPT-5.5 (+6.8+6.8 points), with Gemini and DeepSeek retaining their parents; for DocVQA, Gemini selects Normal (+0.5+0.5), GPT-5.5 retains its parent, and DeepSeek is unavailable. Accepted revisions are sparse, and selected rounds vary across models, benchmarks, and feedback views. Figure 2 summarizes 388 evaluated candidates across 42 observed feedback trajectories, of which 55 establish byte-distinct validation bests. SearchQA selects an evolved skill for all three models, at round 9 for GPT-5.5 and round 1 for both Gemini and DeepSeek. SpreadsheetBench also selects an evolved skill for all three models, at rounds 6, 7, and 3, respectively. OfficeQA selects a round-9 GPT-5.5 skill but retains the parent for Gemini and DeepSeek. DocVQA retains the GPT-5.5 parent and selects a round-4 Gemini skill, while DeepSeek is unavailable. LiveMath selects round-3, round-9, and round-8 skills for GPT-5.5, Gemini, and DeepSeek, respectively. Across all 14 model–benchmark settings, 11 select an evolved skill and three retain the parent; five evolved selections occur in rounds 1–4 and six in rounds 6–9. Across the 11 settings with a validation-selected evolved skill, robustness and transfer each improve in nine, and seven improve both. Complete test, robustness, and transfer results are available for all 14 model–benchmark settings. Seven of the 11 evolved selections improve both robustness and transfer. SpreadsheetBench is positive on test, robustness, and transfer for all three models. SearchQA improves transfer for all three models, while robustness changes by +0.2+0.2, −0.3-0.3, and −0.2-0.2 points for GPT-5.5, Gemini, and DeepSeek. LiveMath improves all three measures for Gemini and DeepSeek; GPT-5.5 instead gains 8.38.3 points in robustness while losing 6.66.6 on test and 16.716.7 on transfer. Gemini–DocVQA gains 0.50.5 points on test and 0.30.3 on robustness but loses 0.30.3 on transfer. GPT-5.5–OfficeQA improves all three measures by 6.86.8, 11.311.3, and 11.711.7 points. Analysis Cross-Model Evidence on SearchQA SearchQA provides the broadest cross-model coverage. Appendix A12 extends the comparison beyond GPT-5.5 to seven models with complete released-test results: Claude Opus, Kimi, Gemini, DeepSeek, GLM, Grok, and Qwen3.5-Plus. Across these models, 191 of 210 candidates change the incoming skill, but only 29 establish byte-distinct validation bests. This gap separates revision activity from retained progress: modification is common, whereas validation-confirmed improvement is sparse. Selected rounds span 1–7, compared with round 9 for GPT-5.5, so selection timing is not uniform across models. Six of the seven validation-selected skills improve released-test performance by 0.86–15.07 points, while Gemini decreases by 0.360.36 points. Test-best and validation-selected views differ in six settings, indicating model-dependent ranking shifts between validation and test. Qwen3.5-Plus also improves robustness by 2.192.19 points and transfer by 2.552.55 points. Across all eight SearchQA models, seven validation-selected skills improve released-test performance, while the selected view, round, and validation-to-test ranking differ by model. Generalization of Evolved Skills Cross-model evidence establishes breadth across model families. The robustness and transfer results in Table 2 and Appendix A15 test whether the retained behavior extends beyond the released-test distribution. They capture different consequences of a selected skill rather than interchangeable forms of improvement. SpreadsheetBench improves released-test, robustness, and transfer performance for all three models, whereas SearchQA improves transfer in all three models with complete diagnostics but leaves robustness at or below the parent. Gemini–DocVQA shows small released-test and robustness gains but a small transfer decrease. LiveMath is model dependent: GPT-5.5 improves robustness but loses released-test and transfer performance, whereas DeepSeek and Gemini improve all three; Gemini gains 22.622.6 points on released test, 18.518.5 on robustness, and 16.716.7 on transfer. Generalization therefore depends on the interaction among the benchmark, model, and retained skill, not on validation improvement alone. The GPT-5.5 probe-level results in Appendix A14 further distinguish local robustness from transfer. SpreadsheetBench remains positive across R1–R3 and T2, with the transfer gain narrowing to +2.9+2.9 on T3. LiveMath also improves across all robustness probes, but its transfer differences are 0.00.0 on T2 and −33.3-33.3 on T3. The four- and two-item transfer panels identify probe-level effects, while the 106-item released test provides broader evidence for the GPT-5.5–LiveMath validation-to-test reversal. Together, these probes characterize benchmark-specific generalization across the evaluated robustness and transfer conditions. Effects of Feedback Composition Tables 2 and 3 show that no feedback view has a fixed advantage across model–benchmark settings. Fail-only produces a byte-distinct validation improvement in 11 of 14 settings, compared with 10 for Normal and six for Success-only. Normal is nevertheless selected most often: nine settings, versus two for Fail-only and none for Success-only. Normal and Fail-only each improve released-test performance in nine settings. Fail-only improves robustness and transfer in nine of 14 settings, compared with eight and nine for Normal. Success-only improves robustness and transfer in five and six settings, respectively, and improves fewer settings on every reported metric. These view-specific counts exclude parent-identical branches, and a setting may contribute to more than one row. View Selected Val.↑ Test↑ Robust.↑ Transfer↑ Normal 9 10 9 8 9 Fail-only 2 11 9 9 9 Success-only 0 6 5 5 6 Table 3: Counts of model–benchmark settings improved or selected by each feedback view. The artifact audits in Appendices A17, A19, and A20 clarify the contrast between Normal and Fail-only. On OfficeQA, failures expose unsupported first-turn answers, while successful traces give Normal a reference for what should be preserved. The resulting skill combines a retrieve–read–compute rule with answer-format constraints and exceeds Fail-only on validation, robustness, and transfer. On SpreadsheetBench, Fail-only focuses on a verifier-visible defect: formula strings leave required cells empty. Its explicit write–reopen–check procedure yields the strongest test, robustness, and transfer results. Normal can use successes to broaden a repair, whereas Fail-only targets the defect. Appendix A21 explains Success-only’s rarity. Successes show what worked but provide no direct contrast for identifying what must be corrected. When only a few successes are available, the optimizer can mistake incidental commonalities for task-level rules. In DeepSeek–LiveMath, the round-3 candidate extrapolates from eight successful traces, adds a “strongest/equivalence” heuristic, and changes the required output from an option label to the full option text. Validation falls from 40.0 to 11.4, and the arm ultimately retains the parent. Opus and Qwen3.5-Plus are the two Success-only selections among the eight SearchQA models. Opus’s successful traces support a repeated task-wide specification—identify the clue referent and return a short canonical answer—and the round-7 skill improves validation from 77.0 to 79.0 and released-test performance by 3.79 points. Qwen3.5-Plus selects its round-3 skill, improving validation from 73.0 to 76.5 and released-test performance by 2.93 points. Together, these cases suggest that Success-only is most useful when positive trajectories expose a stable shared specification, whereas negative contrast helps distinguish task-level rules from incidental patterns. Effects of Additional Evolution Rounds Across the 42 primary runs summarized in Appendix A1, additional rounds expand the search horizon, but their average yield falls after the early stage. Thirty-eight of the 55 byte-distinct validation bests occur in rounds 1–4. The remaining 17 arise from 221 candidates evaluated after round 4. Late search remains consequential: six of the 11 selected evolved skills first appear in rounds 6–9, compared with five in rounds 1–4. A four-round budget would capture most new-best events but miss most final evolved selections. The late gains do not arise from steady improvement. SearchQA Normal first reaches 81.0 at round 1, then evaluates seven candidates without exceeding it before reaching 82.0 at round 9. OfficeQA Normal improves at rounds 1, 8, and 9, with regressions between them. SpreadsheetBench shows saturation: Normal peaks at round 3 and all five later candidates score lower, while Fail-only reaches its final best at round 6 and then produces four lower-scoring candidates. DocVQA never exceeds its 96.2 parent across 23 candidates. These trajectories characterize evolution as validation-filtered search rather than monotonic refinement. Additional rounds can uncover a late skill after many rejected candidates, but after saturation they add search cost without retained improvement. Rollback preserves the best validation result; it does not create progress or guarantee test improvement. LiveMath makes this distinction explicit: its round-3 Fail-only skill is the only validation improvement, yet it reduces released-test performance by 6.6 points. Round budgets should therefore be evaluated by the timing and frequency of new bests, not by the number of revisions alone. These diminishing late-round returns motivate a complementary question: whether additional per-instance inference can recover the gains produced by persistent skill evolution. Figure 3: GPT-5.5 test-time scaling on benchmarks with byte-distinct evolved skills. (a) One-call parent and evolved-skill performance compared with maximum-budget Parallel Sampling and Sequential Refinement. (b–e) Common-support budget curves; horizontal references show one-call evolved-skill performance. Parallel reports oracle any-success, and Sequential reports the final attempt. Self-Evolution versus Test-Time Scaling Figure 3 shows that oracle Parallel Sampling nearly matches the evolved skill on SearchQA but recovers little of its advantage on SpreadsheetBench. On SearchQA, the frozen parent execution scores 75.64, the evolved skill 77.93, Parallel 77.50, and Sequential 75.79; Parallel is therefore only 0.43 points below the evolved skill. On SpreadsheetBench, the corresponding scores are 50.53, 85.77, 54.80, and 45.20, leaving a 30.96-point gap between evolution and Parallel. This contrast separates response diversity from persistent policy change. The SearchQA skill mainly adds answer-form guidance—answer type, canonical naming, and concise output—so independent samples can explore alternatives and oracle selection recovers most of the modest gain. SpreadsheetBench instead requires a multi-step workflow: inspect the workbook, execute a self-contained script, materialize values, save the output, and verify target cells. The 30.96-point gap indicates that additional parent-skill samples rarely reproduce this complete procedure. Sequential Refinement also remains near the parent on SearchQA and OfficeQA and declines on SpreadsheetBench, showing that conditioning on a preceding response is not itself corrective feedback. OfficeQA provides a second sampling-recoverable case: maximum-budget Parallel matches the evolved score at 69.59. On LiveMath, the validation-selected skill scores 42.45 on test, the parent 49.06, and Parallel 68.87; this comparison combines a sampling gain with a validation-to-test reversal. Across these controls, test-time scaling can exploit response diversity available under the parent skill, while persistent skill improvement reflects retained changes across task instances. Oracle score parity and persistent skill improvement therefore measure different outcomes. Conclusion Persistent skill self-evolution behaves as sparse search rather than steady improvement. Across 42 feedback runs in 14 model–benchmark settings, only 55 of 388 candidates establish byte-distinct validation bests, and validation selects an evolved skill in 11 settings. Selected skills appear in early and late rounds, while other runs saturate or retain the parent. Normal accounts for nine selected skills and Fail-only for two, so all 11 evolved selections use failed trajectories; Success-only is never selected in the primary study, while the leading failure-containing view varies across models and benchmarks. A broader SearchQA analysis across eight models shows similarly sparse, feedback-dependent dynamics. Selected skills improve released-test performance, robustness, and transfer in nine settings each, with seven improving both robustness and transfer. In the GPT-5.5 test-time-scaling controls, oracle Parallel Sampling comes within 0.43 points of the evolved SearchQA skill but remains 30.96 points behind on SpreadsheetBench, while Sequential Refinement reproduces neither gain. These findings characterize self-evolution as feedback-conditioned, validation-filtered search over persistent skills and show why endpoint scores alone are insufficient. Evaluations should report search trajectories, skill identity, downstream generalization, and explicit test-time-compute controls. Limitations Although the primary study covers 14 model–benchmark settings across five heterogeneous benchmarks, its coverage of agent-skill settings remains incomplete. In particular, we do not evaluate on dedicated skill benchmarks such as SkillsBench and SkillLearnBench (Li et al. 2026a; Zhong et al. 2026), which cover broader skill-dependent tasks and continual skill generation. Future work can apply our matched feedback-isolation and validation-selection protocol to these benchmarks to test whether the observed feedback rankings and sparse search dynamics generalize beyond the present task suite. References Alzubi et al. (2026) Alzubi, S.; Provenzano, N.; Bingham, J.; Chen, W.; and Vu, T. 2026. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766. Chen et al. (2026) Chen, Z.; Guo, Z.; Huang, B.; Lu, B.; Lin, J.; Zhou, Y.; and Zhang, W. 2026. SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior. arXiv:2606.11543. Dunn et al. (2017) Dunn, M.; Sagun, L.; Higgins, M.; Guney, V. U.; Cirik, V.; and Cho, K. 2017. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine. arXiv:1704.05179. Han et al. (2026) Han, T.; Zhang, Y.; Song, W.; Fang, C.; Chen, Z.; Sun, Y.; and Hu, L. 2026. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? arXiv:2603.15401. He et al. (2026) He, L.; Yu, Q.; Dong, H.; Liao, B.; Xu, X.; Goldblum, M.; Bian, J.; and Mesgarani, N. 2026. LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches. arXiv:2604.01754. Huang et al. (2026) Huang, Z.; Xu, J.; Yang, Y.; Gong, Z.; Yang, Q.; Tian, M.; Wang, X.; Lv, C.; Gao, X.; Dai, Q.; Liu, B.; Qiu, K.; Yang, X.; Chen, D.; Zheng, X.; and Luo, C. 2026. From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills. arXiv:2605.23899. Jiang et al. (2026a) Jiang, G.; Su, Z.; Qu, X.; and Fung, Y. R. 2026a. XSkill: Continual Learning from Experience and Skills in Multimodal Agents. arXiv:2603.12056. Jiang et al. (2026b) Jiang, S.; Ma, L.; Hong, Z.; Wang, K.; Lu, Z.; Wang, T.; Chen, S.; Zhang, J.; Pan, T.; Li, W.; Liang, J.; and Xiao, Y. 2026b. SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment. arXiv:2604.08988. Kim et al. (2026) Kim, J.; Yang, W.; Niu, K.; Zhang, H.; Zhu, Y.; Helenowski, E.; Silva, R.; Chen, Z.; Iyer, S.; Zaheer, M.; Fried, D.; Hajishirzi, H.; Arora, S.; Synnaeve, G.; Salakhutdinov, R.; and Goyal, A. 2026. Scaling Test-Time Compute for Agentic Coding. arXiv:2604.16529. Li et al. (2026a) Li, X.; Liu, Y.; Chen, W.; You, B.; Di, Z.; He, Y.; Zheng, S.; Choe, K. W.; Sun, J.; Wang, S.; Tao, C.; Li, B.; Zhao, X.; Geng, H.; Wu, X.; Zhou, J.; Chen, X.; Xing, H.; Li, Y.; Zeng, Q.; Wang, D.; Wang, Y.; Chaim, R. B.; Jiang, P.; Shen, H.; Kong, L.; Liu, X.; Wang, R.; Liu, X.; Li, J.; Lan, X.; Lin, Y.; Ye, W.; He, J.; Li, S.; Zhang, Y.; Gao, Y.; Li, Y.; Ma, Z.; Jing, L.; Wang, T.; Li, K.; Xue, Y.; Lyu, H.; He, Y.; Tian, Y.; Wu, S.; Wang, B.; Gao, Y.; Chen, B.; Liu, L.; Cheng, S.; Bao, J.; Tong, S.; Xu, S.; Zhuo, T. Y.; Ye, T.; Qi, Q.; Li, M.; Liao, L.; Tan, Z.; Shi, C.; Tang, X.; Tankasala, S.; Yuan, B.; Qian, Y.; Tu, J.; Wang, C.; Sun, Y.; Wang, W.; Taylor, A.; Yang, Z.; Guan, C.; Dong, Z.; Zhang, X.; Dillmann, S.; chung Lee, H.; and Song, D. 2026a. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670. Li et al. (2026b) Li, X.; Ming, R.; Setlur, P.; Paladugu, A.; Tang, A.; Kang, H.; Shao, S.; Jin, R.; and Xiong, C. 2026b. Benchmark Test-Time Scaling of General LLM Agents. arXiv:2602.18998. Li and Hu (2026) Li, Z.; and Hu, Y. 2026. SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History. arXiv:2606.08671. Liu et al. (2026a) Liu, H.; Yang, H.; Jiang, T.; Tang, B.; Xiong, F.; Luo, Y.; and Li, Z. 2026a. SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution. arXiv:2605.18401. Liu et al. (2026b) Liu, X.; Luo, X.; Li, L.; Huang, G.; Liu, J.; and Qiao, H. 2026b. SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support. arXiv:2604.08618. Liu (2025) Liu, Y. 2025. Exploring ConceptNet Knowledge for Enhancing LLMs Performance in Argument Impact Classification Tasks. In Proceedings of 17th International Conference on Machine Learning and Computing, volume 1475 of Lecture Notes in Networks and Systems, 379–395. Springer. Liu et al. (2026c) Liu, Y.; Ji, J.; An, L.; Jaakkola, T.; Zhang, Y.; and Chang, S. 2026c. How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings. arXiv:2604.04323. Liu et al. (2026d) Liu, Y.; Liu, Y.; Yin, X.; Wang, B.; Zhang, C.; Yin, H.; and Han, Z. 2026d. OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories. arXiv:2605.29253. Liu et al. (2026e) Liu, Y.; Su, Z.; Xie, L.; Zhang, Y.; Zong, Q.; Guo, J.; Xie, Z.; Ji, Y.; Yim, Y.; Luo, H.; Ren, X.; Ruan, C.; Li, H.; and Song, Y. 2026e. SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision. arXiv:2606.01139. Ma et al. (2026) Ma, Y.; Huang, Y.; Bao, H.; Zhuang, H.; Shukla, S.; Galley, M.; Zhang, X.; and Feuerriegel, S. 2026. SkillGen: Verified Inference-Time Agent Skill Synthesis. arXiv:2605.10999. Ma et al. (2024) Ma, Z.; Zhang, B.; Zhang, J.; Yu, J.; Zhang, X.; Zhang, X.; Luo, S.; Wang, X.; and Tang, J. 2024. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. arXiv:2406.14991. Mathew, Karatzas, and Jawahar (2021) Mathew, M.; Karatzas, D.; and Jawahar, C. V. 2021. DocVQA: A Dataset for VQA on Document Images. arXiv:2007.00398. Ni et al. (2026) Ni, J.; Liu, Y.; Liu, X.; Sun, Y.; Zhou, M.; Cheng, P.; Wang, D.; Zhao, E.; Jiang, X.; and Jiang, G. 2026. Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills. arXiv:2603.25158. Shen, Li, and Zhang (2026) Shen, Y.; Li, B.; and Zhang, X. 2026. SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibe. arXiv:2607.03451. Shridhar et al. (2021) Shridhar, M.; Yuan, X.; Côté, M.-A.; Bisk, Y.; Trischler, A.; and Hausknecht, M. 2021. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. arXiv:2010.03768. Su et al. (2026) Su, Z.; Gao, J.; Guo, H.; Liu, Z.; Zhang, L.; Geng, X.; Huang, S.; Xia, P.; Jiang, G.; Wang, C.; Zhang, Y.; Fung, Y. R.; and He, J. 2026. AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios. arXiv:2602.23166. Tian et al. (2026) Tian, Y.; Chen, J.; Zheng, L.; Tao, M.; Zeng, X.; Yin, Z.; Su, H.; and Sun, X. 2026. Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO. arXiv:2604.27488. Wang et al. (2026) Wang, Y.; Zhu, H.; Hu, Z.; Yuan, Y.; Chen, Z.; Senthil, S.; Hajishirzi, H.; Tsvetkov, Y.; Dasigi, P.; and Xiao, T. 2026. Rethinking the Evaluation of Harness Evolution for Agents. arXiv:2607.12227. Xia et al. (2026) Xia, P.; Chen, J.; Wang, H.; Liu, J.; Zeng, K.; Wang, Y.; Han, S.; Zhou, Y.; Zhao, X.; Chen, H.; Zheng, Z.; Xie, C.; and Yao, H. 2026. SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. arXiv:2602.08234. Xie et al. (2026a) Xie, Z.; Liao, R.; Wang, Z.; Chen, C.; Hua, X.-S.; and Luo, X. 2026a. GALA: Geometric Data Selection with Strategic Prospecting for Large Language Model Self-training. In Findings of the Association for Computational Linguistics: ACL 2026, 10281–10293. Association for Computational Linguistics. Xie et al. (2026b) Xie, Z.; Ren, X.; Zheng, T.; Bai, J.; Fan, W.; Xu, B.; Li, H.; Jing, H.; and Song, Y. 2026b. A Survey on AI Agent Harness. ResearchGate preprint. Yang et al. (2026a) Yang, C.; Jiang, Y.; Deng, Y.; and Wu, C. 2026a. Skill-Conditioned Visual Geolocation for Vision-Language Models. arXiv:2604.09025. Yang et al. (2026b) Yang, H.; Zhao, K.; Ma, M.; Lu, X.; Wang, X.; and Qian, H. 2026b. OptSkills: Learning Generalizable Optimization Skills from Problem Archetypes via Cluster-Based Distillation. arXiv:2605.29829. Yang et al. (2026c) Yang, Y.; Gong, Z.; Huang, W.; Yang, Q.; Zhou, Z.; Huang, Z.; Li, Y.; Gao, X.; Dai, Q.; Liu, B.; Qiu, K.; Yang, Y.; Chen, D.; Yang, X.; and Luo, C. 2026c. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904. Yu et al. (2026) Yu, Z.; Xie, X.; Yao, W.; Wang, C.; Liang, L.; Qi, X.; and Deng, S. 2026. SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories. arXiv:2606.01311. Zhang et al. (2026a) Zhang, H.; Fan, S.; Zou, H. P.; Chen, Y.; Wang, Z.; Zhou, J.; Li, C.; Huang, W.-C.; Yao, Y.; Zheng, K.; Liu, X.; Li, X.; and Yu, P. S. 2026a. CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification. arXiv:2604.01687. Zhang et al. (2026b) Zhang, H.; Long, Q.; Bao, J.; Feng, T.; Zhang, W.; Yue, H.; and Wang, W. 2026b. MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents. arXiv:2602.02474. Zhang et al. (2026c) Zhang, X.; Wang, G.; Cui, Y.; Li, Z.; Qiu, W.; Zhu, B.; and He, P. 2026c. Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents. arXiv:2607.12790. Zhang et al. (2025a) Zhang, Y.; Duan, S.; Su, J.; Liu, C.; and Han, P. 2025a. SPFT-SQL: Enhancing Large Language Model for Text-to-SQL Parsing by Self-Play Fine-Tuning. In Findings of the Association for Computational Linguistics: EMNLP 2025, 1092–1110. Association for Computational Linguistics. Zhang et al. (2026d) Zhang, Y.; Yan, L.; Duan, S.; Zha, X.; Su, J.; Han, P.; and Liu, C. 2026d. AFT-Tab: Adversarial Fine-Tuning for Tabular Data Synthesis with Long Text Columns. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 4581–4594. Association for Computational Linguistics. Zhang et al. (2025b) Zhang, Z.; Ding, S.; Dong, X.; He, S.; Lin, J.; Tang, J.; Zang, Y.; Cao, Y.; Lin, D.; and Wang, J. 2025b. SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction. arXiv:2507.15852. Zhang et al. (2026e) Zhang, Z.; Li, Y.; Ding, S.; Zang, Y.; Ding, S.; Xing, L.; Wang, Y.; Zhang, Q.; and Wang, J. 2026e. SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction. arXiv:2605.20110. Zhong et al. (2026) Zhong, S.; Lu, Y.; Ning, J.; Wan, Y.; Feng, L.; Ao, Y.; Ribeiro, L. F. R.; Dreyer, M.; Ammirati, S.; and Xiong, C. 2026. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv:2604.20087. Zhou et al. (2026) Zhou, X.; Liu, J.; Li, L.; Rossi, R. A.; and Hu, X. 2026. Counterfactual Trace Auditing of LLM Agent Skills. arXiv:2605.11946. Zhu et al. (2026) Zhu, J.; Mao, K.; Guo, Y.; He, D.; Xu, S.; Gu, S.; and Yue, Y. 2026. SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use. arXiv:2607.01874. Zhu et al. (2025) Zhu, K.; Li, H.; Wu, S.; Xing, T.; Ma, D.; Tang, X.; Liu, M.; Yang, J.; Liu, J.; Jiang, Y. E.; Zhang, C.; Lin, C.; Wang, J.; Zhang, G.; and Zhou, W. 2025. Scaling Test-time Compute for LLM Agents. arXiv:2506.12928. Appendix A Experimental Protocol and Evaluation Design Benchmarks, Metrics, and Data Splits Benchmark Task and output Source and fixed pool Primary metric Auxiliary metric SearchQA Short-answer QA over retrieved web snippets SearchQA (Dunn et al. 2017); fixed SkillOpt-Lite subset (400/200/1,400) 1 iff the normalized prediction exactly matches any reference Maximum token F1 OfficeQA Grounded QA over U.S. Treasury Bulletins with search/read tools OfficeQA Full; complete 246-item release (49/49/148) 1 iff the adapter-normalized prediction exactly matches the target Token F1 SpreadsheetBench Instruction-following spreadsheet manipulation Verified-400 (Ma et al. 2024) (80/39/281) 1 iff every required workbook test case passes Fraction of test cases passed ALFWorld Multi-step household task completion in a text environment ALFWorld (Shridhar et al. 2021); fixed episodes (200/140/134) 1 iff the simulator returns won Same binary outcome LiveMath Multiple-choice reasoning over recent mathematics papers LiveMathematicianBench (He et al. 2026); Nov. 2025–Feb. 2026 snapshot (36/35/106) 1 iff the parsed option equals the gold option Same binary outcome DocVQA Question answering over document-page images DocVQA (Mathew, Karatzas, and Jawahar 2021); sample from the official validation split (107/53/374) 1 iff the maximum ANLS across references is at least 0.999 Maximum ANLS Table A1: Benchmark tasks, sources, fixed evaluation pools, and metrics. ALFWorld is reported as an additional analysis, separately from the primary aggregates. Note. Every reported Score is 100 times the mean of the binary outcome in the fourth column. Auxiliary metrics enter the validation gate or tie-breaking rule when configured, while reported hard scores use the primary metric. OfficeQA uses normalized exact match in our fixed adapter; its official release (github.com/databricks/officeqa) also provides tolerance-based numerical scoring. The DocVQA hard score thresholds maximum ANLS at 0.999. Evolution Protocol and Feedback Isolation Protocol component Fixed setting Run structure Each benchmark–view run starts from the same benchmark-specific parent skill and continues for at most ten rounds. Per-round budget Each round executes 40 training trajectories, except LiveMath, which uses its full 36-item training split; a nonempty view triggers one optimizer call, up to four edits, and full-split validation. Feedback exposure Normal exposes successful and failed trajectories, Fail-only exposes failed trajectories, and Success-only exposes successful trajectories. Revision discipline The same revision instruction applies to all three feedback views: make at most four minimal, task-general edits supported by visible evidence. Acceptance and stopping A candidate becomes the next-round skill when validation does not decrease; otherwise, the incumbent is retained. Only a strict validation improvement updates the best checkpoint. A run stops after ten rounds, five consecutive regressions or no-ops, or an empty view-specific feedback pool; no optimizer proposal is made in the last case. Selection and identity The validation-best skill is restored for each run. Cross-view reporting uses validation hard score and validation soft score; candidate and retained artifacts are compared byte-for-byte. Table A2: Common multi-round evolution protocol. Control Setting Matched scope Underlying model GPT-5.5; gemini-3.1-pro-preview; deepseek-v4-pro Within each setting, executor and optimizer use the same model. Across rounds, only the incumbent skill carries over; earlier trajectories and rejected candidates do not. Execution pathway Fixed within each model–benchmark setting All three feedback views use the same executor and tool interface within a matched setting. Scheduling GPT-5.5: 20 workers on SearchQA, 8 on SpreadsheetBench, and 4 on the other three benchmarks; Gemini and DeepSeek: 50 Stage-1 lanes Parallelism affects scheduling only; per-round batch sizes, the optimizer-call cap, and the validation gate are defined in Table A2. Randomness and retry Fixed splits, item identities, and commands; infrastructure-only completion retries Seeds determine loader or panel order where exposed, while API generation may vary. Behavior failures remain scored outcomes, and infrastructure-incomplete items are retried under the same protocol. Matched optimizer contract Same parent, optimizer instructions, sample schema, task panel, edit cap, and validation gate Within each setting, the optimizer prompt is fixed across rounds and differs across views only in its visible-directory declaration. Each call receives the incumbent skill and the permitted training records. Every record contains the task input, the executor’s final output, and its success label and score. Records also include the execution trace when available; failed records retain verifier diagnostics, including expected–observed mismatches when available. Arm-specific evidence Normal: passed and failed records; Fail-only: failed records; Success-only: passed records All eligible records from each configured round batch are exposed without balancing or resampling; only visibility changes, and neither records nor evolving state cross views or settings. Selection and evaluation boundary Validation hard score, then the configured auxiliary score; SHA-256 artifact identity Validation is external to the optimizer and drives acceptance and cross-view selection. Post-selection evaluation comprises test, R/T, TTS, and verifier sensitivity. The optimizer modifies the skill artifact; a new best requires both a strict validation improvement and a distinct skill hash. Byte-identical artifacts share the same skill identity. Table A3: Execution, feedback-isolation, and artifact-identity controls for matched comparisons across the three feedback views. Post-Selection Evaluation Design Benchmark Train/Val/Test Post-selection diagnostics TTS budget SearchQA 400/200/1,400 Three-repeat evaluation; R/T; verifier n=1,400n=1,400, K≤6K≤ 6; +4,924+4,924 calls/control OfficeQA 49/49/148 Three-repeat evaluation; R/T; verifier n=148n=148, K≤6K≤ 6; +740+740 calls/control ALFWorld 200/140/134 R/T – SpreadsheetBench 80/39/281 Three-repeat evaluation; R/T; verifier n=281n=281, K≤4K≤ 4; +843+843 calls/control LiveMath 36/35/106 Three-repeat evaluation; R/T; verifier n=106n=106, K≤8K≤ 8; +742+742 calls/control DocVQA 107/53/374 Three-repeat evaluation; R/T; verifier n=374n=374, K≤4K≤ 4; +1,122+1,122 calls/control Table A4: Benchmark splits and GPT-5.5 post-selection evaluation budgets. Diagnostics summarize the completed GPT-5.5 analyses; TTS budget gives the maximum additional target-model calls per control. Tier Definition R1 Same task and gold; surface form or input representation changes. R2 Same task and gold; irrelevant context or distractors are added. R3 Same task and gold; the output or interface contract changes. T1 New official task and gold from the same source artifact. T2 New official task and gold from a new source in a task subtype represented in the base panel. T3 New official task and gold from a new source with a deliberate subtype, difficulty, or distribution shift. Table A5: Definitions of robustness and transfer tiers; unavailable tiers are omitted from macro-averages. Benchmark Same-task robustness probes Official-task transfer probes SearchQA R1: semantic question paraphrase (n=97n=97); R2: answer-excluding irrelevant documents with context-order perturbation (n=100n=100); R3: strict JSON answer contract (n=100n=100). T1: unavailable in the fixed pool; T2: new-source QA subtype represented in the base panel (n=100n=100); T3: new-source QA subtype absent from the base panel (n=9n=9). OfficeQA R1: instruction reframing with the same question and source (n=57n=57); R2: answer-excluding irrelevant documents (n=50n=50); R3: strict single-key JSON answer (n=57n=57). T1: new official question on the same document (n=5n=5); T2: new-source easy-difficulty items (n=33n=33); T3: new-source hard-difficulty items (n=47n=47). SpreadsheetBench R1: task-ID and workbook-path renaming (n=99n=99); R2: irrelevant request from another workbook (n=100n=100); R3: fixed transform(input_path, output_path) response contract (n=100n=100). T1: unavailable in the fixed pool; T2: new-workbook cell-level manipulation (n=100n=100); T3: new-workbook sheet-level manipulation (n=57n=57). LiveMath R1: symbol alpha-renaming (n=97n=97) and choice-order/label permutation (n=100n=100); R2: irrelevant excerpt from another paper (n=100n=100); R3: strict single-key JSON answer (n=100n=100). T1: unavailable in the fixed pool; T2: new-paper theorem-type combination represented in the base panel (n=4n=4); T3: new-paper theorem-type combination absent from the base panel (n=2n=2). DocVQA R1: instruction reframing with the same question and page (n=100n=100); R2: irrelevant page paired with the target page (n=100n=100); R3: strict single-key JSON answer (n=100n=100). T1: new official question on the same page (n=18n=18); T2: new-page dominant what-question subtype (n=100n=100); T3: new-page non-dominant interrogative subtype (n=100n=100). Table A6: Benchmark-specific robustness and transfer probes. Reported n values are eligible items per condition and deployment. Appendix B Complete Evolution and Endpoint Results Validation and Released-Test Results ALFWorld results. Under the same evolution and evaluation protocol, ALFWorld’s three feedback runs generate 30 candidates, of which three establish new validation bests. Success-only produces the validation-selected round-9 skill and improves released-test performance by 2.2 points, whereas Normal is stronger on both robustness and transfer. Results appear in Tables A7, A9, and A14. Identity-aware validation results. Gemini–OfficeQA Success-only evaluates the same skill eight times, with hard scores ranging from 71.43% to 83.67% (mean 76.28%, standard deviation 3.92 points). Because artifact identity is unchanged, the branch remains at round 0. Gemini–DocVQA Success-only likewise retains the parent hash after its round-1 evaluation increase, and all three DeepSeek–OfficeQA views retain the parent. Their test and diagnostic entries characterize repeated execution of the parent artifact. Benchmark Condition Best r Validation Released test Primary Aux. Primary Aux. SearchQA Parent 0 160/200 (80.00) 88.00 1059/1400 (75.64) 85.68 Normal 9 164/200 (82.00) 88.78 1091/1400 (77.93) 86.81 Fail-only 2 163/200 (81.50) 88.52 1082/1400 (77.29) 86.32 Success-only 4 163/200 (81.50) 88.77 1085/1400 (77.50) 86.84 OfficeQA Parent 0 38/49 (77.55) 83.33 93/148 (62.84) 67.72 Normal 9 43/49 (87.76) 89.12 103/148 (69.59) 70.43 Fail-only 8 42/49 (85.71) 87.07 103/148 (69.59) 70.24 Success-only 6 42/49 (85.71) 86.39 99/148 (66.89) 67.23 SpreadsheetBench Parent 0 16/39 (41.03) 41.03 141/281 (50.18) 50.18 Normal 3 32/39 (82.05) 82.05 231/281 (82.21) 82.21 Fail-only 6 32/39 (82.05) 82.05 241/281 (85.77) 85.77 Success-only 0 16/39 (41.03) 41.03 134/281 (47.69) 47.69 ALFWorld Parent 0 116/140 (82.86) – 117/134 (87.31) – Normal 8 120/140 (85.71) – 115/134 (85.82) – Fail-only 0 116/140 (82.86) – 119/134 (88.81) – Success-only 9 121/140 (86.43) – 120/134 (89.55) – LiveMath Parent 0 18/35 (51.43) – 52/106 (49.06) – Normal 0 18/35 (51.43) – 43/106 (40.57) – Fail-only 3 20/35 (57.14) – 45/106 (42.45) – Success-only 0 18/35 (51.43) – 46/106 (43.40) – DocVQA Parent 0 51/53 (96.23) 97.81 344/374 (91.98) 96.53 Normal 0 51/53 (96.23) 97.81 343/374 (91.71) 96.72 Fail-only 0 51/53 (96.23) 97.81 343/374 (91.71) 96.66 Success-only 0 51/53 (96.23) 97.81 341/374 (91.18) 96.24 Table A7: GPT-5.5 validation and released-test metrics for the parent and three feedback views across five primary benchmarks and ALFWorld. Note. Primary entries give successes over eligible n, followed by percentages in parentheses. Auxiliary entries are percentages under the metrics in Table A1; – denotes no distinct auxiliary metric. A run with best r=0r=0 retains the parent skill, and its released-test entry is an independent execution of that artifact. Model Benchmark Condition Best r Val. primary Val. aux. Test primary Test aux. Gemini 3.1 Pro SearchQA Parent 0 79.00 85.97 76.93 86.07 Normal 1 82.00 88.42 76.57 85.71 Fail-only 6 81.50 88.80 77.57 86.66 Success-only 1 81.00 87.78 77.79 86.53 OfficeQA Parent 0 77.55 79.25 65.54 66.55 Normal=P 0 77.55 79.25 65.54 66.10 Fail-only=P 0 77.55 79.25 64.86 65.70 Success-only=P 0 77.55 79.25 67.57 68.13 SpreadsheetBench Parent 0 38.46 – 41.99 – Normal 7 82.05 – 79.72 – Fail-only 1 76.92 – 81.49 – Success-only 4 48.72 – 50.18 – LiveMath Parent 0 42.86 – 41.51 – Normal 9 74.29 – 64.15 – Fail-only 10 74.29 – 56.60 – Success-only 2 62.86 – 61.32 – DocVQA Parent 0 92.45 96.77 95.45 97.66 Normal 4 94.34 97.18 95.99 97.93 Fail-only 4 94.34 97.18 95.19 97.82 Success-only=P 0 92.45 96.77 95.72 98.14 DeepSeek V4-Pro SearchQA Parent 0 74.50 82.71 73.00 82.49 Normal 1 76.50 83.88 73.86 83.41 Fail-only 2 76.00 83.37 74.21 84.05 Success-only=P 0 74.50 82.71 72.00 82.14 OfficeQA Parent 0 73.47 75.51 56.76 57.69 Normal=P 0 73.47 75.51 54.73 55.62 Fail-only=P 0 73.47 75.51 52.70 53.77 Success-only=P 0 73.47 75.51 56.76 57.76 SpreadsheetBench Parent 0 38.46 – 39.86 – Normal 3 76.92 – 68.68 – Fail-only 1 69.23 – 64.77 – Success-only 6 48.72 – 39.15 – LiveMath Parent 0 40.00 – 10.38 – Normal 8 54.29 – 20.75 – Fail-only 8 48.57 – 28.30 – Success-only=P 0 40.00 – 23.58 – Table A8: Gemini 3.1 Pro and DeepSeek V4-Pro validation and released-test metrics for the parent and three feedback views across nine supported model–benchmark settings. Note. Scores are percentages under the primary and auxiliary metrics in Table A1; – denotes no distinct auxiliary metric. The superscript =P marks a feedback branch whose validation-best artifact is byte-identical to the parent. Its validation entries therefore match the parent; released-test entries report independent executions of the same artifact. Evolution Dynamics Model Benchmark View Executed rounds First new best Best r # new bests Cross-view choice GPT-5.5 SearchQA Normal 10 1 9 2 Yes Fail-only 10 2 2 1 – Success-only 10 4 4 1 – OfficeQA Normal 10 1 9 3 Yes Fail-only 10 3 8 3 – Success-only 10 2 6 3 – SpreadsheetBench Normal 8 2 3 2 – Fail-only 10 2 6 4 Yes Success-only 6 – 0 0 – LiveMath Normal 5 – 0 0 – Fail-only 8 3 3 1 Yes Success-only 5 – 0 0 – DocVQA Normal 9 – 0 0 Parent Fail-only 7 – 0 0 – Success-only 7 – 0 0 – Gemini 3.1 Pro SearchQA Normal 10 1 1 1 Yes Fail-only 10 5 6 2 – Success-only 10 1 1 1 – OfficeQA Normal 10 – 0 0 Parent Fail-only 10 – 0 0 – Success-only 10 – 0 0 – SpreadsheetBench Normal 10 1 7 4 Yes Fail-only 10 1 1 1 – Success-only 10 2 4 2 – LiveMath Normal 10 1 9 5 Yes Fail-only 10 1 10 4 – Success-only 10 2 2 1 – DocVQA Normal 10 4 4 1 Yes Fail-only 10 4 4 1 – Success-only 10 – 0 0 – DeepSeek V4-Pro SearchQA Normal 10 1 1 1 Yes Fail-only 10 2 2 1 – Success-only 10 – 0 0 – OfficeQA Normal 10 – 0 0 Parent Fail-only 10 – 0 0 – Success-only 10 – 0 0 – SpreadsheetBench Normal 10 1 3 3 Yes Fail-only 10 1 1 1 – Success-only 10 2 6 3 – LiveMath Normal 10 8 8 1 Yes Fail-only 10 6 8 2 – Success-only 3 – 0 0 – GPT-5.5 ALFWorld Normal 10 8 8 1 – Fail-only 10 – 0 0 – Success-only 10 6 9 2 Yes Table A9: Evolution summary for the 42 primary feedback runs and three additional GPT-5.5–ALFWorld runs. The primary runs contain 388 evaluated candidates and 55 byte-distinct validation new bests. A dash under First new best and r=0r=0 under Best r indicate that the parent remains best; fewer than ten executed rounds indicate early stopping. Figure A1: Candidate validation differences from each setting’s round-0 parent across the 42 primary feedback runs. Panels correspond to GPT-5.5, Gemini 3.1 Pro, and DeepSeek V4-Pro. White circles mark byte-distinct validation new bests, black squares mark setting-level selections, and gray cells mark rounds omitted after early stopping. Cells report raw validation differences; artifact improvements require a byte-distinct candidate. N, F, and S denote Normal, Fail-only, and Success-only. Round 1 2 3 4 5 6 7 8 9 10 Candidates attempted 42 42 42 41 41 39 38 36 34 33 New-best events 11 11 8 8 2 5 2 4 3 1 Per-round yield (%) 26.2 26.2 19.0 19.5 4.9 12.8 5.3 11.1 8.8 3.0 Cumulative events 11 22 30 38 40 45 47 51 54 55 Table A10: Round-wise candidate yield across the 42 primary feedback runs. At each round, yield is the fraction of attempted candidates that establish a byte-distinct validation new best; the number of attempted candidates decreases as runs stop early. Feedback evidence Runs Candidates New bests Yield Runs improved Selected Normal 14 132 23 17.4% 10/14 9 Fail-only 14 135 21 15.6% 11/14 2 Failure-containing 28 267 44 16.5% 21/28 11 Success-only 14 121 11 9.1% 6/14 0 Table A11: Search outcomes by feedback evidence across the 42 primary runs. The Failure-containing row aggregates Normal and Fail-only. Runs improved denotes at least one byte-distinct validation new best within a run; Selected counts setting-level evolved-skill selections. Failure-containing feedback and late-round selection. Pooling Normal and Fail-only yields 44 byte-distinct validation new bests in 267 candidates (16.5%), compared with 11/121 (9.1%) for Success-only. At least one new best appears in 21/28 failure-containing runs (75.0%) and 6/14 Success-only runs (42.9%); all 11 primary evolved selections come from the former. Rounds 1–4 account for 38/55 new bests (69.1%), while later rounds account for 17/55 (30.9%). Yet six of the 11 selected evolved skills first appear in rounds 6–9, compared with five in rounds 1–4. Later discovery is less frequent but determines a majority of final evolved selections. Cross-Model SearchQA Results Model Selected view Selected r Val. Δ Test Δ Test-best view Selection gap Claude Opus 4.8 Success-only 7 +2.0+2.0 +3.79+3.79 Success-only 0.000.00 Kimi K2.5 Fail-only 2 +7.0+7.0 +5.14+5.14 Success-only 0.710.71 Gemini 3.1 Pro Normal 1 +3.0+3.0 −0.36-0.36 Success-only 1.211.21 DeepSeek V4-Pro Normal 1 +2.0+2.0 +0.86+0.86 Fail-only 0.360.36 GLM-5.1 Fail-only 5 +7.0+7.0 +3.43+3.43 Normal 0.070.07 Grok 4.5 Normal 3 +25.0+25.0 +15.07+15.07 Fail-only 0.360.36 Qwen3.5-Plus Success-only 3 +3.5+3.5 +2.93+2.93 Fail-only 0.570.57 Table A12: SearchQA results for seven additional models. All three views execute ten rounds per model; among 210 candidates, 191 change the incoming skill and 29 establish a byte-distinct validation new best. Selection gap is the released-test score of the test-best feedback view minus that of the validation-selected view. GLM is evaluated on a common 1,398-item test subset, and Qwen3.5-Plus on 1,397 items. Differences are percentage points. Eight-model SearchQA consistency. Including GPT-5.5, validation-selected skills improve released-test performance in seven of eight SearchQA models. Normal or Fail-only is selected in six models, while Success-only is selected for Claude Opus and Qwen3.5-Plus. Selected rounds span 1–9, extending the primary study’s sparse, feedback-dependent dynamics across model families (Xie et al. 2026a, b; Zhang et al. 2026e, 2025b). Appendix C Post-Selection Generalization Deployment Variability Repeated evaluations cover the five GPT-5.5 benchmark settings, while robustness and transfer probes additionally include ALFWorld. Probe panels are fixed before evaluation and report N/A when a tier is absent from the benchmark pool. The following tables give the probe-level results; Table A15 gives aggregate robustness and transfer results for all selected primary skills. Benchmark Condition n Test Δ Repeat Δ1 _1 Repeat Δ2 _2 Repeat Δ3 _3 Mean Δ Parent Evolved SearchQA Normal 100 +2.3+2.3 −2.0-2.0 −3.0-3.0 0.00.0 −1.7-1.7 78.0 76.3 OfficeQA Normal 57 +6.8+6.8 +17.5+17.5 +19.3+19.3 +15.8+15.8 +17.5+17.5 58.5 76.0 DocVQA Parent skill 100 −0.3-0.3 −2.0-2.0 −2.0-2.0 +2.0+2.0 −0.7-0.7 90.7 90.0 LiveMath Fail-only 100 −6.6-6.6 +6.0+6.0 +7.0+7.0 +5.0+5.0 +6.0+6.0 52.3 58.3 SpreadsheetBench Normal 100 +32.0+32.0 +28.0+28.0 +32.0+32.0 +30.0+30.0 +30.0+30.0 54.0 84.0 Table A13: Three-repeat evaluation of GPT-5.5 artifacts on fixed benchmark-specific panels. Test Δ is the released-test difference; Repeat Δ1 _1–Δ3 _3 are paired-panel differences, and Parent and Evolved are their mean scores. The SpreadsheetBench row is the paired Normal diagnostic; Tables 2 and A14 report the selected Fail-only skill. DocVQA compares independent parent executions. Values are percentage points. Robustness and Transfer Same-task robustness Official-task transfer Benchmark Condition R1 Δ (n) R2 Δ (n) R3 Δ (n) T1 Δ (n) T2 Δ (n) T3 Δ (n) SearchQA Normal −0.7-0.7 (97) +2.3+2.3 (100) −1.0-1.0 (100) N/A +7.3+7.3 (100) +3.7+3.7 (9) Fail-only −2.1-2.1 (97) +0.3+0.3 (100) −1.3-1.3 (100) N/A +8.3+8.3 (100) 0.00.0 (9) Success-only +1.0+1.0 (97) +3.3+3.3 (100) +1.3+1.3 (100) N/A +6.3+6.3 (100) 0.00.0 (9) OfficeQA Normal +15.8+15.8 (57) +10.0+10.0 (50) +8.2+8.2 (57) +20.0+20.0 (5) +13.1+13.1 (33) +2.1+2.1 (47) Fail-only +10.5+10.5 (57) +5.3+5.3 (50) +8.2+8.2 (57) +20.0+20.0 (5) +5.1+5.1 (33) 0.00.0 (47) Success-only +11.7+11.7 (57) +1.3+1.3 (50) +1.8+1.8 (57) 0.00.0 (5) +10.1+10.1 (33) +3.5+3.5 (47) ALFWorld Normal +3.0+3.0 (44) +2.3+2.3 (44) +1.5+1.5 (44) N/A +2.3+2.3 (44) 0.00.0 (11) Fail-only† +0.8+0.8 (44) +2.3+2.3 (44) −1.5-1.5 (44) N/A −0.8-0.8 (44) −3.0-3.0 (11) Success-only +3.0+3.0 (44) +2.3+2.3 (44) −1.5-1.5 (44) N/A +3.0+3.0 (44) −3.0-3.0 (11) SpreadsheetBench Normal +31.3+31.3 (99) +36.3+36.3 (100) +38.3+38.3 (100) N/A +52.0+52.0 (100) +3.5+3.5 (57) Fail-only +26.3+26.3 (99) +39.0+39.0 (100) +41.3+41.3 (100) N/A +53.3+53.3 (100) +2.9+2.9 (57) Success-only −10.1-10.1 (99) +0.7+0.7 (100) +0.7+0.7 (100) N/A −1.0-1.0 (100) +1.2+1.2 (57) LiveMath Fail-only R1a: +3.8+3.8 (97) R1b: +6.3+6.3 (100) +12.0+12.0 (100) +11.0+11.0 (100) N/A 0.00.0 (4) −33.3-33.3 (2) DocVQA Parent skill −0.3-0.3 (100) −2.0-2.0 (100) −1.0-1.0 (100) +1.9+1.9 (18) −1.0-1.0 (100) 0.00.0 (100) Table A14: GPT-5.5 probe-level robustness and transfer differences across five primary benchmarks and ALFWorld. Entries are three-deployment mean differences from parent; parentheses give eligible n. The table includes every validation-improving run and three parent-identical comparison conditions. Macro-averages weight available probes equally and omit N/A tiers; LiveMath R1a and R1b count separately. Values are percentage points. Model Benchmark Selected skill R1 Δ R2 Δ R3 Δ T1 Δ T2 Δ T3 Δ R T GPT-5.5 SearchQA Normal, r=9r=9 −0.7-0.7 +2.3+2.3 −1.0-1.0 N/A +7.3+7.3 +3.7+3.7 +0.2+0.2 +5.5+5.5 GPT-5.5 OfficeQA Normal, r=9r=9 +15.8+15.8 +10.0+10.0 +8.2+8.2 +20.0+20.0 +13.1+13.1 +2.1+2.1 +11.3+11.3 +11.7+11.7 GPT-5.5 SpreadsheetBench Fail-only, r=6r=6 +26.3+26.3 +39.0+39.0 +41.3+41.3 N/A +53.3+53.3 +2.9+2.9 +35.5+35.5 +28.1+28.1 GPT-5.5 LiveMath Fail-only, r=3r=3 +3.8/+6.3+3.8/+6.3 +12.0+12.0 +11.0+11.0 N/A 0.00.0 −33.3-33.3 +8.3+8.3 −16.7-16.7 Gemini 3.1 Pro SearchQA Normal, r=1r=1 −0.7-0.7 +1.0+1.0 −1.3-1.3 N/A +3.0+3.0 0.00.0 −0.3-0.3 +1.5+1.5 Gemini 3.1 Pro SpreadsheetBench Normal, r=7r=7 +44.8+44.8 +44.3+44.3 +42.3+42.3 N/A +50.3+50.3 +0.6+0.6 +43.8+43.8 +25.5+25.5 Gemini 3.1 Pro LiveMath Normal, r=9r=9 +23.7/+16.3+23.7/+16.3 +17.0+17.0 +17.0+17.0 N/A +16.7+16.7 +16.7+16.7 +18.5+18.5 +16.7+16.7 Gemini 3.1 Pro DocVQA Normal, r=4r=4 +1.0+1.0 −0.7-0.7 +0.7+0.7 0.00.0 −1.3-1.3 +0.3+0.3 +0.3+0.3 −0.3-0.3 DeepSeek V4-Pro SearchQA Normal, r=1r=1 +1.4+1.4 −0.7-0.7 −1.3-1.3 N/A +3.7+3.7 0.00.0 −0.2-0.2 +1.8+1.8 DeepSeek V4-Pro SpreadsheetBench Normal, r=3r=3 +33.7+33.7 +31.0+31.0 +32.3+32.3 N/A +31.0+31.0 +11.1+11.1 +32.3+32.3 +21.1+21.1 DeepSeek V4-Pro LiveMath Normal, r=8r=8 +13.4/+7.7+13.4/+7.7 +11.0+11.0 +0.7+0.7 N/A +16.7+16.7 +50.0+50.0 +8.2+8.2 +33.3+33.3 Table A15: Probe-level robustness and transfer differences for all 11 byte-distinct skills selected in the primary study. R1 contains two panels for LiveMath and one for the other benchmarks. R and T are equal-weight macro-averages over available panels and match Table 2; panel definitions and sample sizes are in Tables A5 and A6. Values are percentage points relative to the corresponding parent. Breadth of downstream improvement. Among the 11 selected evolved skills, nine improve released-test performance, nine improve robustness, and nine improve transfer. Seven improve all three measures, and every selected skill improves at least one of robustness or transfer. SpreadsheetBench is the strongest cross-model case: all three models select an evolved skill, released-test gains range from 28.8 to 37.7 points, and robustness and transfer are positive for every model. Evaluation unit. Each model–benchmark–view cell retains a complete multi-round evolution trajectory. Fixed-panel scores and three post-selection robustness and transfer deployments characterize downstream behavior; cross-setting summaries aggregate the matched trajectories. Validation-to-Test Selection Gaps Model Benchmark Val.-selected Test Δ Test-best Test Δ Selection gap GPT-5.5 SearchQA Normal +2.3+2.3 Normal +2.3+2.3 0.00.0 GPT-5.5 OfficeQA Normal +6.8+6.8 Normal / Fail-only +6.8+6.8 0.00.0 GPT-5.5 SpreadsheetBench Fail-only +35.6+35.6 Fail-only +35.6+35.6 0.00.0 GPT-5.5 LiveMath Fail-only −6.6-6.6 Parent 0.00.0 6.66.6 GPT-5.5 DocVQA Parent 0.00.0 Parent 0.00.0 0.00.0 Gemini 3.1 Pro SearchQA Normal −0.4-0.4 Success-only +0.9+0.9 1.31.3 Gemini 3.1 Pro OfficeQA Parent 0.00.0 Parent 0.00.0 0.00.0 Gemini 3.1 Pro SpreadsheetBench Normal +37.7+37.7 Fail-only +39.5+39.5 1.81.8 Gemini 3.1 Pro LiveMath Normal +22.6+22.6 Normal +22.6+22.6 0.00.0 Gemini 3.1 Pro DocVQA Normal +0.5+0.5 Normal +0.5+0.5 0.00.0 DeepSeek V4-Pro SearchQA Normal +0.9+0.9 Fail-only +1.2+1.2 0.30.3 DeepSeek V4-Pro OfficeQA Parent 0.00.0 Parent 0.00.0 0.00.0 DeepSeek V4-Pro SpreadsheetBench Normal +28.8+28.8 Normal +28.8+28.8 0.00.0 DeepSeek V4-Pro LiveMath Normal +10.4+10.4 Fail-only +17.9+17.9 7.57.5 Table A16: Released-test selection gap under validation-based cross-view selection in the 14 primary settings. The gap is the best released-test score among the parent and byte-distinct skills returned by the three feedback views minus that of the validation-selected skill. Parent-identical views share the parent entry. Values are percentage points. Appendix D Artifact-Level Evidence Selected Skills and Provenance Model–benchmark Selected skill Retained operational guidance Observed scope GPT-5.5–SearchQA Normal, r=9r=9 Resolve the requested answer type; prefer canonical naming and concise answer-only output. +2.0/+2.3/+0.2/+5.5+2.0/+2.3/+0.2/+5.5 GPT-5.5–OfficeQA Normal, r=9r=9 Retrieve and read before computing; track operands and units; match the requested answer surface. +10.2/+6.8/+11.3/+11.7+10.2/+6.8/+11.3/+11.7 GPT-5.5–SpreadsheetBench Fail-only, r=6r=6 Inspect the workbook; execute a self-contained script; materialize values; save, reopen, and verify target cells. +41.0/+35.6/+35.5/+28.1+41.0/+35.6/+35.5/+28.1 GPT-5.5–LiveMath Fail-only, r=3r=3 Audit all options, quantifiers, hypotheses, equality cases, implication direction, and theorem strength; return only the label. +5.7/−6.6/+8.3/−16.7+5.7/-6.6/+8.3/-16.7 Gemini 3.1 Pro–SearchQA Normal, r=1r=1 Remove unnecessary corporate suffixes; use common canonical names or surnames when appropriate; answer only. +3.0/−0.4/−0.3/+1.5+3.0/-0.4/-0.3/+1.5 Gemini 3.1 Pro–SpreadsheetBench Normal, r=7r=7 Use valid openpyxl syntax; handle duplicate headers and nulls; preserve blocks; compute values and verify with dual loading. +43.6/+37.7/+43.8/+25.5+43.6/+37.7/+43.8/+25.5 Gemini 3.1 Pro–LiveMath Normal, r=9r=9 Reject partial or overstrong options; inspect quantifiers and formulas; return the exact complete option text. +31.4/+22.6/+18.5/+16.7+31.4/+22.6/+18.5/+16.7 Gemini 3.1 Pro–DocVQA Normal, r=4r=4 Discriminate neighboring labels and preserve exact punctuation and symbols in the answer. +1.9/+0.5/+0.3/−0.3+1.9/+0.5/+0.3/-0.3 DeepSeek V4-Pro–SearchQA Normal, r=1r=1 Identify the target entity type and return a concise canonical answer without elaboration. +2.0/+0.9/−0.2/+1.8+2.0/+0.9/-0.2/+1.8 DeepSeek V4-Pro–SpreadsheetBench Normal, r=3r=3 Separate sheet names; type-coerce comparisons; delete rows with marker-aware indexing; compute literal values and verify the saved workbook. +38.5/+28.8/+32.3/+21.1+38.5/+28.8/+32.3/+21.1 DeepSeek V4-Pro–LiveMath Normal, r=8r=8 Enforce the single-label contract and check meta-options and stronger-result traps before answering. +14.3/+10.4/+8.2/+33.3+14.3/+10.4/+8.2/+33.3 Table A17: Selected-skill change cards for the 11 byte-distinct evolved skills selected in the primary study. Note. The observed-scope entries report percentage-point differences in validation/released test/robustness/transfer order. Each row pairs a byte-distinct validation-selected artifact with its retained operational guidance and downstream scope. Model Benchmark Selected skill Parent SHA-256 Selected SHA-256 GPT-5.5 SearchQA Normal, r=9r=9 d3ed21de4a52 5f9c3ce70aef GPT-5.5 OfficeQA Normal, r=9r=9 9d377d1c2906 ca689fecd40d GPT-5.5 SpreadsheetBench Fail-only, r=6r=6 5c49d03008ae 859ab2927df6 GPT-5.5 LiveMath Fail-only, r=3r=3 84d545e168a0 e9bc725bfd2d Gemini 3.1 Pro SearchQA Normal, r=1r=1 d3ed21de4a52 5676bd5f0ae Gemini 3.1 Pro SpreadsheetBench Normal, r=7r=7 5c49d03008ae 26e7ec56b1fd Gemini 3.1 Pro LiveMath Normal, r=9r=9 84d545e168a0 5e24d6ca337 Gemini 3.1 Pro DocVQA Normal, r=4r=4 60fa9bae9059 5ce713b3c15f DeepSeek V4-Pro SearchQA Normal, r=1r=1 d3ed21de4a52 2b9b8273632c DeepSeek V4-Pro SpreadsheetBench Normal, r=3r=3 5c49d03008ae 60478e37ca7e DeepSeek V4-Pro LiveMath Normal, r=8r=8 84d545e168a0 be370bcfc14f Table A18: Artifact identity for the 11 selected evolved skills. Twelve-character SHA-256 prefixes are shown for readability; every selected artifact differs from its parent. Validation selectivity and artifact identity. Validation filters 388 evaluated candidates to 55 byte-distinct new bests (14.2%), and cross-view selection retains 11 evolved skills while keeping the parent in three settings. Each selected evolved skill has a hash distinct from its parent, linking the retained validation improvement to a persistent artifact update. Cross-Model Convergence in Selected Skills The selected artifacts reveal benchmark-level convergence across independently evolved skills (Su et al. 2026; Zhang et al. 2025a, 2026d; Liu 2025). The comparison below focuses on SearchQA, SpreadsheetBench, and LiveMath, for which all three primary models select byte-distinct evolved skills; Table A17 gives the corresponding setting-level cards. SearchQA: answer-surface convergence. All three selected artifacts require concise canonical entity answers, explicit identification of the requested entity type, and answer-only output. GPT-5.5 Normal at round 9 further distinguishes property values from category nouns and creators from work titles. Gemini Normal at round 1 removes corporate suffixes and unnecessary name components, while DeepSeek Normal at round 1 makes entity-type and creator-versus-title resolution explicit. SpreadsheetBench: executable postconditions. All three selected artifacts inspect the workbook, compute literal values in Python, preserve its structure, save the result, and verify the output. GPT-5.5 Fail-only at round 6 reopens the saved workbook with data_only=True and verifies populated target cells. Gemini Normal at round 7 adds dual loading, duplicate-header handling, numeric-null rules, and sortable-block headers. DeepSeek Normal at round 3 adds sheet-name separation, type-coerced matching, and marker-aware row deletion. LiveMath: theorem-level option comparison. All three selected artifacts compare options at theorem level, audit hypotheses and quantifiers, handle the recurring meta-option, and enforce the benchmark-specific answer contract. GPT-5.5 Fail-only at round 3 adds equality-case, implication-direction, and strongest-result checks. Gemini Normal at round 9 distinguishes partial from overstrong statements and returns the exact option text, while DeepSeek Normal at round 8 combines guarded meta-option selection with label-only output. Across the nine artifacts, the shared retained mechanisms cluster by benchmark—answer normalization for SearchQA, executable workbook postconditions for SpreadsheetBench, and theorem-level option auditing for LiveMath—while model-specific specialization appears in the operational clauses. Comparative Case Studies of Feedback Views We compare optimizer reports and skill.md differences from paired Normal and Fail-only runs. Each case connects the evidence visible to each view with retained rule changes and validation outcomes. GPT-5.5–OfficeQA: evidence retrieval and answer format. The two views repeatedly encounter unsupported first-turn answers on numerical and table questions. In the Normal run, round 9 inspected 15 failed summaries and final trace turns, together with three passed traces for contrast. UID0101 answered an arc-elasticity question without retrieval and produced −0.153-0.153 rather than the expected −1.162-1.162; UID0010 answered a Treasury-value question from the prompt rather than the cited evidence; UID0086 and UID0160 added a percent sign when numeric-only answers were required. The accepted revision made targeted search and reading a first-action contract and specified the numeric percent surface. Fail-only round 8 saw the same unsupported-answer pattern without passed examples and retained a shorter hard-stop and search–read replacement rule. A subsequent Normal candidate at round 10 further expanded the checklist around the same failure pattern, but was rejected by validation. View Round Evidence inspected Retained or proposed skill.md change Validation gate Normal 8 Read 16 failed samples and 3 passed contrasts; hard numerical failures frequently answered on turn 1 without evidence retrieval. Elevated retrieval to a pre-answer gate and added an ordered ledger for multi-stage transformations, units, and statistics. Accepted: 85.7 Normal 9 Read 15 failed summaries/final trace turns and 3 passed contrasts; unsupported first-turn answers remained dominant, with two numeric-percent surface failures. Required targeted search and reading before any answer, then specified when percent-change outputs omit %. Accepted: 87.8 Normal 10 Read 8 of 17 failed samples and 2 passed contrasts; again found one-turn unsupported answers in FX, regression, and “cannot be determined” tasks. Proposed a literal stop condition and a detailed grep → read → raw cells → compute checklist, with expanded formatting rules. Rejected: 79.6 Fail-only 8 Read all 16 failed samples and no passed samples; every inspected failure answered on turn 1 without source retrieval. Added a hard stop against turn-1 answers and a concrete search–read–compute replacement pattern. Accepted: 85.7 Table A19: GPT-5.5–OfficeQA case study. Counts summarize the trajectories inspected by the optimizer; validation is the hard-score percentage on the fixed validation split. GPT-5.5–SpreadsheetBench: formula-like instructions and verifier-observable values. Both views encounter spreadsheet edits that write formula strings while the verifier reads None rather than the intended scalar values. Normal round 3 inspected 12 failed samples and two passed contrasts. It generalized the failure into broad formula-as-values, lookup, and full-grid population guidance. Fail-only round 6 inspected six failures without passed examples and focused on the verifier-visible condition: after saving, the output workbook must be reopened with data_only=True and every required target cell must be populated. Both candidates reached 82.1 validation. The reported SpreadsheetBench result uses the round-6 Fail-only skill. View Round Failure evidence Accepted skill.md change Validation gate Normal 3 Read 12 failed samples and 2 passed contrasts. Recurrent cell_level tasks wrote formulas or incompletely filled copied ranges; the verifier read None. Added formula-as-values guidance for INDEX/MATCH, SUMIFS, COUNTIFS, and copied ranges, with a lookup-grid pattern that fills every target cell. Accepted: 82.1 Fail-only 6 Read all 6 failed samples. Formula-like tasks left target cells unpopulated or wrote formula strings that the verifier read as None. Made the verifier-visible requirement explicit: write scalar values, reopen the saved workbook with data_only=True, and check that required target cells are not None. Accepted: 82.1 Table A20: GPT-5.5–SpreadsheetBench case study. The two views address the same verifier-observable failure but retain different levels of operational specificity. Success-only: stable specifications versus incidental patterns. The broader SearchQA runs provide two positive Success-only cases, while DeepSeek–LiveMath provides a contrasting rejected revision. These cases indicate that Success-only can help when positive trajectories repeatedly support a task-wide specification, whereas sparse positive evidence can support incidental rules. Table A21 summarizes the observed evidence and validation outcomes. Setting Round Observed positive evidence Candidate rule Val. gate Test Δ DeepSeek–LiveMath 3 Eight successful trajectories in the round sample. Added a “strongest/equivalence” heuristic and changed the required output from an option label to full option text. 40.0→11.440.0→ 11.4; rejected – Claude Opus–SearchQA 7 Successful traces repeatedly identify the clue referent and use a short canonical answer. Made referent identification and concise canonical answer form explicit. 77.0→79.077.0→ 79.0; selected +3.79+3.79 Qwen3.5-Plus–SearchQA 3 Successful traces support a shared answer-form specification across questions. Retained the common short-answer specification without adding task-specific content. 73.0→76.573.0→ 76.5; selected +2.93+2.93 Table A21: Contrasting Success-only cases. The LiveMath candidate is rejected and the branch retains its parent; the two validation-selected SearchQA skills encode shared answer-form specifications supported across successful traces. End-to-End Evolution Traces Tables A22–A24 connect score changes to candidate content across three complete branches: a late answer-form improvement, a cumulative procedural repair, and a mixed-versus-positive-only contrast. “Feedback used” reports the trajectories visible to each optimizer call; validation scores and gate actions follow the recorded evolution histories. Late answer-form repair. GPT-5.5–SearchQA first improves in round 1, when a broad minimal-span rule raises validation from 80.0 to 81.0. Rounds 2–8 repeatedly revise how creator names and surnames should be expressed, but none surpasses that checkpoint. Round 9 then makes a narrower change—return the value of a requested property without its category noun—and establishes the selected 82.0 checkpoint. The late gain is a one-clause correction discovered after several plausible alternatives fail validation. Procedural repair. The Gemini SpreadsheetBench run improves from 38.5 to 69.2 in round 1 by replacing unevaluated formulas with literal values. Rounds 2 and 3 add import-path and dual-load safeguards, each contributing another 5.1 points. Three subsequent proposals fail to improve the best. Round 7 then addresses four concrete execution defects—cell-call syntax, duplicate headers, numeric nulls, and headers inside sortable blocks—and raises validation to 82.1. Later revisions tie or regress and are rolled back. Negative evidence and validation gating. DeepSeek-V4-Pro–LiveMath illustrates the role of rejected candidates in the evolution trajectory. Normal proposes seven unsuccessful revisions before the round-8 combination of label formatting and guarded meta-option selection improves validation from 40.0 to 54.3; the two later candidates fall to 20.0 and do not replace it. Success-only never exceeds the parent. Its round-3 revision infers a strength/equivalence rule from eight successes and changes the output contract to full option text, reducing validation to 11.4. The branch then has no eligible positive feedback and stops with the parent intact. r Feedback used Candidate revision Val. Gate Best r 0 – Round-0 parent; no learned rules. 80.0 Baseline 0 1 34 S + 6 F Add minimal answer spans, surname and company shortening, and creator-relation checks. 81.0 New best 1 2 39 S + 1 F Prefer the creator for bare descriptions of books, films, series, or trilogies. 81.0 Flat 1 3 39 S + 1 F Narrow the creator cue to descriptions such as “a trilogy/series set in …”. 79.0 Reject 1 4 39 S + 1 F Broaden the creator cue to bare “set in” and “about” descriptions. 80.5 Flat 1 5 38 S + 2 F Balance surname-only answers with full creator names when the clue does not identify the person. 80.0 Flat 1 6 40 S + 0 F No byte-distinct candidate (VALID_NULL). 79.0 Reject 1 7 37 S + 3 F Make surname-only the default for biographical clues unless the surname is ambiguous. 79.0 Reject 1 8 38 S + 2 F Use full names for ambiguous office, title, role, or family-relation clues. 79.5 Flat 1 9 39 S + 1 F For a property query, return only the value word rather than the category noun. 82.0 New best 9 10 40 S + 0 F No byte-distinct candidate (VALID_NULL). 81.0 Flat 9 Table A22: Complete GPT-5.5–SearchQA Normal evolution. Two of ten proposals establish byte-distinct validation bests; the selected skill first appears in round 9. r Feedback used Candidate revision Val. Gate Best r 0 – Round-0 parent. 38.5 Baseline 0 1 16 S + 24 F Compute literal values in Python; avoid unevaluated Excel formulas and unnecessary pandas. 69.2 New best 1 2 34 S + 6 F Add the sandbox-safe import-path workaround before loading openpyxl. 74.4 New best 2 3 33 S + 7 F Add dual workbook loading for formula values and reverse-order row deletion. 79.5 New best 3 4 33 S + 7 F Emulate blank-as-zero formula behavior, delete rows one by one, and exclude headers from sorting. 76.9 Reject 3 5 33 S + 7 F Add date/time conversion and an explore-first workbook inspection procedure. 74.4 Reject 3 6 33 S + 7 F Ban pandas and use a two-pass row-deletion procedure. 79.5 Flat 3 7 33 S + 7 F Fix cell-call syntax, duplicate-header mapping, numeric nulls, and headers inside contiguous blocks. 82.1 New best 7 8 35 S + 5 F Add dynamic header discovery, date-type checks, and safeguards for relative deletions. 79.5 Reject 7 9 34 S + 6 F Add partial header matching, occurrence-aligned duplicate mapping, and rounded time conversion. 82.1 Flat 7 10 34 S + 6 F Use zip for duplicate columns and explicit zero defaults for numeric grids. 82.1 Flat 7 Table A23: Complete Gemini-3.1-Pro–SpreadsheetBench Normal evolution. Validation retains four cumulative procedural revisions and rolls back the other six. View r Feedback used Candidate revision Val. Gate Best r Normal 0 – Round-0 parent. 40.0 Baseline 0 Normal 1 3 S + 33 F Add a best-effort, nonempty fallback. 5.7 Reject 0 Normal 2 3 S + 33 F Require a single-label guess when uncertain. 17.1 Reject 0 Normal 3 5 S + 31 F Add strength-aware comparison among options. 25.7 Reject 0 Normal 4 3 S + 33 F Require a valid A–E label and add a recurring meta-option rule. 31.4 Reject 0 Normal 5 3 S + 33 F Require a nonempty label for the strongest remaining option. 8.6 Reject 0 Normal 6 3 S + 33 F Expand the stronger-result and meta-option heuristic. 14.3 Reject 0 Normal 7 2 S + 34 F Recognize the recurring “one remaining option” meta-option. 14.3 Reject 0 Normal 8 3 S + 33 F Combine exact label formatting with a guarded meta-option strategy. 54.3 New best 8 Normal 9 11 S + 25 F Separate label-versus-statement output and add broader theorem checks. 20.0 Reject 8 Normal 10 4 S + 32 F Make the meta-option default more aggressive while retaining label-only output. 20.0 Reject 8 Success-only 0 – Round-0 parent. 40.0 Baseline 0 Success-only 1 3 S Add no task rule; the generated diagnostic report changes the candidate bytes. 5.7 Reject 0 Success-only 2 5 S Add a “strongest statement” heuristic. 37.1 Reject 0 Success-only 3 8 S Add strength/equivalence rules and change the required output from a label to full option text. 11.4 Reject 0 Success-only stop 0 S No eligible positive feedback remains after round 3; retain the parent without another optimizer call. – Early stop 0 Table A24: Complete DeepSeek-V4-Pro–LiveMath trajectory contrast. Normal selects its round-8 repair after seven candidate revisions; Success-only evaluates three revisions before the positive-feedback pool is exhausted. Representative Selected-Skill Excerpts The selected skills encode three distinct forms of retained revision: answer-surface normalization for SearchQA, theorem-level option comparison for LiveMath, and an executable workbook workflow for SpreadsheetBench. ⬇ # Question Answering Skill Answer with the shortest conventional answer that satisfies the clue. - Prefer the common quiz-bowl/search answer form over a fuller encyclopedia name: for people, use surname-only when that is the conventional unambiguous answer, especially for artists, authors, or military figures in biographical clues anchored by dates, deaths, places, or famous works; use the full conventional name when the clue identifies the person mainly by office, title, role, or family relation that could fit multiple people with the same surname. Omit corporate suffixes such as "Corporation" when the common company name is enough, and reduce descriptive phrases to the requested property when the clue asks for "this kind/type of" something; for a property value such as a charge, color, direction, or polarity, give only the value word, not the category noun. - Read the clue’s relation before choosing the span. If the clue gives a title, subtitle, setting, or work description, ask whether it wants the creator/author rather than another part of the title or the series name; for bare descriptions of a book, film, series, or trilogy, including clues phrased only as "a trilogy/book/film/series set in..." or "about...", prefer the creator/author unless the wording explicitly asks for the work’s name, and give the creator’s full conventional name rather than surname-only when the clue does not already name them. - Do not pad the answer with explanatory context. Return only the minimal entity, person, place, property, or title needed for an exact match. Listing 1: GPT-5.5–SearchQA, Normal, round 9. ⬇ # Live Mathematical MCQ Heuristics ## Answer-Format Trap - When the expected answer is a single option label (A, B, C, D, E), output **exactly** that character with no surrounding text, no explanation, no Markdown, and no LaTeX. Never emit an empty string, a partial LaTeX fragment, or a quoted value. If the answer is ‘E‘, output ‘E‘ alone. ## Option Selection Strategy - Most livemath questions have a hidden meta-option: one of the listed options is correct, but a stronger result can be proven. If a question asks for "the strongest statement that can be proved," check whether any option claims a maximal or exhaustive classification when only a partial result is available. The canonical correct choice in many prompts is ‘One of the remaining options is correct, but a stronger result can be proven‘. ## Theorem-Level Precision - Check whether an option weakens the conclusion by dropping a characterization, equality clause, or full equivalence. - Check whether an option overstates the theorem by upgrading regularity, removing scale restrictions, or changing an existential statement into a universal one. ## Hypotheses - Verify the hypotheses and domain carefully. Distractors often keep the theorem shape but alter the required assumptions. - Pay close attention to equality cases, extremal conditions, and whether a result applies to the full family or only a restricted subfamily. ## Final Answer - Output the final answer as the single option label only. Listing 2: DeepSeek-V4-Pro–LiveMath, Normal, round 8. Part I: lines 30–86 ⬇ ### Diagnosis 31- **Sampling report**: Read 7/7 failed samples (3 sheet_level, 4 cell_level) and 1 passed sample. Dominated by execution mismatches (value mismatches and one syntax error). Chose to address a cluster of logic and syntax pitfalls across both ‘cell_level‘ and ‘sheet_level‘ tasks (support count: 4/7 failed). - **Syntax Error (‘ws.cell‘ keyword arguments)**: In ‘44017.md‘, the agent wrote ‘ws_data.cell(row=r, c)‘, which causes a fatal Python ‘SyntaxError: positional argument follows keyword argument‘. - **Duplicate Headers Overwriting**: In ‘130-9.md‘, the agent mapped headers using a simple dict (‘dest_headers[str(val).strip()] = col‘). When spreadsheets contain duplicate headers, this overwrites earlier indices and maps data to the wrong columns, causing a mismatch (got ’C’ instead of a numeric value). - **Nulls vs Zeros**: In ‘48983.md‘, the agent correctly mapped data but left empty destination cells as ‘None‘. The evaluator expected financial/count zeros (‘gt=0‘), leading to a failure (‘pred=None‘). - **Sorting Headers in Blocks**: In ‘142-19.md‘, the agent successfully identified contiguous blocks of data separated by blank rows, but blindly sorted the entire block including the header (‘gt=’Number’ pred=’A 37’‘), corrupting the structure. 36 ### Changes applied - File: ‘workspace/skill.md‘ - Snapshot: ‘workspace/.skillopt/history/$TS__before.md‘ - Added ‘## Python Syntax Pitfalls (CRITICAL)‘ warning against using positional column arguments in ‘ws.cell()‘. 41- Added ‘## Text Matching & Duplicate Headers‘ rule to enforce case-insensitivity and provide a safe ‘setdefault(name, []).append(c)‘ pattern for mapping columns. - Added ‘## Nulls vs. Zeros‘ to remind the agent to write ‘0‘ instead of ‘None‘ for counts and financial data. - Added ‘## Contiguous Blocks and Headers‘ edge-case warning to prevent sorting header rows within data blocks. ### Expected impact 46- Fixes fatal syntax errors from mixed kwargs in openpyxl cell access. - Improves column mapping reliability across all tasks with tabular structures and duplicate headers. - Fixes evaluation mismatches where empty cells should be explicitly zeroed out. - Prevents structural corruption when sorting separated tables/blocks in a single sheet. - Regressions to watch for: Strict zeroing might fail tests that strictly expect empty cells (‘None‘) if the agent misinterprets the context. 51 ### Verify next ‘bash # gate-style: full val, same as /skillopt-loop’s gate bash run.sh --skill workspace/skill.md --split val --eval_limit 0 --limit 0 56‘ Tell the user to compare the new ‘hard=‘ (and ‘soft=‘) line against the previous val run under ‘workspace/.skillopt/_eval_run/‘. # Spreadsheet Manipulation Skill (xlsx) 61## Overview This skill guides agents in manipulating Excel (.xlsx) spreadsheets using Python. **Primary libraries**: ‘openpyxl‘ (structure-preserving read/write). ‘pandas‘ can be used for data transformation, but is sometimes unavailable in the environment. Never use any other third-party libraries. 66 --- ## Environment Sandbox Workaround (CRITICAL) 71**CRITICAL:** The evaluation environment has a strict security sandbox that blocks ‘os.listdir(’/tmp’)‘. Because your script executes from ‘/tmp‘, standard library imports triggered by ‘openpyxl‘ (such as ‘numpy‘, ‘PIL‘, or ‘datetime‘) will frequently raise a ‘RuntimeError: SKILLOPT_GENERATED_CODE_FILE_SCOPE_BLOCK: os.listdir:/tmp‘. To prevent this crash and save execution turns, you **must** filter ‘/tmp‘ out of ‘sys.path‘ at the very beginning of your script, *before* importing ‘openpyxl‘. --- 76 ## Python Syntax Pitfalls (CRITICAL) Always use explicitly named keyword arguments for both row and column when accessing cells: ‘ws.cell(row=r, column=c)‘. Using a positional argument after a keyword argument (e.g., ‘ws.cell(row=r, c)‘) causes a fatal ‘SyntaxError‘ and immediately fails the task. 81--- ## Python Computation vs. Excel Formulas **CRITICAL:** Even if the instruction asks to "create an Excel formula" (e.g., "create a formula to sum...", "write a VLOOKUP..."), you **MUST compute the actual values in Python** and write the literal results back to the cells. Part I: lines 87–140 ⬇ ### Diagnosis - **Sampling report**: Read 7/7 failed samples (3 sheet_level, 4 cell_level) and 1 passed sample. Dominated by execution mismatches (value mismatches and one syntax error). Chose to address a cluster of logic and syntax pitfalls across both ‘cell_level‘ and ‘sheet_level‘ tasks (support count: 4/7 failed). - **Syntax Error (‘ws.cell‘ keyword arguments)**: In ‘44017.md‘, the agent wrote ‘ws_data.cell(row=r, c)‘, which causes a fatal Python ‘SyntaxError: positional argument follows keyword argument‘. - **Duplicate Headers Overwriting**: In ‘130-9.md‘, the agent mapped headers using a simple dict (‘dest_headers[str(val).strip()] = col‘). When spreadsheets contain duplicate headers, this overwrites earlier indices and maps data to the wrong columns, causing a mismatch (got ’C’ instead of a numeric value). 91- **Nulls vs Zeros**: In ‘48983.md‘, the agent correctly mapped data but left empty destination cells as ‘None‘. The evaluator expected financial/count zeros (‘gt=0‘), leading to a failure (‘pred=None‘). - **Sorting Headers in Blocks**: In ‘142-19.md‘, the agent successfully identified contiguous blocks of data separated by blank rows, but blindly sorted the entire block including the header (‘gt=’Number’ pred=’A 37’‘), corrupting the structure. ### Changes applied - File: ‘workspace/skill.md‘ 96- Snapshot: ‘workspace/.skillopt/history/$TS__before.md‘ - Added ‘## Python Syntax Pitfalls (CRITICAL)‘ warning against using positional column arguments in ‘ws.cell()‘. - Added ‘## Text Matching & Duplicate Headers‘ rule to enforce case-insensitivity and provide a safe ‘setdefault(name, []).append(c)‘ pattern for mapping columns. - Added ‘## Nulls vs. Zeros‘ to remind the agent to write ‘0‘ instead of ‘None‘ for counts and financial data. - Added ‘## Contiguous Blocks and Headers‘ edge-case warning to prevent sorting header rows within data blocks. 101 ### Expected impact - Fixes fatal syntax errors from mixed kwargs in openpyxl cell access. - Improves column mapping reliability across all tasks with tabular structures and duplicate headers. - Fixes evaluation mismatches where empty cells should be explicitly zeroed out. 106- Prevents structural corruption when sorting separated tables/blocks in a single sheet. - Regressions to watch for: Strict zeroing might fail tests that strictly expect empty cells (‘None‘) if the agent misinterprets the context. ### Verify next ‘bash 111# gate-style: full val, same as /skillopt-loop’s gate bash run.sh --skill workspace/skill.md --split val --eval_limit 0 --limit 0 ‘ Tell the user to compare the new ‘hard=‘ (and ‘soft=‘) line against the previous val run under ‘workspace/.skillopt/_eval_run/‘. 116# Spreadsheet Manipulation Skill (xlsx) ## Overview This skill guides agents in manipulating Excel (.xlsx) spreadsheets using Python. 121**Primary libraries**: ‘openpyxl‘ (structure-preserving read/write). ‘pandas‘ can be used for data transformation, but is sometimes unavailable in the environment. Never use any other third-party libraries. --- 126## Environment Sandbox Workaround (CRITICAL) **CRITICAL:** The evaluation environment has a strict security sandbox that blocks ‘os.listdir(’/tmp’)‘. Because your script executes from ‘/tmp‘, standard library imports triggered by ‘openpyxl‘ (such as ‘numpy‘, ‘PIL‘, or ‘datetime‘) will frequently raise a ‘RuntimeError: SKILLOPT_GENERATED_CODE_FILE_SCOPE_BLOCK: os.listdir:/tmp‘. To prevent this crash and save execution turns, you **must** filter ‘/tmp‘ out of ‘sys.path‘ at the very beginning of your script, *before* importing ‘openpyxl‘. 131 --- ## Python Syntax Pitfalls (CRITICAL) 136Always use explicitly named keyword arguments for both row and column when accessing cells: ‘ws.cell(row=r, column=c)‘. Using a positional argument after a keyword argument (e.g., ‘ws.cell(row=r, c)‘) causes a fatal ‘SyntaxError‘ and immediately fails the task. --- ## Python Computation vs. Excel Formulas Listing 3: Gemini-3.1-Pro–SpreadsheetBench, Normal, round 7 (operational excerpt). Appendix E Test-Time Scaling and Verifier Diagnostics Test-Time Scaling Calculation Benchmark Evolved Parallel Sequential Gap SearchQA +2.29+2.29 +1.86+1.86 +0.14+0.14 0.430.43 SpreadsheetBench +35.23+35.23 +4.27+4.27 −5.34-5.34 30.9630.96 Table A25: Primary test-time-scaling differences at maximum budget. Evolved, Parallel, and Sequential report percentage-point changes from the parent; Gap is Evolved minus Parallel. For item i, let zik∈0,1z_ik∈\0,1\ be the frozen verifier outcome of attempt k, and let DkD_k be the outcome-independent set of items assigned a kkth attempt, with Dk+1⊆DkD_k+1 D_k and D1D_1 equal to the evaluation panel. On common support DKD_K, Parallel Sampling uses oracle any-success, whereas Sequential Refinement uses only the last response: Spar(K) S_par(K) =100|DK|∑i∈DKmax1≤k≤Kzik, = 100|D_K| _i∈ D_K _1≤ k≤ Kz_ik, Sseq(K) S_seq(K) =100|DK|∑i∈DKziK. = 100|D_K| _i∈ D_Kz_iK. For the item-specific allocation KiK_i used by the full SearchQA panel, the corresponding dynamic scores are Spardyn=100n∑i=1nmax1≤k≤Kizik,Sseqdyn=100n∑i=1nziKi.S_par^dyn= 100n _i=1^n _1≤ k≤ K_iz_ik, S_seq^dyn= 100n _i=1^nz_iK_i. Let ziparentz_i^parent and zievoz_i^evo be the frozen verifier outcomes of one-call parent and evolved-skill execution. Their scores are Sparent=100n∑iziparent,Sevo=100n∑izievo.S_parent= 100n _iz_i^parent, S_evo= 100n _iz_i^evo. For m∈par,seq,evom∈\par,seq,evo\, we report changes from the one-call parent as Δm=Sm−Sparent _m=S_m-S_parent. Because attempt 1 reuses the frozen parent output, the number of new target calls for one control is Cadd=∑k=2Kmax|Dk|,Call=|D1|+Cadd,C_add= _k=2^K_ |D_k|, C_all=|D_1|+C_add, where CallC_all counts all score-bearing attempts, including the reused baseline. For fixed-panel controls, Cadd=n(Kmax−1)C_add=n(K_ -1). Parallel oracle any-success assumes perfect post-hoc selection and is therefore an upper bound, not a deployable pass@1 estimator. For Figure A2, Bevo=2,750B_evo=2,750 counts target-model calls in the selected SearchQA Normal evolution run. Its amortized selected-run call metric over n deployments is Cavg(n)=1+Bevo/nC_avg(n)=1+B_evo/n, where one is the evolved-skill deployment call. This metric covers the selected evolution run and subsequent deployments. Benchmark n K_ (||,…,||) (|D_2|,…,|D_K_ |) C_add C_all SearchQA 1,400 6 (1,400,1,375,915,686,548)(1,400,1,375,915,686,548) 4,924 6,324 OfficeQA 148 6 (148,148,148,148,148)(148,148,148,148,148) 740 888 DocVQA 374 4 (374,374,374)(374,374,374) 1,122 1,496 LiveMath 106 8 (106,106,106,106,106,106,106)(106,106,106,106,106,106,106) 742 848 SpreadsheetBench 281 4 (281,281,281)(281,281,281) 843 1,124 Table A26: GPT-5.5 test-time-scaling allocations and call counts. SearchQA uses a nested, outcome-independent allocation: 25 items receive Ki=2K_i=2, 460 receive 3, 229 receive 4, 138 receive 5, and 548 receive 6 attempts. The other benchmarks evaluate every item through KmaxK_ , with identical allocations for Parallel and Sequential. Figure A2: GPT-5.5 SearchQA accuracy and amortized evolution cost across deployment scales. Each x tick pairs deployment size n with Parallel budget K on the corresponding nested subset. The upper panel compares oracle Accuracy@K with one-call evolved-skill deployment; the lower panel reports 1+2,750/n1+2,750/n calls per deployment for the selected evolution run. Parallel uses 2,740–2,750 additional calls at n∈548,686,915,1375n∈\548,686,915,1375\ and 1,400 calls at the full-panel K=2K=2 point. Figure A3: Deployment-scale reaggregation of GPT-5.5 TTS outputs under a fixed, outcome-independent SHA-256 item order. KmaxK_ is 6, 8, and 4 for OfficeQA, LiveMath, and SpreadsheetBench, respectively. These curves isolate deployment behavior; Figure A2 separately gives the SearchQA evolution-cost comparison. Benchmark (panel) Method Evolved = K=1 = K=2 = K=3 = K=4 = K=5 = K=6 = K=7 = K=8 SearchQA (common 548) Parallel oracle 417 (76.09) 410 (74.82) 419 (76.46) 421 (76.82) 421 (76.82) 422 (77.01) 423 (77.19) – – Sequential last 410 (74.82) 411 (75.00) 410 (74.82) 411 (75.00) 410 (74.82) 411 (75.00) – – OfficeQA (148) Parallel oracle 103 (69.59) 93 (62.84) 99 (66.89) 102 (68.92) 103 (69.59) 103 (69.59) 103 (69.59) – – Sequential last 93 (62.84) 95 (64.19) 94 (63.51) 92 (62.16) 94 (63.51) 92 (62.16) – – DocVQA (374) Parallel oracle 343 (91.71) 344 (91.98) 347 (92.78) 350 (93.58) 352 (94.12) – – – – Sequential last 344 (91.98) 344 (91.98) 343 (91.71) 342 (91.44) – – – – LiveMath (106) Parallel oracle 45 (42.45) 52 (49.06) 66 (62.26) 71 (66.98) 71 (66.98) 72 (67.92) 73 (68.87) 73 (68.87) 73 (68.87) Sequential last 52 (49.06) 56 (52.83) 59 (55.66) 53 (50.00) 51 (48.11) 47 (44.34) 44 (41.51) 51 (48.11) SpreadsheetBench (281) Parallel oracle 241 (85.77) 142 (50.53) 148 (52.67) 150 (53.38) 154 (54.80) – – – – Sequential last 142 (50.53) 138 (49.11) 134 (47.69) 127 (45.20) – – – – SearchQA full dynamic panel Parent SparentS_parent Evolved SevoS_evo S_par^dyn S_seq^dyn Successes (score, n=1,400n=1,400) 1,059 (75.64) 1,091 (77.93) 1,085 (77.50) 1,061 (75.79) Change from parent (points) 0.00 +2.29+2.29 +1.86+1.86 +0.14+0.14 Table A27: Complete GPT-5.5 test-time-scaling results. Entries report successes and hard-score percentages. SearchQA common support contains 548 items receiving all six attempts; the dynamic panel aggregates item-specific budgets over all 1,400 items. SpreadsheetBench uses the selected Fail-only skill. Each evolved skill receives one call per item; DocVQA compares independent parent executions. Output-Locked Verifier Sensitivity We rescore outcome-blind 100-item panels of GPT-5.5 parent and evolved-skill outputs with the original verifier V0V_0 and a separately fixed V1V_1. Let ΔVj=ScoreVj(sevo)−ScoreVj(sparent) _V_j=Score_V_j(s_evo)-Score_V_j(s_parent). With outputs fixed, ΔV1−ΔV0 _V_1- _V_0 isolates differential rescoring. The verifiers disagree on 48/1,000 verdicts (4.8%), while the measured gain is unchanged for SpreadsheetBench, DocVQA, and LiveMath, changes by 1.0 point for OfficeQA, and changes from −3.0-3.0 to 0.00.0 for SearchQA. This output-locked comparison quantifies differential rescoring under one separately fixed alternative verifier per benchmark. Output-locked verifier sensitivity Benchmark _V_0 _V_1 Diff. SearchQA −3.0-3.0 0.00.0 +3.0+3.0 SpreadsheetBench (N) +32.0+32.0 +32.0+32.0 0.00.0 OfficeQA +5.0+5.0 +6.0+6.0 +1.0+1.0 DocVQA 0.00.0 0.00.0 0.00.0 LiveMath −8.0-8.0 −8.0-8.0 0.00.0 Benchmark V_0 V_1 V_1 identifier SearchQA Normalized exact match against gold aliases Blind GPT-5.5 semantic-equivalence judge with a fixed JSON verdict schema; three judgments per output Prompt SHA 9ec4149d OfficeQA Adapter-normalized exact match Zero-call decimal-exact comparison for numeric answers and canonical normalized-text equality otherwise Verifier SHA 15e15cad SpreadsheetBench Benchmark workbook test cases Zero-call, data_only strict type-and-value equality on every required answer cell Verifier SHA 15e15cad LiveMath Parsed option-label exact match Zero-call exact option label or uniquely matched normalized option text Verifier SHA 15e15cad DocVQA ANLS success at threshold 0.999 Zero-call membership in the set of punctuation-stripped, case-folded, normalized gold aliases Verifier SHA 15e15cad Table A28: Output-locked GPT-5.5 verifier sensitivity and verifier definitions. The upper table reports ΔV0 _V_0, ΔV1 _V_1, and their difference on 100 common non-abstain items per benchmark; the lower table specifies both verdict functions and the alternative-verifier identifier. SpreadsheetBench uses the Normal branch in this diagnostic; N denotes Normal. Scores are percentages and differences are percentage points.