Paper deep dive

SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering

Jingzhi Gong, Ruizhen Gu, Zhiwei Fei, Yazhuo Cao, Lukas Twist, Alina Geiger, Shuo Han, Dominik Sobania, Federica Sarro, Jie M. Zhang

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 17

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/14/2026, 1:52:19 AM

Summary

SkillMOO is a multi-objective optimization framework designed to automatically evolve LLM-based coding agent skill bundles. By utilizing a solver-optimizer loop with NSGA-II survivor selection, it optimizes for pass rate and cost. Empirical results on SkillsBench tasks demonstrate significant improvements in performance and cost-efficiency, with pattern analysis identifying pruning and substitution as the most effective strategies for skill bundle refinement.

Entities (5)

NSGA-II · algorithm · 100%SkillMOO · framework · 100%SkillsBench · benchmark · 100%GLM-5 · llm · 95%Software Engineering · domain · 95%

Relation Signals (4)

SkillMOO → evaluatedon → SkillsBench

confidence 100% · We evaluate SkillMOO on three SkillsBench SE tasks

SkillMOO → optimizes → Skill Bundles

confidence 100% · SkillMOO, a multi-objective optimization framework that automatically evolves skill bundles

SkillMOO → uses → NSGA-II

confidence 100% · SkillMOO... combines LLM-proposed edits and NSGA-II survivor selection

GLM-5 → powers → SkillMOO

confidence 90% · We use GLM-5 [9] for both task solver and skill optimizer agents.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Agent skills provide modular, task-specific guidance for LLM- based coding agents, but manually tuning skill bundles to balance success rate, cost, and runtime is expensive and fragile. We present SkillMOO, a multi-objective optimization framework that automatically evolves skill bundles using LLM-proposed edits and NSGA-II survivor selection: a solver agent evaluates candidate skill bundles on coding tasks and an optimizer agent proposes bundle edits based on failure analysis. On three SkillsBench software engineering tasks, SkillMOO improves pass rate by up to 131% while reducing cost up to 32% relative to the best baseline per task at low optimization overhead. Pattern analysis reveals pruning and substitution as primary drivers of improvement, suggesting effective bundles favor minimal, focused content over accumulated instructions.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

17,029 characters extracted from source content.

Expand or collapse full text

11institutetext: King’s College London 11email: jingzhi.gong, yazhuo.cao, lukas.twist, jie.zhang@kcl.ac.uk 22institutetext: Queen’s University Belfast 22email: r.gu@qub.ac.uk 33institutetext: Nanjing University 33email: zhiweifei@smail.nju.edu.cn 44institutetext: Johannes Gutenberg University Mainz 44email: geiger@uni-mainz.de 55institutetext: University College London 55email: shuo.han.25, f.sarro@ucl.ac.uk 66institutetext: University of Duisburg-Essen 66email: dominik.sobania@uni-due.de SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering Jingzhi Gong Ruizhen Gu Zhiwei Fei Yazhuo Cao Lukas Twist Alina Geiger Shuo Han Dominik Sobania Federica Sarro Jie M. Zhang Abstract Agent skills provide modular, task-specific guidance for LLM-based coding agents, but manually tuning skill bundles to balance success rate, cost, and runtime is expensive and fragile. We present SkillMOO, a multi-objective optimization framework that automatically evolves skill bundles using LLM-proposed edits and NSGA-I survivor selection: a solver agent evaluates candidate skill bundles on coding tasks and an optimizer agent proposes bundle edits based on failure analysis. On three SkillsBench software engineering tasks, SkillMOO improves pass rate by up to 131% while reducing cost up to 32% relative to the best baseline per task at low optimization overhead. Pattern analysis reveals pruning and substitution as primary drivers of improvement, suggesting effective bundles favor minimal, focused content over accumulated instructions. 1 Introduction Large language model (LLM)-based coding agents have emerged as powerful tools for software engineering (SE) tasks, from bug fixing to code migration [5]. As introduced by Anthropic, agent skills are portable folders that package instructions, scripts, and other resources that an agent can load on demand to improve performance on specialized tasks [2]. To systematically evaluate their utility, SkillsBench [4] benchmarks agent skills across diverse domains and shows that skills can improve task performance overall. However, SkillsBench finds that skill gains on SE tasks are modest (+4.5%+4.5\%) [4], and practitioners similarly observe that skill flexibility creates uncertainty about what works best, poor skill bundles may waste agent overhead, and skills need continuous refinement around failure cases [7]. Manual tuning of skill bundles to address these challenges is expensive, fragile, and often requires domain expertise that may not transfer across tasks. This motivates search-based, data-driven skill optimization, and thereby we propose SkillMOO, a multi-objective optimization (MOO) framework that automatically evolves task-specific skill bundles for SE tasks using LLM-proposed edits and NSGA-I survivor selection on pass rate and cost: a task solver agent evaluates candidate skill bundles on coding tasks and a skill optimizer agent proposes bundle edits based on failure analysis. We evaluate SkillMOO on three SkillsBench SE tasks against original skill bundles and no skill baselines, and in summary, our main contributions are: • We propose SkillMOO, a MOO framework that combines LLM-proposed edits and NSGA-I survivor selection in a solver-optimizer loop. • We provide empirical evidence on three SkillsBench tasks demonstrating that skill optimization can simultaneously improve pass rate and reduce cost. • We analyze recurring skill-edit patterns and identify pruning and substitution as the primary drivers of observed improvements, offering practical insights for skill bundle design. • We release a replication package with data, scripts, and skill artifacts to support further research. Related Work. SkillsBench [4] introduced the first systematic paired evaluation of skills across diverse tasks, and SWE-Skills-Bench [3] found that many public SWE skills provide little benefit in practical SE tasks. SkillRouter [11] frames skill usage as a retrieval problem and shows that scalable deployment requires accurate routing over large skill pools. Meanwhile, recent skill-evolution approaches like EvoSkill [1], Meta Context Engineering [8], and EvoSkills [10] demonstrate that skills can themselves be optimized through iterative refinement. Our work builds on these directions but targets a different gap: (1) we target SE tasks in SkillsBench, and (2) we cast improvement as multi-objective search using LLM-proposed bundle edits and NSGA-I survivor selection on pass rate and cost, plus overhead and edit-pattern analysis. Figure 1: SkillMOO workflow: solver-optimizer loop with evolving skill bundles. 2 SkillMOO SkillMOO uses a two-agent iterative workflow with an evolving optimizer skill, as illustrated in Figure˜1. Generation 0 starts from the original skill bundle, and the task solver agent executes the task to obtain pass rate, cost, and error traces. Guided by the current optimizer-skill prompt and the failure evidence from the current parent candidate, the skill optimizer agent proposes an edited child skill bundle through operations such as pruning, substitution, reordering, and rewriting, and updates the optimizer skill itself for the next generation. The edited child bundle is then re-evaluated by the task solver, added to the search archive, and the loop repeats until the generation budget is reached. During search, SkillMOO optimizes candidate skill bundles(ℬB) with a bi-objective formulation: minb∈ℬ⁡(b)=[−pass(b),cost(b)] _b \;f(b)= [-pass(b),\;cost(b) ] where b is the skill, pass(b)pass(b) is test pass rate and cost(b)cost(b) is LLM inference cost. After each child evaluation, all archived candidates are ranked with NSGA-I survivor selection using non-dominated sorting and crowding-distance tie-breaking on these two objectives; the non-dominated candidates form the current Pareto front. We also enforce a pass-preservation guard that rejects a proposed child when its pass rate drops by more than 0.05 from its parent. Runtime is tracked as a secondary metric and used when choosing the final reporting seed via lexicographic order. 3 Experimental Setup Table 1: Tasks, Skill Bundles, and Benchmark Tests for the Evaluation of SkillMOO ID Task Name (Description) # Skill Bundle Skills (Count) Benchmark Tests (Count) Task 1 fix-build-agentops (repair Python build failures and produce a valid patch + run note) 8 build/dependency triage (3); pytest/testing diagnosis (3); patch+diff workflow (1); CI/replay/report hygiene (1) env execution coverage (18); patch safety (7); runtime instrumentation (7); regression tests (5); artifact/replay sanity (3) Task 2 python-scala-translation (translate Python pipeline logic to Scala with equivalent behavior) 6 collections/libraries mapping (2); functional/OOP structure guidance (2); syntax mapping (1); language idioms (1) source/API contract tests (20); behavioral tests (17); compile/unit-test gates (3) Task 3 spring-boot-jakarta-migration (migrate Spring Boot app to Jakarta APIs without regressions) 5 framework/dependency modernization (2); namespace migration (1); security migration (1); REST client migration (1) RestClient migration tests (10); namespace migration tests (9); security migration tests (8); feature preservation tests (6); dependency update tests (5); build/compile tests (2) We selected three tasks with the largest original skill bundles from SkillsBench [4] to focus on settings where manual skill tuning is hardest and quality-cost-runtime tradeoffs are most pronounced. We limited the study to three tasks due to time constraints. Because the default verifier111The set of automatic tests used to judge each solution. coverage in this subset is minimal (roughly 3-10 tests), we enhanced each task to 40 tests via GPT-5.4 to provide denser behavior and safety assertions while preserving compile/build gates, as summarized in Table˜1. We use GLM-5 [9] for both task solver and skill optimizer agents. Due to time constraints, the search uses population size 1 and runs for 5 generations (generation 0 seed evaluation plus generations 1–4 optimization steps). Timeout settings are 900 seconds for both agents. Final selection from the Pareto front follows preference order: maximize pass rate, then minimize cost, then runtime. Each method/task is executed 10 times, and Scott-Knott Effect Size Difference (ESD) pass-rate ranks [6] are computed from these 10-run vectors. We evaluate three methods, SkillMOO, ori_skill (original skill bundle), and no_skill (no guidance), with three research questions (RQs): ➤ RQ1. How do final SkillMOO candidates compare with baselines on pass rate, cost, and runtime? ➤ RQ2. Is optimization economically worthwhile accounting for overhead? ➤ RQ3. Which skill-edit patterns recur in improved candidates? 4 Results and Analysis RQ1: Effectiveness of SkillMOO. Table˜2 reports ten-run means with GLM-5. SkillMOO achieves higher pass rate than ori_skill on all three tasks, with improvements ranging from 2.1% (Task 3) to 131.2% (Task 1). On Task 1, SkillMOO raises pass rate from 0.16 to 0.37 (+131.2%) while reducing cost by 31.7% (1.10 vs 1.61 USD) and runtime by 23.6% (1060.9s vs 1388.8s). On Task 2, pass rate improves from 0.39 to 0.51 (+30.8%), with cost reduced by 5.4% (1.06 vs 1.12 USD) and runtime by 8.0% (612.4s vs 665.3s). On Task 3, pass rate rises from 0.97 to 0.99 (+2.1%), while cost drops by 19.4% (1.25 vs 1.55 USD) and runtime drops by 29.8% (317.4s vs 452.0s). Scott-Knott ESD assigns SkillMOO the top pass-rate rank for each task, as in table˜2. Against no_skill, SkillMOO retains higher pass rate on every task, but cost and runtime are often higher. This pattern is expected because SkillMOO runs with substantive skill guidance rather than an empty prompt. Table 2: Effectiveness against baselines, reported with mean ± standard deviation (SD) and Scott-Knott ESD rank for pass rate (rpr_p). Task Method #Skills Used Pass rate Cost (USD) Runtime (s) rpr_p Task 1 SkillMOO 4 0.37± 0.16 1.10± 0.33 1060.9± 280.0 1 ori_skill 8 0.16± 0.08 1.61± 0.41 1388.8± 97.8 2 no_skill 0 0.10± 0.00 1.06± 0.71 989.2± 265.5 3 Task 2 SkillMOO 2 0.51± 0.12 1.06± 0.25 612.4± 214.2 1 ori_skill 6 0.39± 0.06 1.12± 0.36 665.3± 189.2 2 no_skill 0 0.39± 0.16 0.85± 0.31 455.2± 232.7 2 Task 3 SkillMOO 5 0.99± 0.01 1.25± 0.49 317.4± 36.7 1 ori_skill 5 0.97± 0.00 1.55± 0.27 452.0± 216.0 2 no_skill 0 0.91± 0.01 1.14± 0.23 685.5± 533.1 3 RQ1 answer: Against the best static baseline per task, SkillMOO improves pass rate on all tasks (+2.1% to +131.2%) while lowering cost (-5.4% to -31.7%), with statistical significance. RQ2: Optimization Efficiency. Table˜3 reports the full optimization cost/runtime and resulting hypervolume (HV) uplift over ori_skill, where HV is computed in a two-objective space: passed tests (maximize) and cost (minimize). The last column shows Cost/Δ % of $0.0011 for Task 1, $0.0043 for Task 2, and $0.0059 for Task 3, so each percentage point of relative HV gain costs well under one U.S. cent in search spend. Notably, Task 2 has the lowest baseline HV (0.0056) yet remains modest at $0.0043 per HV% improvement because relative uplift in pass–cost space stays large. Table 3: SkillMOO optimization overhead (generation 1–4), two-objective HV improvement (Δ (%)), and optimization efficiency (Cost/Δ %). Task Opt. Cost (USD) Opt. Runtime (s) HV (ori_skill) HV (SkillMOO) Δ (%) Cost/Δ % Task 1 2.2676 2305.00 0.0081 0.1783 2110 0.0011 Task 2 1.8611 1545.08 0.0056 0.0296 430 0.0043 Task 3 1.7641 867.40 0.0078 0.0311 301 0.0059 RQ2 answer: SkillMOO improves multi-objective quality on all tasks (HV gains of 301%–2110%), with optimization efficiency strongest on Task 1 ($0.0011 per %HV gain). RQ3: Skill-Edit Pattern Evidence. For pattern analysis, we enumerate all logged skill-bundle edits and group them by operation description text produced by the optimizer agent during search; the RQ3 summarization step then groups them deterministically with GPT-5.4. Table˜4 reports operation description, edit frequency, and improvement counts measured against ori_skill baselines. Table 4: Pattern analysis of skill-bundle operations over all available candidates. Operation Description #Edits Pass↑ Cost↓ Time↓ Bundle pruning (remove skill blocks) 7 5/7 7/7 4/7 Bundle substitution (swap skill blocks) 7 5/7 7/7 4/7 Bundle expansion (add skill blocks) 5 0/5 5/5 5/5 Remove RestTemplate mentions (incl. comments/Javadocs) 2 2/2 2/2 1/2 Remove unused guidance sections (never-invoked) 2 0/2 2/2 2/2 Add OOP translation guidance (class/trait/component) 3 0/3 3/3 3/3 Dead-weight pruning (keep effective, drop unused) 1 0/1 1/1 1/1 Migration sequence reorder (bootstrap/build first) 1 1/1 1/1 0/1 Functional-style translation guidance 1 0/1 1/1 1/1 Explicit remove_skill (telemetry only) 1 1/1 1/1 1/1 Explicit reorder_bundle (telemetry only) 1 0/1 1/1 1/1 Table˜4 reveals that bundle pruning and bundle substitution are the most frequent operations with 7 edits each, and both achieve cost reduction in 7/7 cases. Bundle expansion shows 0/5 pass improvements, indicating that adding new guidance rarely helps pass rate. These observational patterns suggest that optimization often benefits from removing irrelevant guidance and selectively replacing misaligned content, rather than adding new instructions. Task-specific patterns also emerge222Detailed task-specific results are provided in the replication package.. For Task 3, removing RestTemplate mentions (including comments/Javadocs) directly addresses strict verifier assertions (2/2 pass improvements). For Task 2, adding OOP translation guidance targets missing structural elements but does not improve pass rate (0/3). RQ3 answer: Pruning and substitution operations dominate successful edits; cost reductions are consistent across operations, while pass improvements depend on task-specific misalignment in the original bundle. Threats to Validity. First, our study uses three selected tasks from a fixed 16-task SE pool, limiting external validity beyond this setting. Second, RQ3 provides observational pattern evidence, not causal attribution of individual edits. Third, verifier expansion to 40 tests via GPT-5.4 may introduce model-linked bias even though compile/build gates are preserved. Finally, we report results for one model (GLM-5) and the three tasks with the largest original bundles; generalization to other LLMs and smaller tasks requires further study. 5 Conclusion We presented SkillMOO, a framework that evolves agent skill bundles with LLM-proposed edits and NSGA-I survivor selection. The study demonstrates the effectiveness and efficiency of using multi-objective search over skills to improve software engineering agent outcomes compared with fixed bundles, and suggests pruning and substitution matter more than simply adding more guidance. Availability. Replication package at: https://doi.org/10.5281/zenodo.19488943. References [1] S. Alzubi et al. (2026) EvoSkill: automated skill discovery for multi-agent systems. External Links: 2603.02766, Link Cited by: §1. [2] Anthropic Introducing agent skills. Note: https://claude.com/blog/skillspublished October 16, 2025 Cited by: §1. [3] T. Han, Y. Zhang, W. Song, C. Fang, Z. Chen, Y. Sun, and L. Hu (2026) SWE-skills-bench: do agent skills actually help in real-world software engineering?. External Links: 2603.15401, Link Cited by: §1. [4] X. Li, W. Chen, Y. Liu, et al. (2026) SkillsBench: benchmarking how well agent skills work across diverse tasks. External Links: 2602.12670, Link Cited by: §1, §1, §1, §3. [5] J. Liu, K. Wang, Y. Chen, X. Peng, Z. Chen, L. Zhang, and Y. Lou (2024) Large language model-based agents for software engineering: a survey. ACM Transactions on Software Engineering and Methodology. Cited by: §1. [6] A. J. Scott and M. Knott (1974) A cluster analysis method for grouping means in the analysis of variance. Biometrics, p. 507–512. Cited by: §3. [7] T. Shihipar Lessons from building claude code: how we use skills. Note: https://x.com/trq212/status/2033949937936085378published March 17, 2025 Cited by: §1. [8] H. Ye, X. He, V. Arak, H. Dong, and G. Song (2026) Meta context engineering via agentic skill evolution. External Links: 2601.21557, Link Cited by: §1. [9] A. Zeng, X. Lv, Z. Hou, Z. Du, et al. (2026) GLM-5: from vibe coding to agentic engineering. External Links: 2602.15763, Link Cited by: §3. [10] H. Zhang, S. Fan, H. P. Zou, Y. Chen, Z. Wang, J. Zhou, C. Li, W. Huang, Y. Yao, K. Zheng, X. Liu, X. Li, and P. S. Yu (2026) EvoSkills: self-evolving agent skills via co-evolutionary verification. External Links: 2604.01687, Link Cited by: §1. [11] Y. Zheng, Z. Zhang, C. Ma, Y. Yu, J. Zhu, Y. Wu, T. Xu, B. Dong, H. Zhu, R. Huang, and G. Yu (2026) SkillRouter: skill routing for llm agents at scale. External Links: 2603.22455, Link Cited by: §1.

Instant research discovery