Paper deep dive
Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance
Chen Shen, Estevam Hruschka
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/9/2026, 4:43:48 AM
Summary
This longitudinal study analyzes the maintenance of public AI agent skill files (SKILL.md) across five repositories from October 2025 to June 2026. The research finds that skill maintenance is a human-governed, AI-assisted process rather than fully autonomous. Key findings include: 62% of substantive edits feature AI co-authorship trailers, yet every edit is authored or merged by a named human. Maintenance operations are predominantly enhancement (60%) rather than corrective (38%), with content expansion and factual correction being the most common operations. The study highlights that reliable coding of 'rule-likeness' in commits remains an open problem and provides a corpus and codebooks for future automated skill curator evaluation.
Entities (13)
Relation Signals (12)
Chen Shen â affiliatedwith â Megagon Labs
confidence 99% · Chen Shen & Estevam Hruschka Megagon Labs
Estevam Hruschka â affiliatedwith â Megagon Labs
confidence 99% · Chen Shen & Estevam Hruschka Megagon Labs
Chen Shen â authored â Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance
confidence 99% · Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance Chen Shen & Estevam Hruschka
Estevam Hruschka â authored â Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance
confidence 99% · Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance Chen Shen & Estevam Hruschka
SKILL.md â maintainedin â trailofbits/skills
confidence 96% · we purposively selected five public repositories... trailofbits/skills... covering 143 SKILL.md files
SKILL.md â maintainedin â getsentry/skills
confidence 96% · we purposively selected five public repositories... getsentry/skills... covering 143 SKILL.md files
SKILL.md â maintainedin â obra/superpowers
confidence 96% · we purposively selected five public repositories... obra/superpowers... covering 143 SKILL.md files
SKILL.md â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Lifelong LLM agents increasingly rely on external skill artifacts as one element for preserving and reusing capabilities over time. These skills (usually portable Markdown files such as this http URL) describe when and how to apply a capability and must be corrected, expanded, and consolidated as tools and usage patterns shift over deployment. Recent work seeks to automate skill curation, but it largely evaluates against automated baselines and treats human maintenance as an unmeasured bottleneck. We study that missing process directly. We mine the full commit histories of five public AI-skill repositories, a purposive sample of AI-tooling organizations, covering 873 commits, 143 skill files, and 254 substantive post-creation edits from October 2025 to June 2026. We code each edit with pre-registered governance, operation, and trigger-evidence codebooks. Three findings emerge. First, every substantive edit is authored or merged through a named human account, while 62% carry an AI co-author trailer, with large repository-level variation. Second, these edits are genuine curation: an audited sample shows that most change skill content, and the coded operations are dominated by additions and corrections. Third, a pre-registered rule-likeness axis fails its reliability gate; reliably coding rule-likeness from commit artifacts remains an open measurement problem. We release the corpus, codebooks, mining scripts, and a replay protocol for automated skill curators. For self-evolving agents, public skill maintenance currently looks less like an autonomous pipeline than a human-governed, AI-assisted loop that future curators must measure against and operate within.
Tags
Links
- Source: https://arxiv.org/abs/2609.05677v1
- Canonical: https://arxiv.org/abs/2609.05677v1
Trouble viewing inline? Open PDF directly â
Full Text
73,224 characters extracted from source content.
Expand or collapse full text
Accepted at the COLM 2026 Workshop on Lifelong Agents Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance Chen Shen & Estevam Hruschka Megagon Labs chen s,estevam@megagon.ai Abstract Lifelong LLM agents increasingly rely on external skill artifacts as one element for preserving and reusing capabilities over time. These skills (usu- ally portable Markdown files such asSKILL.md) describe when and how to apply a capability and must be corrected, expanded, and consolidated as tools and usage patterns shift over deployment. Recent work seeks to automate skill curation, but it largely evaluates against automated baselines and treats human maintenance as an unmeasured bottleneck. We study that missing process directly. We mine the full commit histories of five public AI-skill repositories, a purposive sample of AI-tooling organizations, covering 873 commits, 143 skill files, and 254 substantive post-creation edits from October 2025 to June 2026. We code each edit with pre-registered governance, operation, and trigger-evidence codebooks. Three findings emerge. First, every substantive edit is authored or merged through a named human account, while 62% carry an AI co-author trailer, with large repository-level variation. Second, these edits are genuine curation: an audited sample shows that most change skill content, and the coded opera- tions are dominated by additions and corrections. Third, a pre-registered rule-likeness axis fails its reliability gate; reliably coding rule-likeness from commit artifacts remains an open measurement problem. We release the corpus, codebooks, mining scripts, and a replay protocol for automated skill curators. For self-evolving agents, public skill maintenance currently looks less like an autonomous pipeline than a human-governed, AI-assisted loop that future curators must measure against and operate within. 1 Introduction Agent skills are becoming an important element in the operational memory of LLM agents: small, portable Markdown packages that tell an agent when and how to apply a capability and can play a key role in the self-evolving and self-improving capabilities of the agents. Unlike benchmark prompts or one-off demonstrations, skills live under version control. They are corrected when tools change, expanded when maintainers discover missing cases, reorganized when instructions become unwieldy, and occasionally retired when capabilities become obsolete. Recent 2025â2026 work automates this curation: systems learn to edit skill documents from rollouts (Yang et al., 2026), distill trajectories into skills (Ni et al., 2026), and train curators that evolve an external skill repository (Ouyang et al., 2026). These systems are routinely evaluated against automated or memory-based baselines; manual maintenance is treated mainly as a scalability concern, âdemand[ing] huge human expertise and cannot scaleâ (Ouyang et al., 2026), and is, to our reading, mostly unmeasured. Even the strongest evidence that humans matter concerns an endpoint: a recent benchmark reports human-authored skills at 74.5% task success, while no automated method averages above 31.1% across the evaluated models (Zhong et al., 2026), but says nothing about the human process that produced them. In this paper, we measure that process. This leaves a prerequisite empirical question: who maintains agent skills in public reposito- ries, and what does that maintenance consist of? Closest to us, a concurrent study of agent context files (AGENTS.md/README-style) finds they âevolve like configuration code through frequent, small additionsâ (Chatlatanagulchai et al., 2025). But it studies a different artifact 1 arXiv:2609.05677v1 [cs.CL] 4 Sep 2026 Accepted at the COLM 2026 Workshop on Lifelong Agents and characterizes maintenance only at that coarse level, without a coded operation taxon- omy, commit-authorship governance, or failure-trigger evidence. A marketplace analysis of skills is cross-sectional (Ling et al., 2026), and decision-history methods construct their histories synthetically (Li & Hu, 2026). We are not aware of a longitudinal study of public Markdown skill-file maintenance histories and their curation operations. To characterize this process, we mine git histories directly. From a purposive sample of five public AI-skill repositories (873 commits at pinned clones), we track 143SKILL.mdfiles through 631 commit-file history records, isolate 254 substantive post-creation maintenance edits, and code each under three pre-registered codebooks. Our contributions are: âąA released corpus and protocol for mining skill-file maintenance: commit-level records, three codebooks, and reproducible scripts (§3). 1 The corpus can support replay evaluations of automated skill-curation systems (protocol in Appendix E). âąThe governance finding (§6): trailer-visible AI co-authorship under human repos- itory control. Of the substantive edits, 62% carry an AI co-author trailer, 38% are AI-trailer-absent, and every edit is authored or merged through a named human account, which is all we mean by human-governed. The mix differs sharply by organization, a disclosure-and-merge-culture split as much as an AI-usage one. âąA coded account of maintenance (§4, §6): operation types, observable failure evidence, and descriptive size/cadence trajectories (§5). âąA negative result (§4): a pre-registered ârule-likenessâ axis failed its reliability gate; we report this rather than the unreliable distribution, and use it to motivate future work (§8). The study is descriptive: the five repositories are a purposive, not representative, sample of AI-tooling organizations, and our claims are limited to what their maintenance histories contain. 2 Related Work Automated skill curation builds on lifelong-agent skill libraries that agents grow au- tonomously, from Wang et al. (2024)âs ever-growing library of executable skills to reflective successors that distill experience into reusable insights (Shinn et al., 2023; Zhao et al., 2024). Recent work edits or evolves skill text from experience through text-space optimizers (Yang et al., 2026), trajectory distillation (Ni et al., 2026), RL-trained curators of an external repos- itory (Ouyang et al., 2026), and fully autonomous self-evolving skill-library systems (Lin et al., 2026; Shen et al., 2026; Yu et al., 2026b). Benchmarks measure whether the resulting skills help (Zhong et al., 2026; Li et al., 2026a; Han et al., 2026). None studies the human maintenance process, and Ouyang et al. (2026) runs no manual-curation arm. Empirical mining of evolving text artifacts is our closest methodological area: studies of context-file evolution and adoption (Chatlatanagulchai et al., 2025; Mohsenimofidi et al., 2026), agentic-tool configuration datasets (Galster et al., 2026), prompt evolution in reposito- ries (Tafreshipour et al., 2025), AI-IDE rule files (Cai et al., 2026), and CI-workflow evolution (Rostami Mazrae et al., 2026) all follow the same workflow: open-code a sample into a typed change taxonomy, then scale it. The prompt- and context/rule-file studies report the additive-dominant signature we also find (Tafreshipour et al., 2025; Cai et al., 2026), while workflow edits skew modification-heavy (Rostami Mazrae et al., 2026). Tafreshipour et al. (2025) also show only 21.9% of prompt changes in commit messages, echoing our sparse trigger evidence. The closest concurrent work is a registered report that plans to mine agent context files (CLAUDE.md/AGENTS.md) and derive a maintenance-grounded change tax- onomy (Voria et al., 2026); it excludes skill files and does not attribute who makes each edit. We instead study executableSKILL.mdskills, apply a pre-registered operation taxonomy, and focus on the governance question of who authors or merges each edit. Those studies mine 10 3 â10 4 changes, far more than our 254 coded edits; what our smaller corpus adds is theSKILL.md-specific measurement none of them applies to skill files: commit-authorship governance and coded curation operations. 1 https://github.com/megagonlabs/who-maintains-agent-skills 2 Accepted at the COLM 2026 Workshop on Lifelong Agents Long.ObjectCuration SkillOpt / SkillOS / Trace2Skill (Yang et al., 2026; Ouyang et al., 2026; Ni et al., 2026)âskill filesautomated only SkillLearnBench / SWE-Skills-Bench (Zhong et al., 2026; Han et al., 2026)âskill filesendpoint only Agent READMEs (Chatlatanagulchai et al., 2025)âcontext filescoarse, uncoded Claude-skills marketplace (Ling et al., 2026)âskill filesâ SkillHone (Li & Hu, 2026)âskill filessynthetic SkillGraph / ExpGraph (Li et al., 2026b; Feng et al., 2026)âinternal graphâ TRACE (Zhou et al., 2026)âruntime rulesuser corrections This workâ SKILL.md edit historiescoded + measured Table 1: Where this study sits. To our knowledge, prior work has not mined longitudinal maintenance histories of human-curated Markdown skill files with coded operations and governance. Long. = longitudinal; Object = the artifact studied; Curation = coded human- curation analysis. repository SKILL.mdsubst. editsedit-daysspan getsentry/skills2881402026-01â2026-06 trailofbits/skills7478222026-01â2026-06 obra/superpowers1462262025-10â2026-05 anthropics/skills1819112025-11â2026-06 cloudflare/skills914102026-02â2026-04 total143254 Table 2: The purposive corpus: per-repository skill-file count, substantive edits, distinct edit-days, and span (months). Pinned commits are listed in the reproducibility note. Our governance signal (theCo-Authored-Bycommit trailer as an AI-authorship marker) follows established repository-mining practice (Larsen & Moghaddam, 2026). Structured skill/experience memory includes typed skill graphs co-trained with a policy (Li et al., 2026b), graph-structured experience memory (Feng et al., 2026), and reflective semantic/episodic memory (Hassell et al., 2026); runtime correction compiles user correc- tions into enforcement hooks (Zhou et al., 2026). Both operate on internal stores or runtime artifacts; none mines longitudinal skill-file histories or writes generalized rules back into skill files (Table 1). 3 Corpus and Method Corpus. To study public skill maintenance where it is observable, we purposively selected five public repositories that maintain agent skills as first-class Markdown files and exhibit sustained commit activity:getsentry/skills,trailofbits/skills,obra/superpowers, cloudflare/skills, andanthropics/skills. These AI-tooling organizations use their own agent products, spanning application monitoring (Sentry), security research (Trail of Bits), a single-maintainer personal skill collection (obra), internet infrastructure (Cloudflare), and an AI lab (Anthropic). Before any governance or operation coding, we froze the repository set, pinned each clone to a fixed commit, and did not run a systematic census of candidate repositories. Maintenance-depth quantities are therefore likely to resemble upper bounds for broader public skill repositories, and governance shares are descriptive and organization- dependent rather than population base rates, with the direction of any broader bias left unestimated (external validity is revisited in §8). Across the pinned clones, the corpus spans 2025-10 to 2026-06: 873 commits across the five repositories, 143SKILL.mdfiles yielding 631 commit-file history records, of which 254 are substantive post-creation edits (defined below). Table 2 gives the per-repository pins and counts. Unit and substantive-edit definition. The unit of analysis is oneSKILL.mdfile tracked through renames (git log --follow). A substantive post-creation edit is any commit after the fileâs creation that changes at least five lines of it and is neither bot-authored nor trivial. This yields 254 substantive edits. Because §6 reports no agent-only substantive commit, we checked whether the bot filter could have created that observation. The filter removes only 13 bot-authoredSKILL.mdcommits, all routed through the platform web-merge identity, and only 8 of them would otherwise qualify as substantive (â3% of the would-be-substantive set). The âno agent-only substantive commitâ observation in §6 is therefore not an artifact of this exclusion. A pre-registered sensitivity battery (Appendix C) recomputes the count 3 Accepted at the COLM 2026 Workshop on Lifelong Agents two-way class â=0.64 (0.82) 8 operations â=0.62 (0.70) mass-refactor excluded enhancement (perfective) 152 (60%) corrective 97 (38%) 725019105641 59311441 n=254 n=165 content-expansion (72) restructure (50) description-tuning (19) consolidation-merge (10) deprecation-retirement (1) factual-correction (56) fix-from-failure (41) other (5) Figure 1: Operations over the 254 substantive skill-file edits. Middle row: the eight coded operations (counts in the legend). Top row: the same edits grouped into the corrective and enhancement maintenance classes. Bottom row: the recount on the 165 edits that remain after excluding multi-file mass-refactor commits. of deeply maintained skills (â„3 substantive edits) across thresholds and mass-refactor exclusion. With 27 / 22 / 16 / 15 skills across the four cells, the corpus supports a descriptive study in every cell. A 50-edit audit sample categorized 70% as skill-content changes (the remainder migration, dependency-version churn, or pure style), confirming the substantive- edit definition is not dominated by mechanical churn. Three codebooks. To connect each edit to the study questions, we coded every substantive edit along three axes: governance, derived deterministically from commit metadata (bot author; AI co-author trailer; human author with web-merge committer; human author and committer); curation operation, under a pre-registered eight-operation taxonomy we constructed for skill-file maintenance that can be collapsed onto the classical corrective- versus-enhancement split used in §4; and trigger evidence on a 0â3 ladder (L3 explicit issue/failure link; L2 named concrete failure; L1 generic; L0 none). For operation, trigger, and rule-likeness labels, a single pass of Opus 4.8 coded all 254 edits, and a second pass of the same model coded a 50-edit stratified sample in a separate session. The coding model received the codebook, the commit subject, and the per-file diff. Unlike the de- identified judging in §7, these inputs were not identity-scrubbed (paths and diffs can reveal the repository), which we record as a limitation. We report agreement as CohenâsÎș, raw agreement corrected for chance (Cohen, 1960), and use the 50-edit overlap to estimate coding stability rather than humanâhuman agreement or validation against human labels. Two passes that share a systematic bias can agree while being jointly wrong, so we treat the 8-way operation taxonomy as descriptive and rest headline claims on the two-way collapse. That collapse is validated by a blind single-coder human recode of the 50-edit sample at Îș =0.72 (8-way leaves onlyÎș =0.46) and corroborated, as a post-submission sensitivity check, by a cross-family recode of all 254 edits (collapseÎș =0.65, raw 0.83; Appendix A). Governance is mechanical, so the dual-coding was a quality-control check: coded labels matched the deterministic value on 252 of 254 edits (99.2%), and the two disagreements were annotation errors corrected to the metadata-derived value. All scripts and coded data are released. 4 What Maintenance Does With the corpus and codebooks in place, we first ask what maintainers change when they edit skill files, and the answer leans toward enhancement over repair. Figure 1 reports the distribution of curation operations across the 254 substantive edits; content expansion (72) and factual correction (56) together account for half of all edits. We collapse the eight operations onto the standard corrective-versus-enhancement split of classical maintenance taxonomy (Swanson, 1976; ISO/IEC/IEEE, 2022) as an interpretive grounding, not a recod- ing: factual-correction and fix-from-failure are corrective, and the remaining five operations are grouped as enhancement (perfective improvement). Under this collapse the corpus is 60% enhancement and 38% correction, with 2% other (Figure 1, top row). In practice, roughly three edits enhance a skill for every two that repair one. Reliability and denominators. Because the operation labels are LLM-coded, we use the reliability checks of §3 to separate headline claims from descriptive detail. At the leaf level 4 Accepted at the COLM 2026 Workshop on Lifelong Agents the labels are only moderately stable: inter-LLM-pass agreement on the eight-way taxonomy isÎș =0.62, and cross-family re-codes agree less (Îș =0.36/0.40 on 50 edits, 0.46 over all 254). The two-way collapse is sturdier: it raises raw inter-LLM agreement to 0.82 at a similarÎș(0.64 vs. 0.62), and the blind human recode validates it atÎș =0.72 (Appendix A), which is why headline claims rest on the collapse while leaf counts stay descriptive. The denominator also matters: we count per skill-file edit row, so a multi-file commit contributes one row per skill it touches. Under stricter denominators the skew only sharpens: the 50-edit audit flaggedâŒ30% of rows as non-curation (migration, dependency churn, style), the 35 curation-confirmed audit edits give 77%/23% (Appendix C), and weighting commits equally gives 67% enhancement, 31% corrective, and 2% other over the 187 commits, leaving the row-level 60% the lowest of the weightings we examine. We do not claim a complete maintenance classification: adaptive (migration) edits fall outside the codebookâs scope and appear only as audit residue (8 of the 50 audited edits). The corrective share is partly bulk repair. Of the factual-correction edits, 75% come from multi-file mass-refactor commits, chiefly onetrailofbitsspec fix that contributes 25 edit-rows. Under mass-refactor exclusion, factual correction falls from the second-largest category to the fourth, while content expansion, fixing-from-failure, and restructuring are robust (Appendix C). Consolidation/merging (10) and deprecation (1) are rare: skills accrete and are corrected far more than they are pruned, a pattern directly relevant to the unbounded-growth concern in never-ending skill learning. A pre-registered rule-likeness axis was not reliable enough to report. To ask whether human curation produces generalizable knowledge, we pre-registered a rule-likeness axis: whether an edit is ârule-likeâ (a generalizable constraint) or âinstance-boundâ (a one-off fix). It did not pass its reliability check: inter-raterÎș =â0.02, with raw agreement of only 0.56 and disagreement concentrated at the instance-bound/mixed boundary (confusion matrix, Appendix A). The negativeÎșpartly reflects a base-rate paradox. The second pass labels almost everything ârule-likeâ (38/50), so observed agreement (0.56) sits just below chance (0.57) andÎșturns negative. Prevalence-adjusted, PABAK=0.34 and Gwetâs AC1=0.44, so the failure is as much an instrument and base-rate effect as evidence the two constructs are intrinsically inseparable. Per our pre-registration we do not report the apparent distribution as a finding. Still, the failure is informative: if two independent LLM passes cannot reliably separate rule-like from instance-bound edits, systems that would generalize curation rules face the same coding obstacle (§8). 5 How Skills Evolve After coding what maintenance does, we next ask how skill files change over time. Across the 120 skills with at least two size observations, trajectories are heterogeneous but lean toward growth: 32 grow by more than 10% in resident token length, 7 shrink by more than 10%, and 81 remain stable. Successive commits touch a skill every 5 days at the median (488 intervals over all skill-file commits, including non-substantive ones, and the median remains 5 days when restricted to substantive edits). Given 5â8 month right-censored windows and an uneven depth distribution, the data do not support a strong growth-or- decay narrative: maintenance is frequent and predominantly stable-to-additive, with neither broad compounding nor net decay. This is a suggestive, loosely drawn contrast with the addition-dominated picture reported for agent context files (Chatlatanagulchai et al., 2025). 2 Net shrinks are uncommon, but consolidation and restructuring remain part of the repertoire (§4), consistent with skills accreting and being corrected far more than they are pruned. Observed corrective or failure-related edits are not defect-rate measures: they reflect what is visible in maintenance histories and may equally indicate active detection, documentation, and repair of issues. Per-repository edit counts and the deeply maintained-skill distribution are in Appendix C. The cadence and size-change distributions are in Figure 4. Maintenance concentrates on the skill body. We decompose eachSKILL.mdinto six compo- nents with a deterministic parser (frontmatter; the name/description ârouter â; instructions; 2 The artifacts and metrics differ: their per-commit word add/delete counts versus our net resident- token trajectories. 5 Accepted at the COLM 2026 Workshop on Lifelong Agents examples; code; references) and attribute the changed lines in every substantive edit. The body dominates: 85% of edits touch instructions and 56% touch embedded code, while the router (38%), frontmatter (15%), examples (11%), and standalone references (4%) are edited far less often. 6 Who Maintains, and Why Governance: trailer-visible AI co-authorship under human repository control. Of 254 substantive maintenance edits, 158 (62%) carry an AI co-author trailer in the commit body, while the remaining 96 (38%) are AI-trailer-absent (60 human-authored-and-committed, 36 human-authored 3 and web-merged). Counting commits rather than skill-file edit rows leaves the split essentially unchanged (61%, 114 of 187 commits). Put operationally, three of every five substantive edits carry disclosed AI co-authorship, each released through a human account. Within the non-bot substantive set, every edit is authored or merged by a named human, and we observe no agent-only substantive commit, a claim about commit attribution rather than verified authorship. Bot commits are excluded by definition, and only 8 would-be-substantive bot commits exist (§3), so the exclusion does not manufacture this observation. Governance is therefore a spectrum: the 62% AI-trailered edits are humanâAI collaborations, and the rest show no trailer-visible AI. No size or scope separation. We detect no size or scope separation between AI-trailered and AI-trailer-absent edits: medians of 15 versus 21 changed lines (MannâWhitneyp =0.20) and 4 versus 6 files (p =0.31). The governance signal tracks who authors or merges an edit and carries no detectable information about what the edit changes. We next ask whether editor identity predicts which part of a skill changes. No componentâs governance split (AI-trailered vs. AI-trailer-absent) survives our pre-registered controls (Holm-corrected Fisher, mass-refactor exclusion, and repository-stratified CochranâMantelâ Haenszel). The raw cross-tab misleads: frontmatter edits look 90% AI-trailered against a 62% base rate, but the signal is entirely a multi-file mass-refactor artifact, collapsing to the base rate (on six edits) once fan-out commits are excluded. Together with the size result above, we therefore detect no governance-by-component specialization: across these repositories, who authors or merges an edit predicts neither how large it is nor which part of the skill it changes. Because the battery is low-powered and multiply tested, this null shows only the absence of a detectable division of labor, not strict uniformity (Appendix B). The regime is organization-dependent. The regime varies sharply by organization (Figure 2, Table 3): a single-maintainer project is almost entirely human-authored; two organizations route nearly all maintenance through human-merged pull requests with no AI trailer; two others co-author the large majority of edits with an AI tool. The pooled 62% therefore sum- marizes a bimodal regime, not a population base rate. Because pooling trailer signals across heterogeneous repositories can reverse the apparent conclusion (Yu et al., 2026a), we report governance per organization. The observed mix can reflect several factors that we cannot disentangle, including actual AI involvement, trailer-generation practices, merge/squash workflows, and repository-specific attribution conventions.â Across all five repositories, every edit is still human-authored or human-merged. Automated curation systems that operate on these public artifacts would need to fit that repository process. Construct validity. Four constructs are distinct here. True AI use we do not measure. Trailer- visible AI co-authorship is the deterministic signal we report (aCo-Authored-Bytrailer naming an AI tool, Larsen & Moghaddam, 2026); it undercounts undisclosed use (Robbes et al., 2026; Kraishan, 2025) and can overcount auto-injected trailers. Human authorship-or- merge metadata establishes that a named human account authored or released each edit. Verified human review is what commit metadata cannot establish: âhuman-mergedâ denotes the pull-request web-merge flow, not confirmed review approval. Our claims concern the middle two, and âhuman-governedâ in this paper means exactly that human account- holders control authorship and release. A concrete anchor for the possible undercount: 3 We consider accounts named as human-owned to define âhuman-authoredâ actions. 6 Accepted at the COLM 2026 Workshop on Lifelong Agents 0255075100 share of substantive maintenance edits (%) getsentry trailofbits obra anthropics cloudflare 93% 92% 16%84% 95% 93% n=81 n=78 n=62 n=19 n=14 AI co-author trailer human-committed human-merged (PR) Figure 2: Human-governed maintenance; trailer-visible AI co-authorship varies by or- ganization. Repository shares by governance class from commit metadata (counts in Table 3); every substantive edit is human-authored or human-merged. Classes indicate trailer-visible co-authorship, not measured AI use; squash workflows can drop trailers (§6).These repository-level shares should not be interpreted as measures of organization- wide AI adoption, human contribution, or engineering practice. repositorynAIHCHM getsentry/skills81756â trailofbits/skills787215 obra/superpowers621052â anthropics/skills191â18 cloudflare/skills14â113 total2541586036 Table 3: Governance by repository (deterministic from commit metadata). AI = AI co-author trailer present; HC = human-committed; HM = human-merged via pull request, with no trailer-visible AI co-authorship. anthropics/skills reads as almost entirely human-merged with no AI trailer (Table 3), yet Anthropic reports that, as of May 2026, more than 80% of the code merged into its production codebase is authored by Claude under an engineer-directed review model (Anthropic, 2026). Because that repository is documentation rather than production code, this example anchors the plausible magnitude of undercount rather than a rate for our corpus. Trigger evidence. We quantify observable evidence linking an edit to a usage failure, not a causal claim. Of 254 edits, 61 (24%) carry concrete failure evidence (L2 or L3); 12 cite an explicit issue or describe a reported failure (L3). Most edits (193, L1) are generic maintenance whose connection to a specific failure is not stated in the record. We therefore reserve âfailure-triggeredâ for the L2/L3 minority and report the denominator. No edit fell to L0 in the primary pass, so its realized ladder is L1âL3. TheÎș =0.59 inter-pass reliability is borderline, the disagreement is directional (the second pass reads more failure evidence), and a post-submission cross-family recode of all 254 edits under the identical codebook (gpt-5.5) reads concrete evidence into 63% of edits (Îș =0.18 against the primary pass), almost entirely by promoting generic-cue (L1) edits to L2 while confirming 59 of the 61 primary-pass L2/L3 edits. The concrete-evidence share is thus instrument-dependent: 24% is the stricter reading, not a stable quantity; we treat the trigger counts as descriptive only (a calibration caveat for codebook reusers), drawing no structural claim from them. 7 Does Maintenance Help? A Bounded Transfer-Task Test Beyond describing maintenance, we asked whether it improves a skill. An earlier pilot hinted that the maintained version outperforms the earliest in-window one, but it had four weaknesses: the solver saw only theSKILL.mdfile rather than the full skill package, the exchange was limited to a single turn, the judge came from the same provider as the solver, and the design had roughly 20% statistical power (Appendix D.1). We pre-registered and conducted a replication that fixes each of these. The full design, statistics, and per-skill numbers appear in Appendix D. The study uses 13 public skills withâ„6 substantive edits across four repositories, each paired with 11 transfer tasks (143 total), attempted by a solver from a different model family than the judges (gpt-5.4-mini) underno-skill,vold, 7 Accepted at the COLM 2026 Workshop on Lifelong Agents brainstorming â skill-writer commit create-branch â blog-writing-guide subagent-driven-development writing-plans â writing-skills claude-api iterate-pr â claude-settings-audit modern-python pr-writer â All ( n = 13 ) â1.5 â1.0 â0.5 0.0 0.5 v_new ÂĄ v_old judged quality (1â5) v_new better v_old better -0.09 Figure 3: Maintenance moves individual skills both ways, but not on average. Maintained- minus-earliest judged-quality difference (1â5 rubric). Points show skill means over 11 tasks Ă3 replicates; whiskers show descriptive task-level 95% CIs;â marks deep-history skills; square marks the pre-registered skill-level aggregate. Per-skill differences are specific to this constructed transfer-task harness and should not be interpreted as measures of intrinsic skill quality, maintainer quality, or effectiveness in the repositoriesâ native workflows. vnew; the solver sees the full skill package through on-demand file reads, an equalized multi-turn protocol lets a constrained user-simulator answer clarifying questions from the brief alone, and a sealed panel of blind judges scores de-identified, fingerprint-scrubbed, randomized triads. The inference unit is the skill (n =13), with pre-registered power of 0.77 to 0.90 for the pilot-sized effect at a fixed sample size. At the skill level the effect on judged quality is indistinguishable from zero: the meanvnewâ volddifference isâ0.09 on the 1â5 scale, with a 95% confidence interval of[â0.28,+0.10]. Neither a sign test (p =0.58) nor a Wilcoxon test (p =0.55) rejects the null, and 5 of 13 skills favorvnew. This result survives a length-controlled re-analysis and a weaker-solver re-run undergpt-4o-mini(Appendix D). We read this as no evidence for a pilot-sized benefit of maintenance under this harness, not as evidence that maintenance is inert. The interval still admits small positive effects, and cross-family judge agreement was only moderate (Spearman 0.51; single-measures ICC(2,1)=0.52 on the 100-triad overlap, below the pre-registered 0.6 gate, Appendix D), so a small true effect could be attenuated by measurement noise. Whether maintenance helps a downstream solver is thus design-contingent: the pilotâs positive trend does not persist under the powered design that controls package exposure, interaction channel, judge family, and statistical power. This is a bounded null on author-and-model-constructed transfer tasks. Our descriptive contribution does not rest on this result. Why the per-skill effects split.Two extremes explain the split (illustrative only). brainstorming(most negative,â0.95) andpr-writer(most positive,+0.31) both roughly doubled to tripled in length under maintenance, so size is not the discriminator. brainstormingâs edits added structured elicitation steps (a pre-design gate, a checklist, a process diagram). On a one-shot transfer task this steered the solver toward process management and suppressed design content (judged coverage 2.06â1.33, both skilled versions below the skill-free baseline).pr-writerâs edits instead added output constraints (rewrite-donât-append, prose over headings, bans on placeholder references), which trans- ferred to the judged artifact (non-redundancy 3.45â4.58, the largest single-dimension gain). Across the 13 skills these two regimes cancel (5 positive, 8 negative): the pilotâs apparent benefit averages out once a heterogeneous skill set is scored on transfer tasks, so a powered aggregate reads null while individual skills still move both ways. Relation to in-the-wild skill evaluations. Our powered null aligns with empirical skill- benefit studies that find benefits fragile. In-the-wild benchmarks find skill benefit to be setting-dependent but refinement-recoverable (Liu et al., 2026), and library growth can degrade agents through skill shadowing (Song & Wei, 2026). Both vary deployment or library, whereas we hold the harness fixed, vary only the maintained version, and observe neither benefit nor shadowing-style loss. Continual-skill benchmarks ask whether machine- generated or continually evolved skills help (Zhong et al., 2026; Li & Hu, 2026). We ask whether human maintenance helps a downstream solver, finding no pilot-sized transfer 8 Accepted at the COLM 2026 Workshop on Lifelong Agents benefit, so âdoes maintenance helpâ is itself a measurement question. Native repository workflows or external skill benchmarks could differ, and we release the powered harness to test that (Kim et al., 2025). 8 Discussion: Toward Rule-Generalizing Curation From field observations to a curator. Two results constrain curator design. First, mainte- nance is a standing humanâAI loop that grows and corrects skills far more than it prunes them: the budget pressure anticipated by never-ending skill learning appears here as ac- cretive, human-governed editing. Second, the rule-likeness reliability failure shows that âdoes this edit encode a generalizable rule?â is hard to operationalize reliably from commit artifacts, as two independent LLM coding passes did not reach acceptable agreement. To- gether, these results suggest a curator that records traceable edit rationales and consolidates them into maintainer-approved generalized guidance, rather than rewriting skill text from individual instances. This is closer to never-ending learning over a typed knowledge base (Mitchell et al., 2015) than one-shot text optimization, and complements critique-distilling reflective memory without such structure (Hassell et al., 2026). Similarly, âLLM wikiâ envi- sions an agent-maintained interlinked Markdown knowledge base at near-zero human cost (Karpathy, 2026), and WiNELL trains agents on historical edits to propose never-ending Wikipedia updates for human review (Reddy et al., 2026). Our measurement shows the observed loop still runs through human-owned repository processes. Designing such a curator remains future work. The measured regime and released corpus instead yield design hypotheses. Design implications for autonomous skill-curators. These hypotheses are grounded in the observed maintenance regime, not validated designs. 1.Budget explicitly for pruning. Consolidation and deprecation are only 4.3% of edits (11/254), so a curator may need an explicit consolidation/retirement budget. 2.Keep curation under human-owned release or merge control. Every substantive edit was authored or merged through a named-human repository process, so autonomous curators should surface proposed edits at that boundary. 3. Do not rely solely on failure-provenance to supervise. The concrete-failure-evidence share is instrument-dependent (24% under our primary pass, 63% under a cross- family recode), the trigger ladder is only borderline-reliable even within one family, and rule-likeness failed its gate (Sections 4, 6); failure provenance cannot yet be measured stably enough to serve as a curatorâs primary supervision target. 4.Triage by operation type before size. Operation type is our highest-agreement coded axis (§4), while governance predicts neither edit size nor scope (§6), so a curator should route first by what an edit does rather than by edit size. 5.Model governance per repository, not as one global mix. The 62% AI-trailer prevalence is an observed, two-sided proxy (it can under- and over-count true AI involvement), and the split is bimodal across repositories, so a curator deployed across organiza- tions should model local disclosure and merge cultures rather than assume a single global humanâAI mix. The released corpus makes these hypotheses testable: a candidate curator sees a prior skill version and context, proposes the next edit, and is scored against the human edit (Appendix E). Building a baseline remains future work. 9 Conclusion Skill maintenance in these public repositories is frequent, human-governed in the specific sense that every edit is authored or merged through a named human account, and often visible in commit trailers as AI-assisted. Prior work on automated curation has largely left this maintenance process unmeasured. We release a commit-level corpus, three code- books, and reproducible scripts for studying this process, and we report a pre-registered rule-likeness failure that clarifies the open problem for rule-generalizing curators. For lifelong, self-evolving agents, these repositories suggest that curation systems should be evaluated against human-governed maintenance loops, not only against fully automated skill-evolution pipelines. 9 Accepted at the COLM 2026 Workshop on Lifelong Agents References Anthropic. When AI builds itself. Anthropic Institute, June 2026. URLhttps://w. anthropic.com/institute/recursive-self-improvement . Reports that more than 80% of merged production code was authored by Claude as of May 2026. Guangzong Cai, Ruiyin Li, Peng Liang, Zengyang Li, and Mojtaba Shahin. Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study. arXiv preprint arXiv:2606.12231, 2026. URL https://arxiv.org/abs/2606.12231. Worawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa, Brittany Reid, Kundjanasith Thonglek, Pattara Leelaprute, Arnon Rungsawang, Bundit Manaskasemsak, Bram Adams, Ahmed E. Hassan, and Hajimu Iida. Agent READMEs: An Empirical Study of Context Files for Agentic Coding.arXiv preprint arXiv:2511.12884, 2025.URL https://arxiv.org/abs/2511.12884. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37â46, 1960. Tao Feng, Chongrui Ye, Tianyang Luo, Jingjun Xu, Xueqiang Xu, Haozhen Zhang, Zhigang Hua, Yan Xie, Shuang Yang, Ge Liu, and Jiaxuan You. ExpGraph: Model-Agnostic Experience Learning with Graph-Structured Memory for LLM Agents. arXiv preprint arXiv:2605.30712, 2026. URL https://arxiv.org/abs/2605.30712. Matthias Galster, Seyedmoein Mohsenimofidi, Levi B Ì ohme, Jai Lal Lulla, Muham- mad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. A Dataset of Agentic AI Coding Tool Configurations. In Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware 2026), p. 314â322. ACM, 2026. doi: 10.1145/3805760.3814922. Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? arXiv preprint arXiv:2603.15401, 2026. URLhttps://arxiv.org/abs/2603. 15401. Jackson Hassell, Dan Zhang, Hannah Kim, Tom Mitchell, and Estevam Hruschka. Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation. In Proceedings of the ACM Conference on AI and Agentic Systems (CAIS â26), p. 276â291. ACM, 2026. doi: 10.1145/3786335.3813154. ISO/IEC/IEEE. ISO/IEC/IEEE 14764:2022 Software Engineering â Software Life Cycle Processes â Maintenance. International Standard, 2022. Andrej Karpathy. LLM wiki. GitHub gist, April 2026. URLhttps://gist.github.com/ karpathy/442a6bf555914893e9891c11519de94f. A pattern for an LLM-maintained, inter- linked Markdown knowledge base with periodic consistency linting. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. The BiGGen bench: A principled benchmark for fine-grained evaluation of language models with language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 5877â5919. Associa- tion for Computational Linguistics, 2025. doi: 10.18653/v1/2025.naacl-long.303. URL https://aclanthology.org/2025.naacl-long.303/. Obada Kraishan. The AI Attribution Paradox: Transparency as Social Strategy in Open- Source Software Development. arXiv preprint arXiv:2512.00867, 2025. URLhttps://arxiv. org/abs/2512.00867. 10 Accepted at the COLM 2026 Workshop on Lifelong Agents Oliver Aleksander Larsen and Mahyar T. Moghaddam. Mining Architectural Quality Under Agentic AI Adoption: A Causal Study of Java Repositories. arXiv preprint arXiv:2606.13298, 2026. URLhttps://arxiv.org/abs/2606.13298. Accepted at the 52nd Euromicro Confer- ence on Software Engineering and Advanced Applications (SEAA) 2026 (Springer LNCS); cited from the arXiv preprint. Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiaojun Wu, Junwei Zhou, Xiaokun Chen, Hanwen Xing, Yubo Li, Qunhong Zeng, Di Wang, Yuanli Wang, Roey Ben Chaim, Penghao Jiang, Haotian Shen, Luyang Kong, Xinyi Liu, Runhui Wang, Xuanqing Liu, Jiachen Li, Xin Lan, Yueqian Lin, Wengao Ye, Junwei He, Songlin Li, Yue Zhang, Yipeng Gao, Yijiang Li, Ze Ma, Liqiang Jing, Tianyu Wang, Kaixin Li, Yiqi Xue, Haoran Lyu, Yizhuo He, Yuchen Tian, Shutong Wu, Bowei Wang, Yixuan Gao, Bo Chen, Litong Liu, Sikai Cheng, Jiajun Bao, Shuaicheng Tong, Shuwen Xu, Terry Yue Zhuo, Tinghan Ye, Qi Qi, Miao Li, Longtai Liao, Zelin Tan, Chang Shi, Xilin Tang, Srinath Tankasala, Boqin Yuan, Yaoyao Qian, Jianhong Tu, Chenguang Wang, Yizhou Sun, Wei Wang, Aaron Taylor, Ziyue Yang, Changkun Guan, Zhikang Dong, Xinyu Zhang, Steven Dillmann, Han chung Lee, and Dawn Song. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv preprint arXiv:2602.12670, 2026a. URL https://arxiv.org/abs/2602.12670. Xiaoyuan Li, Moxin Li, Keqin Bao, Yubo Ma, Wenjie Wang, Dayiheng Liu, and Fuli Feng. SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs. arXiv preprint arXiv:2605.12039, 2026b. URLhttps://arxiv.org/abs/2605.12039. Zhiwei Li and Yong Hu. SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History. arXiv preprint arXiv:2606.08671, 2026. URLhttps://arxiv. org/abs/2606.08671. Huawei Lin, Peng Li, Jie Song, Fuxin Jiang, and Tieying Zhang. MUSE-Autoskill: Self- Evolving Agents via Skill Creation, Memory, Management, and Evaluation. arXiv preprint arXiv:2605.27366, 2026. URL https://arxiv.org/abs/2605.27366. George Ling, Shanshan Zhong, and Richard Huang. Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality. arXiv preprint arXiv:2602.08004, 2026. URL https://arxiv.org/abs/2602.08004. Yujian Liu, Jiabao Ji, Li An, Tommi Jaakkola, Yang Zhang, and Shiyu Chang. How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings. arXiv preprint arXiv:2604.04323, 2026. URL https://arxiv.org/abs/2604.04323. Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Justin Betteridge, Andrew Carlson, Bhavana Dalvi Mishra, Matthew Gardner, Bryan Kisiel, Jayant Kr- ishnamurthy, Ni Lao, Kathryn Mazaitis, Thahir Mohamed, Ndapa Nakashole, Em- manouil Platanios, Alan Ritter, Mehdi Samadi, Burr Settles, Richard Wang, Derry Wijaya, Abhinav Gupta, Xinlei Chen, Abulhair Saparov, Malcolm Greaves, and Joel Welling.Never-ending Learning.In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 29, p. 2302â2310, 2015. doi: 10.1609/aaai.v29i1.9498. URL https://ojs.aaai.org/index.php/AAAI/article/view/9498. Seyedmoein Mohsenimofidi, Matthias Galster, Christoph Treude, and Sebastian Baltes. Context engineering for AI agents in open-source software. In Proceedings of the 23rd International Conference on Mining Software Repositories (MSR), 2026. arXiv:2510.21413. Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill: Distill Trajectory- Local Lessons into Transferable Agent Skills. arXiv preprint arXiv:2603.25158, 2026. URL https://arxiv.org/abs/2603.25158. Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zha, Maohao Shen, Vishy Tirumalashetty, George Lee, Jiawei Han, Tomas Pfister, and Chen-Yu Lee. SkillOS: Learning Skill Curation 11 Accepted at the COLM 2026 Workshop on Lifelong Agents for Self-Evolving Agents. arXiv preprint arXiv:2605.06614, 2026. URLhttps://arxiv.org/ abs/2605.06614. Revanth Gangi Reddy, Tanay Dixit, Jiaxin Qin, Cheng Qian, Daniel Lee, Jiawei Han, Kevin Small, Xing Fan, Ruhi Sarikaya, and Heng Ji. WiNELL: Wikipedia never-ending updating with LLM agents. In Proceedings of the ACM Web Conference 2026 (W). ACM, 2026. doi: 10.1145/3774904.3792695. URL https://doi.org/10.1145/3774904.3792695. Romain Robbes, Th Ì eo Matricon, Thomas Degueule, Andre Hora, and Stefano Zacchiroli. Promises, perils, and (timely) heuristics for mining coding agent activity. In Proceed- ings of the 2026 IEEE/ACM 23rd International Conference on Mining Software Repositories (MSR), 2026. URLhttps://2026.msrconf.org/details/msr-2026-technical-papers/ 42/Promises-Perils-and-Timely-Heuristics-for-Mining-Coding-Agent-Activity. Pooya Rostami Mazrae, Alexandre Decan, Tom Mens, and Mairieli Wessel. An Empirical Study of the Evolution of GitHub Actions Workflows. Journal of Systems and Software, 236: 112824, 2026. doi: 10.1016/j.jss.2026.112824. URLhttps://doi.org/10.1016/j.jss.2026. 112824. Shuaike Shen, Wenduo Cheng, Mingqian Ma, Alistair Turcan, Martin Jinye Zhang, and Jian Ma. SkillFoundry: Building Self-Evolving Agent Skill Libraries from Heterogeneous Scientific Resources. arXiv preprint arXiv:2604.03964, 2026. URLhttps://arxiv.org/abs/ 2604.03964. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, p. 8634â8652, 2023. Hongwen Song and Song Wei. More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries. arXiv preprint arXiv:2605.24050, 2026. URL https://arxiv.org/abs/2605.24050. E. Burton Swanson. The dimensions of maintenance. In Proceedings of the 2nd International Conference on Software Engineering (ICSE), p. 492â497, 1976. Mahan Tafreshipour, Aaron Imani, Eric Huang, Eduardo Santana de Almeida, Thomas Zimmermann, and Iftekhar Ahmed. Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories. In Proceedings of the 22nd IEEE/ACM Inter- national Conference on Mining Software Repositories (MSR), p. 686â698. IEEE, 2025. doi: 10.1109/MSR66628.2025.00106. URL https://doi.org/10.1109/MSR66628.2025.00106. Gianmario Voria, Alfonso Cannavale, Andrea De Lucia, Yutaro Kashiwa, Gemma Catolino, and Fabio Palomba. How Do Developers Maintain and Evolve Their Agentsâ Instructions? An Empirical Study. arXiv preprint arXiv:2606.25257, 2026. URLhttps://arxiv.org/abs/ 2606.25257. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. arXiv:2305.16291. Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv preprint arXiv:2605.23904, 2026. URL https://arxiv.org/abs/2605.23904. Haoran Yu, Xiaochong Jiang, Lifei Liu, Su Wang, Pin Qian, and Yihang Chen. Beyond Simp- sonâs Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship. arXiv preprint arXiv:2606.22711, 2026a. URLhttps://arxiv.org/abs/2606.22711. Accepted at the KDD 2026 Workshop on Agentic Software Engineering (SE 3.0); cited from the arXiv preprint. Zhuoyun Yu, Xin Xie, Wuguannan Yao, Chenxi Wang, Lei Liang, Xiang Qi, and Shumin Deng. SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories. arXiv preprint arXiv:2606.01311, 2026b. URL https://arxiv.org/abs/2606.01311. 12 Accepted at the COLM 2026 Workshop on Lifelong Agents Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. Shanshan Zhong, Yi Lu, Jingjie Ning, Yibing Wan, Lihan Feng, Yuyi Ao, Leonardo F. R. Ribeiro, Markus Dreyer, Sean Ammirati, and Chenyan Xiong. SkillLearnBench: Bench- marking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv preprint arXiv:2604.20087, 2026. URL https://arxiv.org/abs/2604.20087. Yujun Zhou, Kehan Guo, Haomin Zhuang, Xiangqi Wang, Yue Huang, Zhenwen Liang, Pin- Yu Chen, Tian Gao, Nuno Moniz, Nitesh V. Chawla, and Xiangliang Zhang. Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents. arXiv preprint arXiv:2606.13174, 2026. URL https://arxiv.org/abs/2606.13174. Limitations The sample is purposive (five AI-tooling organizations) and selected on sustained mainte- nance, so maintenance-depth quantities are upper-bound-like, while governance shares are descriptive and organization-dependent, not population base rates. Also, because inclu- sion requires observable public histories, the study has an inherent transparency-selection effect: organizations and maintainers that expose richer histories make their corrections, AI- attribution signals, and maintenance activity more measurable than less-open counterparts. These observations should therefore not be interpreted as evidence of lower engineering quality, greater AI dependence, or weaker maintenance practices. Trajectories are right- censored over 5â8 months. The operation and trigger codebooks are LLM-coded: the corrective/enhancement collapse is human-validated atÎș =0.72 with a single coder over 50 edits; the 8-way leaves were human-checked but agreed only moderately (Îș =0.46); and the trigger axis has no human check; a full cross-family recode reproduces neither its labels nor its distribution (Îș =0.18; 63% vs. 24% L2+L3), so trigger counts are primary-pass-specific. The governance signal is observedCo-Authored-Bytrailer prevalence, which can both under- and over-count true AI involvement. Trigger evidence is bounded by what commit records make observable, and âhuman-mergedâ denotes the pull-request web-merge flow, not verified review approval. Ethics and reproducibility All data are public repository histories; no human subjects are involved. We note nonetheless that public availability is not consent to be aggregated: our governance signal derives from named human committer/author and AI-tool identities. We therefore report only organization-level aggregates (Table 3), not individual authorship profiles, and our released records expose only what the public commit metadata already contains (mined commit metadata and our own coding labels, with no repository file content redistributed); we add no inferred attributes and apply no de-anonymization beyond the trailer and merge- committer fields git records. The released records do join these public identity fields to our curation labels; this adds no non-public identity data, and we report analyses only at the organization level, noting that for a single-maintainer repository organization-level aggregation is effectively individual-level. Released artifacts (the resource contribution), available at: 4 (i) commit-level records, 631SKILL.mdhistory records over 143 files, with the 254 substantive edits labeled under all three codebooks; (i) a codebook datasheet with label definitions and deterministic/judgment decision rules; (i) the mining, component- attribution, LLM operation-coding, and offline-analysis scripts, with a reproduction script; and (iv) pointers to the five pinned public source repositories (a clone script pinning each commit) with their own licenses. Source repositories are pinned at fixed commits (getsentry b39c7c4, trailofbitsc070b9b, obra6fd4507, anthropics5754626, cloudflare12520fd) and carry their own licenses (trailofbitsis C-BY-SA); we redistribute no repository file content, only derived metadata and our labels, so the release is license-compatible. The labeled-edit 4 https://github.com/megagonlabs/who-maintains-agent-skills 13 Accepted at the COLM 2026 Workshop on Lifelong Agents numbers (Sections 4â6 and the appendices) recompute offline from the released records and annotation passes with no API key; the corpus-level commit counts (e.g. the 873 commits) are reproducible by re-cloning the five pinned repositories and re-running the deterministic mining script. The operation taxonomy is reproducible up to LLM coding noise with any annotator model, while the paperâs exact operation distribution andÎșrecompute offline from the released passes; the maintenance-benefit analysis (§7) regenerates from the archived solver and judge logs, which are frozen and not bit-reproducible but included in the release. A Codebooks and Reliability The operation codebook assigns one label per edit by dominant maintenance purpose, with tie-breaks toward the more specific label. Two axes are kept deliberately separate: fix-from-failureis reserved for edits whose dominant purpose is repairing an observed execution failure (else the same content change isfactual-correction); it is thus an operation- with-observed-failure-provenance category, not a pure edit-mechanism one, and trigger evidence is coded separately for evidence strength (we therefore do not treat operation and trigger as statistically independent axes);factual-correctionchanges truth condi- tions whereasdescription-tuningchanges only the router/wording; andrestructure, consolidation-merge, anddeprecation-retirementare distinguished by whether content is reorganized, merged, or removed. The fiveotheredits are non-curation residue (e.g. licensing/boilerplate). The released codebooks datasheet carries the full definitions, prece- dence rules, and per-label examples; Table 4 summarizes the eight operations with an illustrative edit and the key boundary for each. operationdefinitionillustrative editdistinguished from content-expansionadds new guidance, sections, or examples (substance) adds a âcommon pit- fallsâ subsection description-tuning (wording only) factual-correctionfixes an incorrect statement, value, or API corrects a flag name or version string fix-from-failure (no failure cue) restructurereorganizes existing contentsplits one section into three content-expansion (adds substance) fix-from-failure responds to an observed us- age failure adds a stop condition after a reported loop factual-correction (no failure cue) description-tuningrewords the name / descrip- tion / trigger text tightenstheâuse whenâ line content-expansion (adds substance) consolidation-mergemerges or dedupes content folds two overlapping steps into one restructure(no dedupe) deprecation-retirement marks obsolete / removes a capability adds âdeprecated: use X insteadâ restructure (keeps content) othernon-curation residuea license or boilerplate line the seven substan- tive labels Table 4: The eight curation operations with an illustrative edit and the boundary distinction that most often decides the label. The released datasheet carries full per-label examples and precedence rules. coded operationrepository / skillcommit subject (verbatim)lines content-expansiongetsentry / blog-writing-guide add bad/good examples to AI patterns section 34 factual-correctiontrailofbits / solana- vulnerability-scanner correct duplicate section numbering 15 fix-from-failuregetsentry / create-branch Simplify Step 5 and fix unconditional stash pop 61 restructuregetsentry / agents-mdImprove agent guidance skills 147 Table 5: Four real coded edits from the corpus, grounding the codebook in concrete cases. Subjects are quoted from the commits (conventional-commit prefixes and PR numbers elided); lines = changed lines in the named skill file. 14 Accepted at the COLM 2026 Workshop on Lifelong Agents Rule-likeness confusion. The 3Ă3 confusion matrix (pass A rows, pass B columns) is: rule-likeâ(28, 1, 7); instance-boundâ(6, 0, 4); mixed-unclearâ(4, 0, 0). Disagreement concentrates off the rule-like diagonal. Because pass B assigns ârule-likeâ to 38/50 edits, the lowÎș is prevalence-driven, a base-rate paradox: p o = 0.56, p e = 0.57,Îș =â0.02, while the prevalence-adjusted PABAK=0.34 and Gwetâs AC1=0.44. This is the basis for the downgrade in §4. A later cross-family recode of the same axis as a single binary item, where a reusable rule is present only if a specific added line states one, reachesÎș =0.43 (raw agreement 0.72) between two different-family models; this suggests the failure was largely the abstract three-way operationalization rather than the underlying construct, and we still do not report the rule-likeness distribution as a finding. Uncertainty. The same-family inter-passÎșvalues are point estimates on the 50-edit overlap; we report no confidence intervals for the inter-LLM pass agreements, and atnâ50 a 95% interval onÎș =0.62 is wide enough to straddle the conventional 0.70 threshold, so the 0.62 vs. 0.59 ordering should be read as indistinguishable. Cross-family check. To test whether the operation reliability is an artifact of coding with one model family, we re-coded the same 50-edit overlap with two different-family models (gpt-4oandgpt-5.5) under the identical codebook. The 8-way operation agreement with the Claude pass falls toÎș =0.36 and 0.40 (below the same-family inter-pass 0.62), so the fine-grained leaf labels carry agreement specific to the coding model family. The two-way corrective-versus-enhancement collapse that our headline result uses, however, reaches Îș =0.51 and 0.61 (raw agreement 0.76 and 0.80), below but close to the same-family collapse Îș =0.64. Cross-family agreement is thus substantially better for the collapse than for the eight leaves, though still only moderate; it corroborates the enhancement-versus-correction finding against a same-family-bias confound and supports resting claims on the collapse rather than the eight leaf labels. A post-submission recode of all 254 edits withgpt-5.5 under a bundled three-axis prompt corroborates this picture at scale: 8-wayÎș =0.46, collapseÎș =0.65 (raw 0.83), with the collapse distribution moving at most 3.5 points (60/38/2â58/42/0.4). The trigger axis does not survive the same recode (Îș =0.18; §6), and the three-way rule-likeness axis remains unreliable (Îș =0.17) despite near-identical marginals: the two families assign the same distribution to different edits, confirming the axis measures noise. Human validation. One author then independently re-coded the 50-edit overlap blind to the LLM labels. Human-vs-LLM agreement is moderate on the 8-way taxonomy (Îș =0.46, 95% CI[0.29, 0.62], raw 0.58; below the inter-LLM-pass 0.62, as expected if two passes of one model share a systematic bias) but substantial on the two-way corrective- versus-enhancement collapse our headline uses (Îș =0.72, CI[0.51, 0.88], raw 0.86; above the inter-LLM collapse 0.64). Fourteen of the 21 leaf-level disagreements stay within the same corrective/enhancement bucket (concentrated at the content-expansion/restructure and factual-correction/fix-from-failure boundaries), so they do not move the headline. This is a small blind human check of the headline split and indicates the eight leaves are only moderately human-valid, which is why we rest claims on the collapse; it is a single coder (n =50), so it does not establish a humanâhuman reliability ceiling. Governance proxy. We similarly checked the AI-trailer signal: the same coder judged whether independent (non- trailer) evidence of AI involvement was visible on 50 commits. Clear auto-injection was rare (1 of 25 trailer-present commits, 4%) and 2 of 25 trailer-absent commits showed independent AI evidence despite no trailer, but the majority carried no independent signal either way (31/50 unclear). The trailer is thus rarely a clear false positive yet usually not independently verifiable, which supports reporting the 62% strictly as trailer-visible AI co-authorship rather than an estimate of true AI use. B Component Decomposition and Governance We parse eachSKILL.mdversion into six components by a deterministic, total-precedence rule (YAML frontmatter;name/descriptionârouterâ keys and matching headings; instruc- tions; examples; code fences; standalone references), and attribute each substantive editâs changed lines by diffing the file against its parent (254/254 edits attributed; the parser is released). Edits touching each component (of 254): instructions 215 (85%), code 142 (56%), router 97 (38%), frontmatter 39 (15%), examples 28 (11%), references 11 (4%). For each com- 15 Accepted at the COLM 2026 Workshop on Lifelong Agents ponent we test the association between âedit touches itâ and governance class (AI-trailered vs. human-only) with three pre-registered controls: a Holm-corrected Fisher exact test, the same test with multi-file mass-refactor commits excluded, and a repository-stratified CochranâMantelâHaenszel test. No component survives all three. The largest raw effect is frontmatter, which looks 90% AI-trailered against the 62% base rate (Fisherp<10 â3 ); the effect disappears under mass-refactor exclusion, where only six frontmatter edits remain (p =1.0), and under repository stratification (p =0.22). It is a fan-out artifact of a few AI-co-authored mass refactors. The router shows a residual human lean (47% AI-trailered) that holds under mass-refactor exclusion (p =0.015) but not under repository stratification (p =0.12), so we do not assert it. We report no component-attribution reliability coefficient here (the parser is deterministic; a line-level agreement study against an independent recode is future work), and the null is robust to the unambiguous frontmatter-versus-body split. C Sensitivity Analyses Deeply maintained skills (â„3 substantive edits) by threshold and mass-refactor exclusion: â„5 lines incl. mass = 27;â„10 incl. = 22;â„5 excl. = 16;â„10 excl. = 15. Operation distribution under mass-refactor exclusion (89 of 254 edits are multi-file mass-refactor; 165 remain): content-expansion 59, fix-from-failure 41, restructure 31, factual-correction 14, description-tuning 7, consolidation-merge 7, other 5, deprecation 1. Content expansion, fix-from-failure, and restructuring are robust; factual-correction drops from rank 2 to rank 4 (75% mass-refactor, 25 of its rows from onetrailofbitsspec fix). Per-repository edit counts and the 50-edit audit sample (35 curation / 8 migration / 5 dependency-version / 2 docs-style). Re-running the corrective/enhancement collapse on the second-pass labels of the 35 curation-confirmed edits gives 77% enhancement / 23% corrective, versus 70% / 26% / 4% other on all 50, so restricting to confirmed curation sharpens rather than weakens the enhancement skew reported in §4. 010203040506070 days between successive edits (5-95% whiskers) obra (n=126) getsentry (n=170) cloudflare (n=23) anthropics (n=57) trailofbits (n=112) (a) edit cadence is frequent and organization-dependent -100-500+50+100+200 net change in resident token length per skill (%) 0 20 40 60 80 skills shrink 7stable 81 grow 32 (b) net size change is stable-to-additive (n= 120) Figure 4: Maintenance dynamics (§5). (a) Days between successive edits to a skill, per repository (488 intervals; box = IQR, whiskers 5â95%, dashed line = overall median 5 days): cadence is frequent but organization-dependent, from a 1.5-day median (obra) to 14.5 (trailofbits). (b) Net change in a skillâs resident token length between its first and last observation (120 skills withâ„2 size observations): trajectories are growth-leaning, 32 growing more than 10%, 7 shrinking more than 10%, 81 stable, so the net-change distribu- tion is predominantly stable-to-additive, with neither universal growth nor broad decay. Recomputed from the released records; the tail is clipped at±250%. D Does Maintenance Help? The Transfer-Task Study Pre-registered design (frozen before the run; full materials and harness released). Skills: 13 publicSKILL.mdfiles withâ„6 substantive edits, drawn from four repositories (the five deepest-history skills of the pilot plus eight more);vold= earliest in-window version at the current path (rename-resolved),vnew= latest. Tasks: 11 transfer tasks per skill (143 total), generated to exercise each skillâs domain without telegraphing its conventions, then frozen 16 Accepted at the COLM 2026 Workshop on Lifelong Agents and hashed; a curated minimal domain persona is given to the solver in every condition so theno-skillbaseline stays clean. Conditionsno-skill,vold,vnew, three independent replicates per cell (a per-replicate nonce makes the replicates genuine independent draws). The solver isgpt-5.4-mini, a reasoning model from a different family than the judges, with truncation detection and retry that closes the earlier truncation confound. The full skill package is exposed via an on-demand file-read tool (progressive disclosure), with SKILL.md-only and concatenated-package conditions retained as sensitivities. An equalized multi-turn protocol (up to three turns) lets a constrainedgpt-4o-miniuser-simulator answer clarifying questions from the brief alone, so interaction-designed skills are not penalized; the judge rates the final output. Judging. Each de-labelled, randomized triad is scored 1â5 on correctness, coverage, action- ability, and non-redundancy by a sealed panel of blind judges, Claude models with no tool or web access, drawn from a different family than the solver to fix the pilotâs same-provider judging; agpt-5.5reference panel covers a 100-triad subset. A scrub of namespaced sub- skill identifiers, skill slugs, version tokens, and organization names removes de-blinding fingerprints (a planted-fingerprint canary passes). The inference unit is the skill (n =13); a pre-registered skill-level power simulation on paired skill-mean deltas gives 0.77 (sign test) to 0.90 (t-test) power to detect the pilot-sized effect, at a fixed N (no optional stopping). Results. The skill-levelvnewâ volddifference isâ0.09 (95% CIâ[â0.28,+0.10]; sign-test p =0.58, Wilcoxonp =0.55;d = â0.29; 5 of 13 skills favorvnew). A length-controlled re-analysis (score residualized on log output length) givesâ0.06 (p =0.58), so the null is not a length artifact; the deep-five subset agrees (â0.14). No rubric dimension survives Holm correction, and cross-family judge agreement is moderate (Spearman 0.51, below our pre-registered gate), so dimension-level patterns are exploratory and the aggregate null is conditional on a noisy but independent judging harness. The gate was pre-registered as single-measures ICC(2,1)â„0.6, fixed at design freeze; computed on the 100-triad overlap it is 0.52, below the gate and consistent with the upper bound that Pearson 0.57 places on it. A capability-bracket re-run of four of the deep-five skills with a weaker non-reasoning solver (gpt-4o-mini) yields a near-zero skill-level difference (â0), so the null is not a ceiling effect of solver strength. The exploratory comparison of the maintained version against the no-skill baseline is likewise null. Table 6 collects the battery. AnalysisUnitâ (vnewâvold)Statistic / check Primary (full package)13 skills â0.09sign p=.58, Wilcoxon .55; d=â0.29; CI [â0.28, +0.10]; 5/13 favor v new Length-controlled13 skills â0.06sign test p=0.58; score residualized on log output length Deep-five subset5 skills â0.14sign test p=1.0 Weaker-solver bracket4 skills â +0.00solver gpt-4o-mini (non-reasoning) Harness reliability100 triadsÏ=0.51judge agreement: Pearson 0.57; ICC(2,1) = 0.52 (below 0.6 gate) Table 6: Robustness battery for the powered maintenance-benefit study. The primary skill- level test (top row) is the reportable result; the next three rows are sensitivity checks; the last row reports judging-harness reliability rather than a maintenance-benefit test. No rubric dimension survives Holm correction. Limitations. Tasks are author-and-model-constructed transfer scenarios rather than native repository workflows, and the second task author is a different model (on a three-skill subset), a weaker control than an independent human writer. Theno-skillbaseline is more verbose than the skilled conditions; the primaryvnew-versus-voldcomparison is length-balanced, and the length-controlled re-analysis above absorbs any residual. Every number regenerates from archived logs by the released analysis script; the harness, the frozen tasks, and per-skill provenance are released. D.1 Pilot study The pilot covered the five skills with the deepest edit histories, with six transfer tasks per skill and two decoding seeds. The solver,gpt-4o-mini, answered in a single turn and received 17 Accepted at the COLM 2026 Workshop on Lifelong Agents only theSKILL.mdfile; the judge,gpt-4o, shared the solver âs provider. At the pre-registered 65% effect size, the design had roughly 20% statistical power. Its two pre-registered tests pointed in different directions: a task-level sign test favored the maintained version at 15 wins to 6 with 9 ties (p =0.08), while a rubric-score Wilcoxon test reachedp =0.02 with a mean gain of+0.27 points. The comparison of the maintained version against theno-skill baseline was lopsided (12â45), an artifact of single-turn scoring: a skill that asks a clarifying question is penalized for an incomplete answer, which the powered studyâs equalized multi-turn protocol removes. These limitations, in the package, the interaction protocol, the judge family, and the statistical power, motivated the powered study of Appendix D; under the corrected design, the pilotâs positive trend does not reappear. E A Replay Protocol for Skill Curation The released corpus can be used for retrospective replay. It includes trajectories and a scoring recipe, but no held-out leaderboard and no baseline curator; for that reason we describe it as a protocol rather than a benchmark. An automated skill curator is evaluated by replaying historical maintenance: âą Input: a priorSKILL.mdversion plus the context fields available at that point (commit message, linked issue/PR, repository). âą Target: the next human substantive edit, scored by the operation label, the compo- nent touched, and similarity to the actual patch or summary. âą Split: chronological per skill (predict each edit from earlier history) or repository- held-out (train on four repositories, test on the fifth), with the split named by the released pinned commits. âąMetrics: operation and component match rates; normalized token overlap with the actual diff; ROUGE-L for one-sentence summaries; and a pruning check. The pruning check asks whether the curator ever consolidates or deprecates content rather than only adding it. The observed human rate is about 4%, but it is noisy and selection-dependent; use it as a reference, not a target. For content edits, the diff can also be scored by atomic-fact coverage, following WiNELL (Reddy et al., 2026). Decompose the human edit into atomic units, then credit a soft match if the fact appears anywhere in the file and a hard match if it appears in the correct component. WiNELL finds that models more often identify what to add than where to put it. That is the placement problem isolated by our component analysis (§5, Appendix B). Scope. The context fields may reveal the editâs rationale after the fact, so replay is a hindsight task. A low match rate is still informative: the curator cannot reconstruct edits humans made even when given the historical context around the edit. A high match rate is weaker evidence. It does not show that the same curator could have discovered the edit prospectively, and matching human edits is not deployable autonomous curation. Running a baseline on this protocol is future work. 18