Paper deep dive
SkillApt: Learning When to Activate Agent Skills from Counterfactual Evidence
Shuang Guo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 87%
Last extracted: 9/24/2026, 3:45:13 AM
Summary
The paper introduces SkillApt, a post-retrieval activation framework for Large Language Model (LLM) agents that decides whether to load a retrieved Skill based on counterfactual execution evidence. By comparing WITH/WITHOUT execution outcomes, SkillApt determines the marginal utility of a Skill in a specific state, allowing for selective activation. On the SRA-Bench evaluation, SkillApt-E maintained accuracy comparable to BM25 Top-1 retrieval (0.838) while reducing the Skill activation rate from 100% to 31.5% and mean token usage by 74.3%. The study highlights that Skill utility and activation boundary learnability vary across different base models and execution states.
Entities (11)
Relation Signals (11)
SkillApt-E → achievesaccuracy → 0.838
confidence 95% · SkillApt-E achieved the same observed accuracy as BM25 Top-1 (0.838 vs. 0.838)
SkillApt → evaluatedon → SRA-Bench
confidence 95% · On the frozen confirmatory SRA-Bench evaluation, SkillApt-E achieved the same observed accuracy
SkillApt-E → reducesactivationrate → 31.5%
confidence 95% · reducing the Skill activation rate from 100% to 31.5%
SkillApt-E → reducestokenusage → 74.3%
confidence 95% · reducing ... mean token usage by 74.3%
SkillApt → uses → Counterfactual Evidence
confidence 95% · SkillApt builds execution evidence from matched WITH/WITHOUT runs and uses outcomes from similar historical states to make a LOAD/ABSTAIN decision
SkillApt → appliesto → LLM Agents
confidence 90% · Large language model agents increasingly retrieve reusable Skills... We present SkillApt, a post-retrieval activation framework
SkillApt → comparedwith → BM25 Top-1
confidence 90% · SkillApt-E achieved the same observed accuracy as BM25 Top-1
SkillApt → testedonmodels → Terra
confidence 80% · SpreadsheetBench supplies separately frozen model-conditioned diagnostics for Terra... These studies share the Skill artifact
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model agents increasingly retrieve reusable Skills and inject them into the active context. However, a retrieved Skill can be relevant yet unnecessary, costly, or even harmful in the current execution state. We present SkillApt, a post-retrieval activation framework that decides whether a retrieved Skill should actually be loaded. SkillApt builds execution evidence from matched WITH/WITHOUT runs and uses outcomes from similar historical states to make a LOAD/ABSTAIN decision for each candidate Skill. On the frozen confirmatory SRA-Bench evaluation, SkillApt-E achieved the same observed accuracy as BM25 Top-1 (0.838 vs. 0.838) while reducing the Skill activation rate from 100% to 31.5% and mean token usage by 74.3%. Further diagnostics show that both Skill utility and the learnability of its activation boundary vary across base models. These results suggest that Skill retrieval and Skill activation should be treated as separate decisions: retrieval identifies which Skill may be relevant, while SkillApt determines whether using it is worthwhile in the current state.
Tags
Links
- Source: https://arxiv.org/abs/2609.26863v1
- Canonical: https://arxiv.org/abs/2609.26863v1
Trouble viewing inline? Open PDF directly →
Full Text
62,395 characters extracted from source content.
Expand or collapse full text
SkillApt: Learning When to Activate Agent Skills from Counterfactual Evidence Shuang Guo Central China Normal University Abstract Large language model agents increasingly retrieve reusable Skills and inject them into the active context. However, a retrieved Skill can be relevant yet unnecessary, costly, or even harmful in the current execution state. We present SkillApt, a post-retrieval activation controller that decides whether a retrieved Skill should actually be loaded. SkillApt builds per-Skill evidence from matched WITH/WITHOUT executions and, for a new state, aggregates outcomes from similar historical states to make a LOAD/ABSTAIN decision. On the frozen SRA-Bench confirmatory evaluation, SkillApt-E achieved the same observed accuracy as BM25 Top-1 (0.838 vs. 0.838) while reducing the activation rate from 100% to 31.5% and mean token usage by 74.3%. Further diagnostics show that both Skill utility and the learnability of its activation boundary vary across base models. These results indicate that retrieval and activation should be treated as separate decisions: retrieval identifies which Skill may be relevant, while SkillApt determines whether using it is worthwhile now. Keywords: LLM agents, skill applicability, counterfactual evaluation, selective activation, evidence accumulation 1 Introduction Large language model (LLM) agents increasingly rely on reusable skills. A skill may encode a repair procedure, a tool-use recipe, a domain-specific instruction set, or a multi-step workflow that would be expensive to rediscover on every task Sharma et al. (2022); Wang et al. (2024, 2025a); Zheng et al. (2025). As skill libraries grow, a common execution pattern is to retrieve a relevant skill, inject it into the active context, and let the agent proceed. This pattern is attractive because it separates persistent procedural knowledge from the base model and allows an agent system to reuse experience across tasks Wang et al. (2023, 2025b); Packer et al. (2023). Relevance, however, is not the same as execution util- ity. Consider a spreadsheet-repair skill. In one workbook state, the skill may rescue an execution that would other- wise produce an incorrect formula. In another, the agent may already know how to complete the repair, so the same document merely adds prompt tokens, tool calls, and latency. In a third state, extra instructions may steer the agent toward the wrong range or an unnecessary edit. All three tasks can be semantically related to spreadsheet repair. The practical question is therefore not only which skill is relevant, but: given a candidate skill, should it actually be activated in the current execution state? Existing research provides several pieces of the sur- rounding pipeline. Retrieval narrows a large library to candidate skills Lewis et al. (2020); Robertson and Zaragoza (2009); Su et al. (2026); incorporation and pro- gressive loading study how a model accesses those can- didates Su et al. (2026); skill acquisition and refinement improve what a reusable procedure contains Yang et al. (2026); Tang et al. (2026); Zhao et al. (2024). These are complementary to our question. They do not, by them- selves, estimate the state-conditioned marginal utility of activation from matched executions, nor do they specify how the activation boundary of an existing skill should change as positive, neutral, and negative outcomes ac- cumulate. We therefore treat applicability as a distinct post-retrieval decision rather than as another name for retrieval quality. SkillApt is a post-retrieval controller that uses his- torical WITH/WITHOUT execution evidence to decide whether a retrieved Skill should actually be activated. For a candidate skill and a task state, we compare other- wise matched executions with and without the skill. The resulting marginal evidence records whether activation improves correctness, reduces cost when correctness is tied, has no material effect, or produces negative transfer. A lightweight controller then decides LOAD or ABSTAIN using the current state and historical counterfactual evi- dence. As evidence accumulates, its applicability bound- ary can expand, contract, or remain uncertain. We sepa- rately track environment validity because a historically successful procedure can become unusable when an API, validator, dependency, or tool protocol changes. Empirically, SkillApt-E preserves BM25 Top-1’s ob- served accuracy while substantially reducing activation and token cost. Retrieval remains an upstream bottle- neck, and separate model diagnostics show that both Skill utility and activation-boundary learnability vary across models. The evidence therefore supports selective, model-conditioned activation rather than a universal con- troller. This work makes three contributions: 1.Problem. We formulate skill applicability as state- and model-conditioned marginal utility, distinct from semantic retrieval relevance. 2. Method.We instantiate the counterfactual ap- plicability formulation with a lightweight evidence- conditioned controller and a LOAD/ABSTAIN policy, while tracking environment validity separately. 3. Paired protocol and empirical characterization. We hold task, model, environment, decoding, and 1 arXiv:2609.26863v1 [cs.MA] 22 Sep 2026 evaluator fixed across WITH/WITHOUT executions, and characterize state-dependent utility, retrieval bot- tlenecks, selective activation, model-dependent utility and boundary learnability, and real-world environ- ment validity. 2 Related Work Skill acquisition and evolution. Agents can induce reusable procedures from demonstrations or trajectories, store executable programs, and refine Skill documents from later feedback Sharma et al. (2022); Wang et al. (2023, 2024, 2025a). Recent systems optimize Skill con- tent against practice or held-out outcomes Zheng et al. (2025); Yang et al. (2026); Alzubi et al. (2026); Tang et al. (2026). They primarily ask what a Skill should contain or whether an edit improves it. SkillApt instead holds the procedure fixed and estimates whether activating it is useful in the current state. Skill retrieval and incorporation. Retrieval- augmented generation and lexical ranking provide the standard candidate-generation pattern Lewis et al. (2020); Robertson and Zaragoza (2009). Agent systems retrieve tools, workflows, or Skills and control how those artifacts enter context Yuan et al. (2023); Schick et al. (2023); Wang et al. (2025b). SRA-Bench most directly separates Skill retrieval, incorporation, and downstream execution over a 26,262-Skill corpus Su et al. (2026). We adopt that decomposition: retrieval asks which Skill to consider, whereas applicability asks whether the surfaced candidate should be activated now. Skill evaluation and applicability. Prior bench- marks and Skill-improvement systems already use ab- lations, held-out task outcomes, or measured acceptance criteria Su et al. (2026); Yang et al. (2026); Tang et al. (2026); Alzubi et al. (2026); Zhao et al. (2024). SkillApt studies a narrower estimand: matched per-state outcomes for an already-surfaced, unmodified Skill are accumulated into a state-conditioned activation policy. We therefore do not claim the first Skill ablation, counterfactual Skill evaluation, or activation mechanism. Experience-based agents. Runtime feedback can up- date verbal memory, strategies, workflows, or an agent’s own design Shinn et al. (2023); Madaan et al. (2023); Zhao et al. (2024); Ouyang et al. (2025); Zhang et al. (2025b); Hu et al. (2024). Aggregate improvements and self-judged successes, however, need not isolate the effect of one Skill from task difficulty or base-model capability. SkillApt uses the matched potential-outcome contrast Rubin (1974) and permits abstention as in selective pre- diction Geifman and El-Yaniv (2017). Its deterministic state representation uses feature hashing Weinberger et al. (2009); uncertainty is reported with paired bootstrap re- sampling Efron (1979). Appendix Table 4 gives a detailed, non-ranking comparison. 3 SkillApt 3.1 Problem Setup LetS=S i N i=1 be a Skill library. An upstream retriever returnsC K =R(q,S); applicability is the subsequent activation decision for each candidate, not the retrieval operation. For execution statezand base modelm, define poten- tial outcomes Y (1) i (z,m) : execution with Skill i,(1) Y (0) i (z,m) : execution without Skill i.(2) The state- and model-conditioned marginal utility is ∆ i (z,m) = E h U (Y (1) i )− U (Y (0) i )| z,m i ,(3) and the ideal applicability decision is A i (z,m) = 1[∆ i (z,m) > τ ].(4) SkillApt is a post-retrieval controller that uses historical WITH/WITHOUT execution evidence to decide whether a retrieved Skill should actually be activated. Given a current statez, a candidate SkillS i , and base model m, the SkillApt controller emits Load or Abstain. Re- trieval asks which Skill should be considered; SkillApt asks whether that candidate should enter the active con- text. Its three operational components are counterfactual evidence construction, evidence-conditioned applicability estimation, and a LOAD/ABSTAIN activation policy. Evidence accumulation and environment validity govern later boundary revision and lifecycle actions. Figure 1 separates retrieval, activation, execution, and evidence accumulation. The low-capacity controller iso- lates the activation signal supplied by paired execution evidence. 3.2Counterfactual Evidence Construc- tion For candidateS i and statez j , SkillApt collects a WITH- OUT executionY (0) ij and a WITH executionY (1) ij . The arms share task, model, decoding configuration, environ- ment, evaluator, and companion Skills where applicable; only target-Skill availability changes. The observed con- trast is δ ij = U (Y (1) ij )− U (Y (0) ij ), E t i =(z j ,m j ,δ ij ) t j=1 . (5) The general formulation permits multidimensional ex- ecution utility. In the SRA evaluation, utility is lex- icographic: official correctness has priority, and to- ken, tool-call, and latency costs are consulted only under correctness ties. The frozen taxonomy keeps correctness-positive, correctness-harmful, correctness- tied cost-positive, correctness-tied cost-negative, and neutral outcomes separate. The concrete v1 estimator below is narrower: it encodes only the correctness rela- tion, so both kinds of correctness tie receive target 1/2 even though their cost labels remain distinct for evalua- tion. Infrastructure failure in either arm censors the pair 2 Figure 1: SkillApt architecture. An optional retriever proposes candidates; the applicability estimator combines the current state with stored paired-execution evidence; the activation policy emits LOAD or ABSTAIN; and environment validity can quarantine a Skill independently of its applicability score. Paired execution constructs supervision offline, while evidence accumulation revises the Skill-specific applicability boundary without editing its procedure. and never creates a negative label. Paired execution is used for evidence construction and controlled evaluation; deployment-time SkillApt does not run both arms. 3.3Evidence-Conditioned Applicability Estimation State representation. The frozen SRA v1 estimator represents the task question with a deterministic vector φ(z)∈ R 256 . It lowercases and whitespace-tokenizes the question, adds adjacent word bigrams, assigns each token to a coordinate and sign using SHA-256, sums signed counts, and appliesℓ 2 normalization. This is signed feature hashing, not a learned state encoder. The sep- arate Skill vectorψ(S i ) applies the same transform to immutable name, description, and content text. The in- tentionally low-capacity representation isolates the signal provided by historical execution evidence. Evidence score. The executed SRA v1 scoring path maps historical correctness effects to y ij = 1, Y (1) ij correct and Y (0) ij incorrect, 0, Y (1) ij incorrect and Y (0) ij correct, 1 2 , correctness ties. (6) Thus cost-positive and cost-negative ties remain distinct evaluation labels but both have target 1/2 in the frozen v1 estimator. For a new statez ∗ , the non-negative similarity weight is w j (z ∗ ) = max0,φ(z ∗ ) ⊤ φ(z j ).(7) LetN t i,5 (z ∗ ) be the five highest-weight entries inE t i , or all entries when fewer than five exist. The primary SkillApt- E score is exactly p E,t i (z ∗ ) = 0.5 + P j∈N t i,5 (z ∗ ) w j (z ∗ )y ij 1 + P j∈N t i,5 (z ∗ ) w j (z ∗ ) .(8) This is a unit-weight prior with mean 0.5. With no evidence or no positive similarity, the score is exactly 0.5; no additional clipping is applied.p E,t i is an evidence- conditioned activation score, not a calibrated causal-effect estimate, uncertainty estimate, or hierarchical model. 3 3.4Semantic Baseline and Fusion Abla- tion Semantic-only is a relevance baseline computed as s i (z ∗ ) = 1 + φ(z ∗ ) ⊤ ψ(S i ) 2 .(9) The signed, normalized vectors make the cosine lie in [−1,1], so the affine mapping yieldss i ∈[0,1]. The semantic-fusion ablation, denoted SkillApt-E+Sem, is the implemented arithmetic mean p F,t i (z ∗ ) = 1 2 p E,t i (z ∗ ) + 1 2 s i (z ∗ ).(10) Semantic-only asks whether relevance is sufficient; SkillApt-E is the primary evidence-conditioned controller; SkillApt-E+Sem tests complementarity. The fusion vari- ant is an ablation/high-cost operating point, not the default SkillApt controller. 3.5 LOAD/ABSTAIN Activation Policy Forp i =p E,t i in SkillApt-E, orp i =p F,t i only in the fusion ablation, the frozen SRA threshold isγ= 0.5 and equality abstains: a i (z ∗ ) = Quarantine, V i = invalid, Load,V i ̸= invalid, p i (z ∗ ) > 0.5, Abstain,otherwise. (11) The validity stateV i ∈valid, unknown, invalidrecords current tool and validator preconditions. Validity is a hard precondition rather than an input feature. The frozen SRA scorer assumes included benchmark Skills are valid; the separate real-world validity protocol demon- strates quarantine when a validator or tool contract no longer holds. 3.6 Evidence Accumulation and Bound- ary Revision Controlled prefixesE 1 i ⊂ · ⊂ E t i induce scores p 1 i (z),...,p t i (z) and regionsR t i =z:p t i (z)> γ. Adding evidence can expand, contract, or leave the boundary unchanged because neighbor membership and weighted targets can change. We call this evidence ac- cumulation, evidence-prefix adaptation, and boundary revision. The SRA study uses a frozen ordering for con- trolled analysis; it is neither real temporal drift nor au- tonomous recursive self-improvement, and revision need not be monotonic. 3.7 Algorithm 1 Algorithm 1 SkillApt: counterfactual evidence construc- tion and runtime activation Phase A — Offline evidence construction Require:statez j , candidateS i , modelm, evaluatorQ 1: Y (0) ← Execute(()z j ,m, WithoutSkill(S i )) 2: Y (1) ← Execute(()z j ,m, WithSkill(S i )) 3: if infrastructure failure in either arm then 4:censor pair; assign no utility label 5: else 6: δ ij ← Utility(()Y (1) ,Y (0) ,Q) 7: y ij ← 1, 0, or 1/2 from Eq. (6) 8:append (z j ,m,δ ij ,y ij ) to E i 9: end if Phase B — Runtime activation Require: z ∗ ,S i , evidenceE t i , validityV i , modeb ∈ E,E+Sem 10: if V i = invalid then 11:return Abstain and quarantine S i 12: end if 13: q ← φ(z ∗ ) using normalized 256-d signed hashing 14: N ← five entries (z j ,y ij )∈E t i with largest q ⊤ φ(z j ) 15: for all (z j ,y ij )∈ N do 16: w j ← max0,q ⊤ φ(z j ) 17: end for 18: p E i ← (0.5 + P j∈N w j y ij )/(1 + P j∈N w j ) 19: if b = E+Sem then▷ ablation only 20: s i ← (1 + q ⊤ ψ(S i ))/2; p i ← (p E i + s i )/2 21: else 22: p i ← p E i ▷ primary SkillApt-E 23: end if 24: if p i > 0.5 then 25:return Load S i 26: else 27:return Abstain 28: end if Algorithm 1 is the concrete SRA SkillApt v1 procedure. It has no learned parameters or threshold optimization: the score equation and strict>0.5 decision are fixed. SpreadsheetBench uses separately fitted diagnostic con- trollers and is not an implementation of this algorithm. At ordinary inference, SkillApt makes one activation decision and the agent performs one actual execution. The WITH/WITHOUT pair is required for controlled evidence collection, not for deployment-time inference. 4 Experimental Setup Benchmarks and models. SRA-Bench is the primary evaluation, with 5,400 tasks, 26,262 corpus Skills, and 636 gold Skills Su et al. (2026). We froze 48 Skill identities and five development plus five locked-final states per Skill. Four domains were executable: CHAMP Mao et al. (2024), LogicBench Parmar et al. (2024), MedCalc-Bench Khandekar et al. (2024), and TheoremQA Chen et al. (2023). SRA runs use the API-reportedgpt-5.6-terra identity with temperature 0, top-p1, low reasoning effort, and unchanged official evaluators. SpreadsheetBench Ma et al. (2024) supplies separately frozen model-conditioned 4 diagnostics for Terra, Qwen3-32B, Mistral-Small-3.2-24B, and Claude Sonnet 5. Appendix Table 5 gives complete coverage. Policies and metrics. Never Load measures the base agent; Gold Always receives the known gold set as a refer- ence; and BM25 Top-1 always loads the first retrieved can- didate. Semantic-only, SkillApt-E, and SkillApt-E+Sem score BM25 Top-10 and load at most one candidate. We report official correctness, activation, useful-target recall, correctness-harm and cost-negative avoidance, unneces- sary activation, tokens, tool calls, and reliable latency. Frozen protocol and uncertainty. Skill/state sam- pling and the 5+5 split were frozen with seed 20260922. After a 32-state exploratory retrieval screen, the end-to- end protocol was frozen and the remaining 113 states des- ignated confirmatory. Two persistent no-Skill timeouts were censored, leaving 111 complete states; infrastructure failures never create utility labels. The primary compar- isons use task-level paired bootstrap resampling with the same seed and McNemar’s test. SpreadsheetBench uses seed 20260921 and model-specific fitted controllers rather than the SRA estimator. Appendix Figure 5 and the frozen manifests record the full protocol. Code and release-safe artifacts will be released. 5 Results 5.1 Utility Heterogeneity and Relevance Semantic relevance is not a reliable activation rule by itself. Appendix Figure 6 shows overlapping semantic- score distributions across Helpful, Neutral, and Harmful outcomes. More directly, the 145 executable development pairs contain 24 positive, 17 neutral, and 104 negative outcomes, of which 102 are correctness-tied cost-negative and only two are correctness-harmful. 020406080100 Observed outcomes per Skill (%) champ-005 champ-006 champ-010 champ-012 champ-013 champ-016 champ-022 logic-004 logic-006 logic-007 logic-010 logic-013 logic-014 logic-016 logic-017 logic-018 med-002 med-006 med-009 med-014 med-016 med-017 med-020 med-021 med-028 med-030 med-034 med-052 theorem-292 CHAMP LogicBench MedCalc-Bench TheoremQA (a) Four-class utility composition by Skill 0102029 Number of Skills (of 29) ≥2 classes Positive and negative Single class 24/29 19/29 5/29 Criteria overlap; positive + negative is a subset of ≥2 classes. (b) Within-Skill heterogeneity Utility Heterogeneity Across Skills and States PositiveNeutralCorrectness-tied cost-negativeCorrectness-harmful Figure 2: Utility heterogeneity across 29 executable Skills and 290 paired outcomes. Each bar aggregates one Skill’s own states into four utility classes; the summary reports within-Skill heterogeneity. No cross-Skill state alignment is implied. Figure 2 avoids aligning unrelated D/F positions across Skills. Across each Skill’s own ten observed states, 24/29 5 Skills exhibit at least two utility classes, 19/29 exhibit both positive and negative utility, and only 5/29 are single-class. Utility therefore varies both across Skill identities and within a Skill across its execution states. 5.2Selective Activation: Confirmatory Result BM25 retrieves the target Skill at ranks 1/5/10 for 0.239/0.389/0.513 of the 113 confirmatory targets (MRR@10 0.301). Figure 3 and Table 1 report the 111 complete policy states. 0.00.20.40.60.81.0 Activation rate ↓ 0.80 0.82 0.84 0.86 0.88 O ffi cial accuracy ↑ Never BM25 Top-1 Semantic-only SkillApt-E SkillApt-E+Sem Gold Always† (a) Accuracy vs. activation 10002000300040005000 Mean tokens ↓ 0.80 0.82 0.84 0.86 0.88 O ffi cial accuracy ↑ Never BM25 Top-1 Semantic-only SkillApt-E SkillApt-E+Sem Gold Always† (b) Accuracy vs. token use Frozen confirmatory SRA quality–selectivity tradeoffs (111 complete; 2 censored) Figure 3: Frozen confirmatory operating points. SkillApt-E has the same observed accuracy as BM25 Top-1 with lower activation and token use; SkillApt-E+Sem is a higher-cost fusion point. Gold Always is not an end-to-end retrieval policy. H Table 1: Flagship SRA retrieval–applicability result on 111 complete confirmatory states. Two of 113 frozen targets are infrastructure-censored. Gold Always receives the known gold Skill set and is a reference rather than an end-to-end retrieval policy. PolicyAcc. ↑Activation ↓Mean tokens ↓Useful recall ↑Corr.-harm avoid. ↑Cost-neg. avoid. ↑ Never0.7930.0009350.0001.0001.000 Gold Always a 0.8831.0002,0261.0000.0000.000 BM25 Top-10.8381.0004,7820.1500.5000.759 Semantic-only0.8471.0004,6930.3001.0000.795 SkillApt-E0.8380.3151,2310.2501.0000.916 SkillApt-E+Sem0.8651.0004,6410.4001.0000.819 a The full-gold reference may contain companion gold Skills; retrieval policies load at most one candidate. Bold identifies the proposed controller and its defensible selectivity/cost advantages, not the globally highest accuracy. SkillApt-E and BM25 Top-1 both achieve observed accuracy 0.838, while SkillApt-E reduces activation from 1.000 to 0.315 and mean tokens from 4,782 to 1,231 (74.3%). The paired accuracy difference is 0.0 p with 95% CI [−6.31,+6.31] p and McNemarp= 1.0; this is observed equality, not statistical equivalence or su- periority. The mean token difference is−3,551 with 95% CI [−4,577,−2,629]. SkillApt-E has useful-target recall 0.250, correctness-harm avoidance 1.000, and cost- negative avoidance 0.916. 6 5.3Retrieval Bottleneck and Failure Analysis SkillApt cannot activate a candidate that retrieval never surfaces. On the 111 complete states, SkillApt-E succeeds on 0.879 of the 58 Top-10-hit states and 0.792 of the 53 Top-10-miss states. R@1R@5R@10 0.0 0.2 0.4 0.6 0.8 1.0 Target-Skill recall 0.239 0.389 0.513 (a) BM25 retrieval 01020304050 Diagnostic count Target retrieval miss Useful-target retrieval miss Applicability false abstention Candidate-selection error Negative activation Base-model capability limit Infrastructure censored 53 9 4 2 28 11 2 (b) Non-exclusive failure diagnostics Where failures arise in the end-to-end Skill pipeline Counts overlap by construction and must not be summed. Figure 4: Confirmatory BM25 retrieval and non-exclusive failure diagnostics. Categories overlap by construction and must not be summed. H Table 2: Separately frozen SpreadsheetBench cross-model diagnostics (seed 20260921). These studies share the Skill artifact and paired-evaluation objective but are not exact implementations of SRA SkillApt v1 or Algorithm 1. Parameters are model-specific and no row is a cross-model ranking. ModelTrain P/N/Neg Valid. P/N/Neg Never acc. Always acc. Evidence acc./act. Full acc./act. Diagnostic conclusion Terra18/1/525/2/130.3500.8250.825 / 0.8500.825 / 0.900 selective boundary learnable Qwen3-32B2/0/22not run— ≥ 90% majority-class stop Mistral-Small-3.2-24B4/0/20—0.0250.2250.025 / 0.0000.025 / 0.000 all-abstain collapse Claude Sonnet 54/0/2016/0/240.3750.7500.375 / 0.0000.450 / 0.125 useful Skill, boundary not learned The archived Mistral report supplies the train distribution and validation policy metrics but not a directly comparable validation P/N/Neg row in the manuscript registry. A protocol audit comparing SpreadsheetBench with SRA SkillApt v1 found 1/13 components identical, 1/13 not applicable, and 11/13 different. The table therefore diagnoses model-conditioned utility and activation-boundary learnability, not transfer of one controller implementation. These diagnostics support a conditional conclusion: Skill utility and the learnability of a selective boundary are distinct, model-dependent properties. Appendix Fig- ure 8 visualizes the same result, and Appendix Figure 7 reports Terra’s controlled evidence-prefix adaptation. 6 Discussion and Limitations Retrieval and controller errors. Incomplete re- trieval bounds attainable downstream benefit. The v1 controller is also deliberately lightweight: useful-target recall is limited and negative activations remain. Appli- cability is therefore a post-retrieval complement, not a substitute for candidate generation or base-model capa- bility. Model conditioning. Both usefulness and activation- boundary learnability vary across models. The Spread- sheetBench diagnostics use separately fitted controllers and do not establish transfer of the SRA policy. A useful Skill in aggregate need not admit a boundary recoverable by the current estimator. Environment validity. An anonymized corpus of real- world multi-agent software-engineering trajectories shows that historical frequency can persist after a validator or tool contract becomes invalid. Environment validity should therefore be checked separately from ordinary applicability. Appendix Table 3 and Figure 9 report the controlled evidence. Scope. The public SRA release has no official train/dev/test split, per-Skill evidence is sparse, and final Skill identities are not held out. The cost-sensitive taxonomy is only partially represented by the v1 es- timator because all correctness ties receive target 1/2. Two SRA domains and external SkillsBench paired vali- dation remain infrastructure-blocked. The anonymized real-world study contains few natural negative examples and supports lifecycle analysis, not broad deployment generalization. Treatment compliance, cold start, Oracle headroom, and future estimator detail are reported in 7 Appendix. 7 Conclusion Skill activation should be treated as a post-retrieval deci- sion grounded in matched execution evidence, not as an automatic consequence of semantic relevance. The frozen SRA result shows that SkillApt-E can retain the observed BM25 Top-1 accuracy operating point while substantially reducing activation and token cost, although the accuracy difference remains uncertain. Retrieval limits attainable benefit, and cross-model diagnostics show that both util- ity and boundary learnability are model-dependent. The evidence supports selective activation, not a universal controller or a new retrieval method. References Salaheddin Alzubi, Noah Provenzano, Jaydon Bingham, Weiyuan Chen, and Tu Vu. EvoSkill: Automated skill discovery for multi-agent systems, 2026. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1–3, 1950. doi: 10.1175/1520-0493(1950)078⟨0001:VOFEIT⟩2.0.CO;2. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, 2023. doi: 10.18653/ v1/2023.emnlp-main.489. B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1–26, 1979. doi: 10.1214/aos/ 1176344552. Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2017. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 2017. Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems, 2024. Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Se- rina S. Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Con- rad W. Safranek, Abid A. Anwar, Andrew Zhang, Aidan Gilson, Maxwell B. Singer, Amisha Dave, Andrew Taylor, Aidong Zhang, Qingyu Chen, and Zhiyong Lu. MedCalc- Bench: Evaluating large language models for medical cal- culations. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. doi: 10.52202/079017-2690. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ̈uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ̈aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks, 2020. Zhenyang Lin, Yurou Chen, and Zhiyong Liu. AutoSkill: Hier- archical open-ended skill acquisition for long-horizon manip- ulation tasks via language-modulated rewards. IEEE Trans- actions on Cognitive and Developmental Systems, 2025. doi: 10.1109/TCDS.2025.3551298. Shuo Liu, Yutong Yang, Haohao Xiao, Mouxing Yang, and Xi Peng. Imagine before retrieval: Prospective skill retrieval for LLM agents, 2026. Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench: Towards challenging real world spread- sheet manipulation, 2024. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Halli- nan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2023. doi: 10.52202/075280-2019. Yujun Mao, Yoon Kim, and Yilun Zhou.CHAMP: A competition-level dataset for fine-grained analyses of LLMs’ mathematical reasoning capabilities. In Findings of the As- sociation for Computational Linguistics: ACL 2024, 2024. doi: 10.18653/v1/2024.findings-acl.785. Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. ReasoningBank: Scaling agent self-evolving with reasoning memory, 2025. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems, 2023. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behav- ior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023. doi: 10.1145/3586183.3606763. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Naka- mura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. LogicBench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. doi: 10.18653/v1/2024.acl-long.739. Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 2009. doi: 10.1561/ 1500000019. Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent, 2025. Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688–701, 1974. doi: 10.1037/ h0037350. Timo Schick, Jane Dwivedi-Yu, Roberto Dess`ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. Advances in Neural Information Processing Systems 36. 8 Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022. doi: 10.18653/v1/2022.acl-long.120. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflex- ion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. doi: 10.52202/075280-0377. Weihang Su, Jianming Long, Qingyao Ai, Qiaozhi He, Yichen Tang, Changyue Wang, Yiteng Tu, Yingbo Wang, and Yiqun Liu. Skill retrieval augmentation for agentic AI, 2026. Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, and Tu Vu. WikiSkill: Com- piling agent experience into persistent knowledge for skill evolution, 2026. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandku- mar. Voyager: An open-ended embodied agent with large language models, 2023. Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Pan- pan Xu, and Lin Lee Cheong. Reinforcement learning for self-improving agent with skill library. In Proceedings of the 64th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 2026. doi: 10.18653/v1/2026.acl-long.69. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In International Con- ference on Machine Learning (ICML), 2024. URLhttps: //openreview.net/forum?id=jJ9BoXAfFa. Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, and Daniel Fried. Inducing programmatic skills for agentic tasks, 2025a. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Gra- ham Neubig. Agent workflow memory. In International Conference on Machine Learning (ICML), 2025b. URL https://openreview.net/forum?id=NTAhi2JEEE. Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML), 2009. doi: 10.1145/1553374.1553516. Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. SkillOpt: Executive strategy for self-evolving agent skills, 2026. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023a. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023b. URL https://openreview.net/forum?id=WE_vluYUL-X. Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji. CRAFT: Customizing LLMs by creating and retrieving from specialized toolsets, 2023. Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Godel machine: Open-ended evolution of self-improving agents, 2025a. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation.In International Conference on Learning Representations (ICLR), 2025b. URLhttps: //openreview.net/forum?id=z5uVAKwmjf . Oral presenta- tion. Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang, Rishi Mullur, Thai Hoang, Zhiyuan Hu, James Zhu, Phil Mui, Silvio Savarese, Ran Xu, and Zeyuan Chen. DarwinX: Evolving agent harnesses through natural selection, 2026. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are ex- periential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. doi: 10.1609/aaai.v38i17. 29936. Qiyan Zhao, Xiaofeng Zhang, Bo Liu, Minda Chen, Wei Xiong, Jingyang Chen, Guanting Ye, Wenhao Yu, Xiaosong Yuan, Shijie Han, Da-Han Wang, Jianmin Ji, Fei Huang, and Xu- Yao Zhang. Coalition-aware skill reliability for self-evolving agents, 2026. Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srini- vasa, Gaowen Liu, Graham Neubig, and Yu Su. SkillWeaver: Web agents can self-improve by discovering and honing skills, 2025. Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control, 2023. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. ToolQA: A dataset for LLM question answering with external tools, 2023. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiao- heng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions, 2024. 9 A Evidence-prefix diagnostic The separately frozen SpreadsheetBench Terra diagnos- tic, rather than SRA SkillApt v1, provides the available evidence-prefix analysis. Its controller differs from Al- gorithm 1; the result diagnoses whether a boundary is learnable under that model-specific protocol. On Terra validation, activation moves 1.00→0.95→0.80→0.90 as 25/50/75/100% of development evidence is exposed. Negative-state avoidance rises from 0 to 0.231, while Brier score falls from 0.336 to 0.198 and ECE from 0.347 to 0.068. Boundary flips are concentrated at the 75% and 100% prefixes. Appendix Figure 7 shows what happens as evidence accumulates under a controlled prefix order in this diag- nostic. It matters because the lifecycle claim is specif- ically that a boundary can be revised, not that it con- verges: the activation rate moves non-monotonically (1.00→0.95→0.80→0.90) while calibration improves steadily (Brier 0.336→0.198, ECE 0.347→0.068). The figure therefore supports boundary revision and calibration gain, and simultaneously refutes a stronger monotone-convergence reading of the same data. B Real-world lifecycle evidence The public benchmark isolates controlled counterfactual utility; an anonymized corpus of real-world multi-agent software-engineering trajectories adds lifecycle context. A deterministic miner extracted 76 episodes spanning lint repair, runtime/build repair, tool/path recovery, com- ponent/API lookup, and delivery self-audit. A small manually defined gold set yielded 95.24% partial bound- ary precision, 100% boundary recall, and 100% noise rejection, but the rules and gold labels were developed on the same samples. These are therefore grade-C life- cycle diagnostics, not public-benchmark generalization evidence or a direct replication of Algorithm 1. The trajectory artifacts also expose a distinct validity problem. A skill can be historically frequent while its associated validator no longer detects the target failure. Applicability should thus be paired with an environment- validity stateV i ∈ valid, invalid, unknown; a lifecycle can promote, restrict, or retire a skill without editing its procedure. Table 3 and Appendix Figure 9 carry the real-world part of the argument. The table matters because it is the only place where a paired counterfactual is run against a deterministic validator outside a public benchmark, and it separates the two effects we actually measured: a pure efficiency effect (±0.0 p success,−18% tokens) and a combined effect (+20.0 p success,−31% tokens). It also records what we refused to measure: the environment- drift family has no WITH/WITHOUT arms because its validator is invalid, so no effect estimate would be meaningful. Figure 9 generalizes that refusal into the lifecycle rule this paper argues for — validity is checked before utility, and historical frequency is never sufficient for promotion. Capability ceiling and oracle headroom. Appli- cability selects between available treatments; it cannot repair a capability failure shared by both arms. On 125 LiveMath held-out cases, an offline correctness-first or- acle reaches 0.696 accuracy at 0.240 activation, versus 0.600 accuracy at 0.432 activation for the frozen con- troller. Both arms are wrong on 38/125 cases (30.4%), directly quantifying the portion that activation policy alone cannot solve. The oracle reuses completed paired outcomes and was not used to modify the controller. Appendix Figure 10 bounds what activation policy can achieve at all. It matters because it distinguishes a controller failure from a capability failure: an offline correctness-first oracle reaches 0.696 accuracy at 0.240 activation against the frozen controller’s 0.600 at 0.432, so headroom exists, but 38 of 125 cases are wrong in both arms and lie outside the reach of any activation decision. The figure supports the limitation claim rather than a performance claim. Treatment compliance. Prompt injection makes the skill available but does not guarantee that the model follows it. Our estimand is intention-to-treat for skill availability, not the effect among trajectories that demon- strably complied. Fine-grained compliance analysis would require a validated trace-level usage classifier and is left for future work. Cold start. The primary split shares skill identities across development and final. A held-out-skill study would require a disjoint identity split and a shared con- troller that represents skill content without per-skill out- comes. Such a study is scientifically valuable but must use a new preregistered manifest; it cannot be retrofitted onto the 5+5 within-skill protocol. Future method extension. A future SkillApt version could replace the current low-capacity score with a shared cross-Skill representation and Skill-specific adaptation that maps (z,S i ,m,E i ) to an estimated marginal utility b ∆ i and uncertaintyσ i . This could support risk-aware activation—conceptually, LOAD when b ∆ i − λσ i > τ — and held-out-Skill cold-start evaluation. This uncertainty- aware rule is future work; it is not implemented, evalu- ated, or used in Algorithm 1. Promotion requires marginal evidence. Historical frequency and semantic relevance are useful discovery sig- nals, but neither establishes net benefit. Promotion into a default-loaded skill should require matched or otherwise credible execution evidence. Where paired execution is too expensive, systems should retain uncertainty rather than converting invocation frequency into a positive label. Retirement is distinct from low applicability. A currently invalid validator or tool contract is not merely a low-probability state. Environment validity should be a hard precondition for applicability: when the precon- dition cannot be reproduced through the agent’s actual tool path, the system should retire or quarantine the skill until revalidation. This prevents strong historical evidence from overpowering present incompatibility. 10 Table 3: Controlled real-world validation on anonymized software-engineering trajectories. Both arms of every fixture share the task, pre-state, validator, tool availability and prompt baseline; the only manipulated variable is whether the target skill is visible. Success is a deterministic validator outcome and token change is a median-to-median comparison. Skill familyApps Runs w/o : with Success w/o → with ∆ success∆ tokens Main observed effect dataset- contract-repair 312 : 11100% → 100% ±0.0 p ▼ 18% efficiency only eslint-global- import-cleanup 315 : 1580% → 100%+20.0 p ▼ 31% correctness + efficiency Not eligible for a WITH/WITHOUT effect estimate Deterministic episode miner 2— in-sample diagnostic a eslint-app- api-chain-repair 4— validator invalid; rejected b Design. Nine fixtures×2 conditions×3 repetitions = 54 designed runs. No run was voided by timeout or circuit breaker. Of the 27 WITH runs, 26 demonstrably loaded the target skill; the single no-treatment run is excluded from the dataset-contract-repair denominator, which is why its WITH arm has 11 rather than 12 runs. Token medians.dataset-contract-repair203,892→166,492;eslint-global-import-cleanup1,051,919→ 729,809. Per-repetition spread is large (up to 133% of the median within one fixture and arm), so the token column is a direction, not a precise effect size. Correctness effect. The +20 p figure comes from two of fiveeslint-global-import-cleanupfixtures whose WITHOUT arm passed only 2/3 and 1/3 repetitions and whose WITH arm passed 3/3. The other seven fixtures were already at 3/3 without the skill, so their effect is confined to cost. a The miner produced 76 episodes from two applications, but its rules and gold labels were developed on those same samples. It is lifecycle evidence, not a measured outcome claim. b A historically frequent validator family that no longer triggers on the current tool path. We deliberately ran no WITH/WITHOUT arms for it; see the environment-validity discussion in Section B. C Detailed related-work compari- son Additional neighboring settings include embodied Skill acquisition Lin et al. (2025), reinforcement learning over Skill libraries Wang et al. (2026), coalition-aware relia- bility Zhao et al. (2026), and prospective Skill retrieval Liu et al. (2026). ReAct and deliberate search orga- nize execution Yao et al. (2023b,a), while trajectory and generative-agent memories persist experience Zheng et al. (2023); Park et al. (2023). Self-referential systems may edit agent code itself Zhang et al. (2025a); Robeyns et al. (2025); Zhang et al. (2026). These are adjacent design ob- jectives rather than priority comparisons. The diagnostic calibration measures use the Brier score Brier (1950) and expected calibration error Guo et al. (2017). D Detailed experimental coverage The frozen collections and model records are summarized in Table 5. E Exploratory retrieval pilot The 32-state screen was observed before the post-screen confirmatory freeze and is therefore exploratory. BM25 retrieved the target Skill in Top-10 for 14/32 states and missed it for 18/32. SkillApt-E and BM25 Top-1 both had observed accuracy 0.938; SkillApt-E activated on 0.344 of states and used 1,219 mean tokens, versus activation 1.000 and 4,474 mean tokens for BM25 Top-1. This direction is consistent with the confirmatory quality–selectivity pat- tern, but the pilot is not pooled into primary confidence intervals. FSupplementary diagnostic fig- ures GGold-candidate utility compo- sition The development-only heterogeneity counts are 19/29 Skills with at least two classes and 13/29 with both signs. Domain coverage is uneven: CHAMP has 35 conditional multi-Skill assignments, LogicBench 45, MedCalc-Bench 60, and TheoremQA five states for one qualifying Skill. These strata do not support balanced domain-level popu- lation claims. The table below reports pre-existing gold- candidate collections used for utility characterization and evidence construction. The 145 locked-final labels are not the untouched retrieval-policy confirmatory set. HAuxiliary paired and real-world results SkillApt-E’s unnecessary activation rate is 0.252. SkillApt-E+Sem achieves observed accuracy 0.865 with activation 1.000, making it a higher-cost fusion point. All three paired accuracy intervals in Table 7 include zero; token reduction against always-active retrieved policies ranges from 73.5% to 74.3%. 11 Table 4: Where applicability estimation sits relative to skill-centric agent research.✓marks a capability the cited work explicitly claims; (✓) marks a related but weaker form, defined in the notes; an empty cell means the capability is outside that work’s stated scope. Columns C and D are deliberately separated: several prior systems already gate skill edits on a measured held-out score, which is not the same as estimating a per-state counterfactual effect of activation. A Skill retrieval B Skill content evolution C Utility-gated acceptance D Paired per-state counterfactual E Activation boundary F Environment validity Voyager Wang et al. (2023)✓ Agent Workflow Memory Wang et al. (2025b)✓ ReasoningBank Ouyang et al. (2025)✓(✓) a SkillWeaver Zheng et al. (2025)✓(✓) b SkillOpt Yang et al. (2026)✓ WikiSkill Tang et al. (2026)✓ EvoSkill Alzubi et al. (2026)✓ SRA-Bench Su et al. (2026)✓ This work(✓) c ✓ Column definitions.A a candidate set is produced from a library rather than enumerated in full.B the stored procedure itself is written, edited or refined.C a change to the library is admitted only if a measured held-out score improves.D the effect of a skill is estimated from matched executions that differ only in whether that skill is available, for an individual task state.E a state-conditioned decision rule determines whether an existing, unmodified skill is activated.F a validity state is tracked separately from utility so a skill can be restricted when its environment changes. a ReasoningBank distils memories from the agent’s self-judged successful and failed experiences; the signal is the agent’s own judgement rather than a held-out score from an official evaluator. b SkillWeaver practises candidate skills and distils the practice experience into APIs, which filters unreliable skills, but admission is not reported as a held-out-score gate. c We consume a candidate set produced by an upstream retriever and make no retrieval contribution; the gate is retriever- agnostic by construction. Figure 5: Detailed paired experimental protocol. Every included state has matched WITH/WITHOUT arms and an official evaluator outcome. Infrastructure failures are censored rather than converted into negative utility. I Frozen SRA sampling protocol The eligibility pool contains the 133 gold skills with at least ten distinct valid task states. Counts by domain are 37 BigCodeBench, 7 CHAMP, 19 LogicBench, 55 MedCalc-Bench, 1 TheoremQA, and 14 ToolQA. Equal allocation was impossible because CHAMP and The- oremQA had insufficient capacity. The deterministic largest-remainder redistribution selected 10, 7, 9, 12, 1, and 9 skills respectively. Within each selected skill, a seeded hash order selected ten globally distinct task IDs; positions 1–5 are development and 6–10 are final locked. Development and final task-ID overlap is zero. TheofficialcorpusarchiveSHA-256is 2e8abf91...673849; the decompressed corpus JSON SHA-256 is16e509a...b5e. Each selected skill has an independent content hash and each state has a canonical metadata hash. All detailed manifests are machine-readable. J Infrastructure exclusions ToolQA Zhuang et al. (2023) and BigCodeBench Zhuo et al. (2024) retain their original frozen assignments. SkillsBench v1.1 was audited, but Docker failed before 12 Table 5: Experimental coverage. Utility rows contain matched WITH/WITHOUT executions; retrieval- policy rows contain frozen end-to-end states. Evidence grades follow the registry in Appendix L: A untouched final, B frozen development or validation, C controlled real-world micro-benchmark. SettingDomains Distinct skills Skill–state pairs Model(s) Evidence grade Public benchmarks SRA-Bench frozen manifest a 648480 gpt-5.6-terra— SRA-Bench development429145 gpt-5.6-terraB SRA gold-candidate locked final429145 gpt-5.6-terraA SRA retrieval pilot b 42932 gpt-5.6-terraD SRA retrieval confirmatory c 429113 gpt-5.6-terraA SpreadsheetBench development1124 gpt-5.6-terraB SpreadsheetBench validation1140 gpt-5.6-terraB SpreadsheetBench transfer screen1124Qwen3-32B d SpreadsheetBench transfer train1124Mistral-Small-3.2-24B SpreadsheetBench transfer validation1140Mistral-Small-3.2-24B SpreadsheetBench transfer train1124Claude Sonnet 5B SpreadsheetBench transfer validation1140Claude Sonnet 5B LiveMath held-out1— e 125— e A Anonymized real-world trajectories Deterministic episode mining1— f 76 routed agent stack g C Controlled micro-benchmark129 routed agent stack g C a The untouched original target. Two domains could not execute on this host (ToolQA corpus unavailable, BigCodeBench evaluator incompatible); they are recorded as missing rather than removed from the protocol. b Observed before the post-screen freeze; reported as exploratory pilot evidence only. c Of 113 frozen targets, 111 have complete conditions and two persistent no-Skill timeouts are censored. d Reported as a distribution screen only. A preregistered degeneracy rule stopped before controller fitting, so no validation row exists. e The released LiveMath controller artifact records paired outcomes and accuracy but not skill identities or the base-model string. f Episodes are grouped into five repair families rather than into a fixed skill set; the two families promoted to fixtures appear in the row below. g Executions used the production agent’s configured model routing inside an isolated benchmark stack; the resolved model identity was not recorded in the run artifacts. HelpfulNeutralHarmful Observed utility class 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 Semantic relevance score Semantic relevance does not determine utility Figure 6: Semantic-score distribution by observed utility class on 17 LiveMath validation states. Dots denote individual states and diamonds denote class means. inference; it supplies no paired outcome. ToolQA requires an external corpus distributed separately from the SRA repository. Attempts from the available networks could not retrieve the official Google Drive object, and no unver- ified replacement corpus was substituted. BigCodeBench generation completed, but the released sandbox assumes Linux-style multiprocessing and resource limits. On the available macOS host, spawn could not pickle the evalua- tor’s local process target, while fork-based adaptations encountered host resource/Objective-C constraints. Be- cause changing the official evaluator would alter the pro- tocol, all BCB evaluation attempts are quarantined as infrastructure diagnostics. KDetailed model-diagnostic in- terpretation The SpreadsheetBench studies share paired execution, a fixed Skill artifact, and a correctness-first utility objective, but not the exact SRA SkillApt v1 estimator. The 13- item protocol audit found one identical component, one not applicable, and eleven differences. The diagnostic seed is 20260921, distinct from the SRA seed 20260922. Terra’s mixed training distribution permits a selective validation boundary. Qwen’s 91.7% negative training share triggers the preregistered degeneracy rule. Mistral completes validation, but its fitted controllers abstain 13 0.0 0.2 0.4 0.6 0.8 1.0 Rate Activation Negative avoidance 0.0 0.1 0.2 0.3 0.4 Calibration error ↓ Brier ECE 255075100 Development evidence used (%) 0 2 4 6 8 10 Boundary fl ips 255075100 Development evidence used (%) 0.0 0.2 0.4 0.6 0.8 1.0 Probability / accuracy Mean probability Policy accuracy Terra validation: controlled evidence accumulation is non-monotonic Figure 7: Terra evidence-prefix behavior under the separately frozen SpreadsheetBench diagnostic protocol. This is not an SRA SkillApt v1 replication. Terra Qwen3-32B Mistral 24B Claude Sonnet 5 0.0 0.2 0.4 0.6 0.8 1.0 Training utility fraction 18/1/52/0/224/0/204/0/20 (a) Paired train utility PositiveNeutralNegative TerraMistral 24BClaude Sonnet 5 0.0 0.2 0.4 0.6 0.8 Accuracy / activation Qwen: stopped before validation (22/24 negative) (b) Validation controller behavior Never acc. Always acc. Evidence acc. Evidence act. Fusion acc. Fusion act. Model-conditioned Skill utility and boundary learnability Figure 8: Cross-model diagnostic summary under the separately frozen SpreadsheetBench protocol. This is not a replication of SRA SkillApt v1. 14 Table 6: Utility composition of every labelled collection. Negative states are split into correctness-harmful and correctness-tied cost-negative outcomes; collapsing them into a single “harmful” category would misstate the observed risk of activation. Setting / splitPairsPos.Neu. Corr. harmful Cost-neg. (tied) Evidence grade SRA-Bench development14524172102B SRA-Bench locked partial final14529102104A SpreadsheetBench train (Terra)24181— a — a B SpreadsheetBench valid. (Terra)40252— a — a B SpreadsheetBench train (Qwen3-32B)2420— a — a B SpreadsheetBench train (Mistral)2440— a — a B SpreadsheetBench train (Claude)2440— a — a B SpreadsheetBench valid. (Claude)40160123B Real-world micro-benchmark b 99000C a The SpreadsheetBench transfer artifacts record a single aggregate negative class and do not separate correctness-harmful from cost-negative outcomes. Their negative counts are 5, 13, 22, 20 and 20 respectively; we do not split a category the source artifact did not record. b Nine paired fixtures, each executed WITH and WITHOUT the target skill for three repetitions (54 designed runs). Two fixtures are correctness-positive and seven are cost-positive at tied correctness; no fixture degraded either correctness or cost. One WITH run did not load its target skill and is excluded from the effect denominator as a no-treatment run. Figure 9: Evidence lifecycle: environment validity is checked separately from counterfactual utility before pro- motion, restriction, or retirement. NeverAlwaysOursOracle 0.45 0.50 0.55 0.60 0.65 0.70 Accuracy 0.512 0.560 0.600 0.696 Held-out accuracy NeverAlwaysOursOracle 0.0 0.2 0.4 0.6 0.8 1.0 Activation rate 0.000 1.000 0.432 0.240 Held-out activation LiveMath held-out oracle headroom (125 cases) Figure 10: LiveMath held-out oracle analysis. The oracle has accuracy and activation headroom, while 38/125 both-wrong cases remain outside applicability selection. Table 7: Paired confirmatory comparisons centered on SkillApt-E (111 states). Accuracy intervals are task-level paired bootstrap intervals; all accuracy differences remain statistically uncertain. Comparison∆Acc. (p)95% CI (p)∆Act. (p)Token reduction vs BM25 Top-10.00[−6.31, +6.31] −68.574.3% vs Semantic-only−0.90[−8.11, +5.41] −68.573.8% vs SkillApt-E+Sem −2.70[−9.01, +2.70] −68.573.5% throughout. Claude Sonnet 5 shows why utility and learnability must remain separate: Always Load improves accuracy from 0.375 to 0.750, yet Evidence-only exactly reproduces Never (0.375 accuracy, zero activation) and Full fusion reaches only 0.450 at 0.125 activation. These negative results constrain the formulation. The model conditionmin ∆ i (z,m) indexes a materially different potential-outcome surface, while the diagnostic protocol separately tests whether its low-capacity estimator can recover a useful boundary. LEvidence grading and release status Grade A denotes post-freeze confirmatory evaluation; Grade B denotes frozen development or validation; Grade C denotes controlled real-world microbenchmarks; Grade D denotes diagnostics or pilot evidence; and Grade E denotes censored or invalid evidence. The 111-complete- state SRA confirmatory result is Grade A within its four-domain subset; the 32-state retrieval screen is Grade D. Terra, Mistral, and Claude are Grade B Spreadsheet- Bench diagnostics under a different controller protocol. Qwen’s train screen is Grade B diagnostic evidence with- out validation. SkillsBench has audit evidence only and no outcome. The trajectory segmentation gold set is Grade C/D due to in-sample construction. No private raw trajectory identifiers, prompts, credentials, endpoints, organization names, product names, app IDs, team iden- tifiers, or absolute paths are included in public-facing 15 Table 8: Threats, observed failure modes, and the claim restrictions they impose. Threat / failure Evidence and claim restriction / mitigation Partial SRA coverage ToolQA corpus unavailable and BigCodeBench evaluator incompatible with the current host. The retrieval confirmatory result covers 111 complete states in four domains and censors two timeouts; retain the original freeze and make no six-domain claim. Retrieval bottleneck Confirmatory target Recall@10 is 0.513. Applicability cannot recover an absent candidate; report conditional success and retrieval misses separately. Sparse per-skill evidence Five development states per skill. Use a shared interpretable controller; claim within-skill state generalization, not precise independent per-skill boundaries. Diagnostic protocol difference SpreadsheetBench differs from SRA SkillApt v1 on 11/13 audited components. Use it only for model-conditioned utility and boundary-learnability diagnostics. Model- conditioned collapse Qwen stops before validation; Mistral and Claude controllers underperform Always Load. Do not claim universal cross-model success. External validation blocked SkillsBench v1.1 was audited, but Docker failed before model inference. Report no SkillsBench outcome. Environment drift A historically frequent validator family no longer triggers. Probe the current tool path; restrict or retire invalid skills. Treatment noncompliance One WITH run did not load its target skill. Report treatment integrity and exclude no-treatment runs from attributable effects. Capability ceiling LiveMath has 38/125 both-wrong cases. Applicability cannot repair a skill or base model that fails in both arms. claims. MWithin-Skill frozen-order diag- nostic The diagnostic matrix in Figure 11 retains the original frozen ordering for auditability only. D1–D5 and F1– F5 denote the first five development and five locked- final positions within each Skill ; D1 for one Skill is not the same execution state as D1 for another. Columns must therefore not be interpreted as globally aligned task states. 16 D1D2D3D4D5F1F2F3F4F5 Within-Skill frozen order (columns are not shared execution states) champ-005 champ-006 champ-010 champ-012 champ-013 champ-016 champ-022 logic-004 logic-006 logic-007 logic-010 logic-013 logic-014 logic-016 logic-017 logic-018 med-002 med-006 med-009 med-014 med-016 med-017 med-020 med-021 med-028 med-030 med-034 med-052 theorem-292 CHAMP LogicBench MedCalc-Bench TheoremQA Within-Skill frozen-order diagnostic D/F positions are not comparable across Skills PositiveNeutralCorrectness-tied cost-negativeCorrectness-harmful Figure 11: Within-Skill frozen-order diagnostic for the 29 executable Skills. Columns show each Skill’s own D1–D5 development order and F1–F5 locked-final order; they are not shared execution states across rows. Colors preserve four utility categories, the vertical rule separates development from final, and horizontal rules preserve domain grouping. 17