Paper deep dive

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review

Shuyang Zhang

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 89

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language model (LLM) agents increasingly draw on external tools and reusable skills selected at run time from libraries that hold thousands of entries. Reports that a retriever, router, or skill library "improves" an agent may refer to retrieval recall, the success change from enabling a library, a paired contrast restricted to triggered tasks, or a gain under an approximate budget constraint. This critical review asks what each design compares and under which assumptions. Building on estimand-based approaches to agent evaluation, we describe tool and skill designs along six axes: treatment contrast, target population, outcome, budget constraint, summary measure, and identification assumptions. Thirteen core empirical studies anchor the evidence synthesis, supplemented by related methodological work and design-level reading of the wider literature. Our contribution is to make explicit distinctions that some source authors already acknowledge through a decomposition of trigger-conditioned pairing, analytic counterexamples, and comparisons across studies. Pairing on the task does not by itself identify an invocation effect; paired gain and regression counts measure protocol-specific discordance rather than the share of tasks whose expected outcomes worsen; and total effects of module deployment answer a different question from budget-constrained efficiency. We compare curated skill provision with retriever replacement, triggered subsets with all-task outcomes, and observed cost reductions with budget-constrained comparisons. A reporting checklist and worked examples connect these distinctions to information that studies can report. The review runs no new experiments; empirical results come from the cited studies, and numerical toy examples are analytical illustrations.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

85,677 characters extracted from source content.

Expand or collapse full text

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review Shuyang Zhang The Hong Kong Polytechnic University, Hong Kong, China Abstract Large language model (LLM) agents increasingly draw on external tools and reusable skills selected at run time from libraries that hold thousands of entries. Reports that a retriever, router, or skill library “improves” an agent may refer to retrieval recall, the success change from enabling a library, a paired contrast restricted to triggered tasks, or a gain under an approximate budget constraint. This critical review asks what each design compares and under which assumptions. Building on estimand-based approaches to agent evaluation, we describe tool and skill designs along six axes: treatment contrast, target population, outcome, budget constraint, summary measure, and identification assumptions. Thirteen core empirical studies anchor the evidence synthesis, supplemented by related methodological work and design-level reading of the wider literature. Our contribution is to make explicit distinctions that some source authors already acknowledge, through a decomposition of trigger-conditioned pairing, analytic counterexamples, and comparisons across studies. Pairing on the task does not by itself identify an invocation effect; paired gain and regression counts measure protocol-specific discordance rather than the share of tasks whose expected outcomes worsen; and total effects of module deployment answer a different question from budget-constrained efficiency. We compare curated skill provision with retriever replacement, triggered subsets with all-task outcomes, and observed cost reductions with budget-constrained comparisons. A reporting checklist and worked examples connect these distinctions to information that studies can report. The review runs no new experiments; empirical results come from the cited studies, and numerical toy examples are analytical illustrations. Keywords: LLM agents; tool retrieval; agent skills; causal inference; estimand; evaluation methodology 1 Introduction Tool use turned LLMs from text generators into agents that act [1–4]. The number of callable capabilities has since grown faster than any context window can hold. Benchmarks now contain thousands of APIs [3, 5], Model Context Protocol (MCP) environments expose hundreds to thou- sands of tools and servers [6, 7], and skill libraries, which package procedures, instructions, and scripts that an agent loads on demand, contain tens of thousands of entries [8, 9]. A recent line 1 arXiv:2609.33153v1 [cs.SE] 27 Sep 2026 of work therefore treats selection as a component in its own right: a retriever or router builds a shortlist, a gate decides whether a candidate is loaded, and the agent decides whether and how to follow it [10, 11]. Evaluation practice has not kept pace with these designs. A paper on a new skill retriever may report Recall@10 on a retrieval benchmark and then an end-to-end success rate with one agent. A paper on a skill library may report the average success change when the library is enabled. A third paper may compare tasks on which the agent retrieved a skill with tasks on which it did not. Each number is sometimes presented as the benefit of skills, yet they compare different things. Cho and Park showed that models can display a positive retrieved-versus-skipped lift together with a negative paired contrast on the tasks where retrieval occurred [12]. Hajimiri et al. found that the gains of online skill and memory modules for web agents often vanish against a vanilla actor given more interaction steps within a comparable budget [13]. Tank and Nama found that the best skill libraries outperform others mainly through fewer paired regressions rather than more gains [14]. In the other direction, SkillsBench reports that curated, task-specific skills raise pass rates in every one of 18 model–harness configurations [15]. Such results do not contradict one another. They answer different questions, and a reader can compare them only after stating which question each design answers. Tool learning [10], agent evaluation and benchmarking [16–19], and trajectory-level failure attri- bution [20] have all been surveyed. These surveys organize the literature by capability, benchmark, metric, or failure type. A directly related methodological study by Huang asks what an LLM- agent leaderboard rank compares [21]. It specifies the target population, measurement source, resource rule, common support, and uncertainty needed to interpret pairwise system comparisons. Estimand-based reasoning is therefore already part of agent evaluation. Our focus is on compar- isons that enable, replace, or trigger a tool or skill component within an agent. We examine how conditioning on a within-run trigger interacts with the coupling of stochastic runs, and why paired outcome flips differ from the share of tasks whose expected outcomes worsen. These component-level questions complement Huang’s analysis of system rankings. The contribution is an explicit analysis and cross-study synthesis of these distinctions, rather than a claim that the source authors overlooked every limitation. Skill Following, for example, already describes its retrieval-invoked contrast as protocol-conditional [12]; Section 3.5 makes the coupling and selection terms explicit. This review contributes: 1. a framework that separates measurement levels from causal paths and describes evaluation designs along six axes, stated in potential-outcome notation (Section 3); 2. an analysis of interpretations that recur in the literature, with analytic counterexamples for trigger-conditioned paired contrasts, for reading stratum effects as invocation effects, and for paired gain and regression counts and their naive task-level summaries, together with the distinction between total and budget-constrained effects (Sections 3.4–3.8); 3. a study-level evidence table for thirteen core studies read in full (Section 4 and Appendix A), 2 followed by a stage-by-stage synthesis of the wider literature (Section 4); 4. a reporting checklist in which each item is tied to the designs it applies to, and a set of open problems (Section 5). 2 Scope and Review Method This is a critical narrative review, not a systematic review or meta-analysis. We do not pool effect sizes or grade study quality. The aim is to examine the design logic of representative studies. The search closed on 25 September 2026. Sources were arXiv, the ACL Anthology, and conference proceedings, searched with combinations of tool retrieval, tool selection, skill retrieval, skill routing, skill library, MCP, counterfactual, and evaluation. We also tracked citations backward and forward from two seed papers: one on actual skill use [12] and one on classifier replacement of agent decision components [11]. We included studies of LLM agents that retrieve, select, activate, or use tools or skills, provided that the studies reported a component-level metric or a task-level outcome. Studies of agent evaluation protocols, such as replay, next-turn scoring, and intervention, were included when their findings bear directly on the same comparisons. Selection was purposive. We sought studies whose designs differ in treatment, population, pairing, or budget, and we included results in which skills help as well as results in which they do not. Evidence was read at two depths. Thirteen empirical studies carry the main component-level arguments of this review. For these we read the full text and recorded the arms, unit of comparison, number of runs and treatment of randomness, any conditioning on post-treatment events, budget handling, endpoint, and the conclusions the design cannot support. We also recorded the target population and any inclusion or censoring rule, the arXiv version read, and the location of each item in the paper (Table 3 and Appendix Table 5). A targeted related-work check on 27 September 2026 added Huang’s methodological study [21], posted before the search cutoff; we read version 1 in full to position the review, without adding it to the thirteen-study component evidence matrix. This check did not update the broader search. For the remaining agent studies we rely on their stated design and headline results. We use them to illustrate the range of designs, not to adjudicate their internal validity. Numbers are quoted as reported. Numbers from different benchmarks, agents, and budgets are never placed on one scale. Most sources are 2026 preprints. Where authors state a venue acceptance, we note it in the references; otherwise, results should be read as unreplicated author reports. One author selected, read, and characterized all studies. 3 A Framework for Tool and Skill Evaluation 3.1 Measurement Levels and Causal Paths We distinguish five levels at which a tool or skill evaluation can take measurements: • Exposure: which candidates the agent can see or call, such as a shortlist, a tool menu, or 3 Trajectory state Exposure RetrievalActivationUseOutcome presence effects: metadata in context, context length actions change the state; the sequence recurs at later steps Figure 1: Measurement levels in tool and skill use (solid arrows) and two paths that a purely sequential reading omits: a direct effect of exposure on the outcome (dashed) and feedback from actions to the trajectory state (dotted). The figure is a reading aid, not a complete causal graph. skill names and descriptions preloaded into the system prompt [22, 23]. • Retrieval: which candidates a retriever or router ranks highly for the current query or state [5, 24]. • Activation: whether a surfaced candidate is loaded into context or executed [25, 26]. • Use: whether the agent follows the loaded procedure or tool output, and how [12, 27]. • Outcome: task success, together with cost, latency, and risk. These levels describe a typical workflow and tell us what an evaluation measures. They are not a complete causal graph. The order can vary: exposure may precede retrieval (preloaded metadata) or follow it (a retrieved shortlist), and in multi-step agents the whole sequence can recur at every step, with each decision changing the state on which later decisions depend. A skill can also affect the outcome without passing through every level. Skill names and descriptions in the system prompt can change behavior when the skill is never invoked [14], and a larger context can change behavior even when selection is correct [28]. Figure 1 shows the typical order together with this direct path and the state feedback. Precise event definitions matter because several studies condition on them. In this review a candidate is surfaced when it appears in the agent’s visible candidate set; retrieval is called (R call = 1) when the agent or harness issues a retrieval request; retrieval returns (R ret = 1) when that request yields at least one candidate; a skill is loaded (R load = 1) when its body enters the context; a tool is executed when a call is issued and returns; and content is followed when the agent’s subsequent actions depend on it in the sense a study defines, for example by a behavioral annotation [12]. A call that returns an empty set differs from one that returns candidates, and the trigger rate and the triggered group change with the event chosen. Studies differ in these definitions, and we note the definition used whenever a result depends on it. Below, R denotes whichever of these events a study conditions on. 4 3.2 Notation, Units, and Coupling We use the potential-outcomes framework [29, 30]. Let x be a task drawn from a task population, and let a denote a configuration of the agent, such as a library enabled, a particular retriever, or a skill forced to load. Agent runs are stochastic even at fixed x, because of sampling, serving nonde- terminism, and environment variation. We write Y a for the outcome of one run under configuration a and μ a (x) =E[Y a | x] for its expectation over runs. When a module is available, R a ∈ 0, 1 records whether retrieval is triggered in that run. R is a post-treatment variable. It exists only in arms where the module is available, and it may depend on the same random events that determine Y a . Three choices must be fixed before any comparison is meaningful. The first is the unit: a task, a task paired with a random seed, or a pair of runs. The second is the coupling of arms. Arms can be run independently given the task. They can share a random seed, as in several studies below. Or both arms can branch from one recorded state. A shared seed specifies a joint sampling protocol. The dependence it induces between the arms should be described, including whether environment and serving randomness are also controlled; a coupling does not require the two arms to produce similar trajectories. The third is the number of runs per task and arm. It determines how precisely μ a (x) can be estimated and which inferences are possible. A single run gives an unbiased but very imprecise estimate. The clinical-trial addendum ICH E9(R1) requires every estimand to specify five attributes: the population, the treatment, the outcome variable, the handling of intercurrent events, and the population-level summary [31]. The same structure applies to agent components. Intercurrent events here include not triggering retrieval, retrieving a skill without loading it, loading it and ignoring it, infrastructure failures, and exhausting the step or token budget. Much of the ambiguity reviewed below comes from leaving one of these attributes, or the unit and coupling, unstated. 3.3 Six Axes for Describing a Design Rather than a single list of “effects”, we describe each design along six axes (Table 1). The treatment contrast answers what is changed. It can be the availability of a module, a swap of one component for another (a retriever, a shortlist head, a tool menu), an activation policy, or the composition of a library. The target population answers over what the quantity is averaged: all tasks, a subgroup defined before treatment, a subgroup defined by a post-treatment event such as triggering, or a set of states within trajectories. The outcome may be success, cost, or risk exposure. The budget constraint may be absent, an ex-ante cap, or an attempt to match realized spending. The summary measure may be a difference in means, a count of paired discordances, the share of tasks whose expected outcome falls, or a value function. The identification assumptions say what must hold for the summary to be a causal quantity rather than a description of one protocol. 5 Table 1: Six axes for describing a tool or skill evaluation design AxisValues found in the literature Treatment contrast Availability (module on vs. off); component swap (retriever, shortlist head, menu constructor); activation policy (load vs. abstain, gate on vs. off); library composition (size, content, admission rule) Target population All tasks; pre-treatment subgroup (domain, difficulty tier); post-treatment subgroup (tasks where retrieval was triggered); states within trajectories OutcomeSuccess or reward; tokens, calls, latency; exposure to risky candidates; intermediate failures Budget constraint None; ex-ante cap with a stated allocation rule; approximate matching of realized spending Summary measure Difference in means; paired discordance counts; share of tasks whose expected outcome falls; state-level value contrast IdentificationRandomized or paired runs over a task distribution; stated coupling between arms; principal ignorability for post-treatment strata; sequential ignorability and positivity for state-level contrasts from logs 3.4 Commonly Reported Quantities and Their Limits Table 2 lists the quantities that recur in the literature, placed on these axes. Two of them are descriptive, not causal. Component metrics such as Recall@k, nDCG, or Hit@1 describe a retriever on a labeled query pool. Retrieved-versus-skipped contrasts,E[Y 1 | R 1 = 1]−E[Y 1 | R 1 = 0] or variants that mix arms, compare two groups that the agent itself selected. Both can be precisely defined statistical quantities, and neither is the effect of anything. Skill Following measures the difficulty gap between the two groups with the paired skill-disabled runs and finds it substantial [12]. The total effect of a contrast between configurations a and a ′ over the task population is τ a,a ′ =E x μ a (x)− μ a ′ (x) .(1) With availability as the contrast, τ answers “should this module, as specified, be switched on?”. With a component swap, it answers “should component A replace component B?”. Both are identified by running both arms on the same tasks, and both are the same kind of quantity. Most retrieval papers that report end-to-end gains estimate a component-swap total effect: the effect of better retrieval, not the effect of skills relative to none. TabAgent’s replacement of an LLM shortlist head by a classifier, with reported maintenance of task-level success on AppWorld [11, 32], is also a component-swap comparison, with shortlisting cost as a second outcome. Budget constraints. A module that consumes tokens changes two things at once: what the agent knows and how much computation it spends. If the treatment is “deploy this module with its overhead”, a well-designed comparison with and without the module identifies the total effect of that package. The extra computation is part of the treatment and one path of its effect, not a confounder. A budget-constrained comparison answers a different question: which configuration 6 Table 2: Commonly reported quantities in tool and skill evaluation, what each supports, and what it does not support QuantityContrast; population SupportsDoes not supportExamples Component metric (Recall@k, nDCG) None; labeled query pool Ranking quality on that pool Any effect on agent outcomes; transfer to other query sources [5, 24, 33] Retrieved vs. skipped contrast None; post-treatment groups Description of where the agent retrieves Any causal effect; groups differ in difficulty criticized in [12] Total availability effect Module on vs. off; all tasks Deploying the module as specified Resource efficiency; which path produced the effect [14, 15, 34] Component-swap effect ComponentA vs.B; all tasks, or fixed evidence paths ReplacingB byA in that agent Effect of skills vs. no skills; value of an additional action [11, 22, 35, 36] Budget- constrained comparison Configurations under a common cap; all tasks Choice between configurations at that budget Total effect without the cap; exactness if the cap is approximate [13] Composition effect Library versions; all tasks Effect of growing or curating the library Effects of individual skills; later library states [28, 37] Trigger- conditioned paired contrast Availability; tasks where the treated run triggered Protocol-specific diagnostic of the triggered segment A run-level invocation effect without coupling assumptions (Section 3.5) [12] Paired discordance counts Any; all tasksFrequency of flips under the protocol Share of tasks harmed; individual harm (Section 3.7) [14, 34] Task-level degradation share Any; all tasksShare of tasks whose expected outcome falls, reported as confidently and possibly degraded shares Mechanism of the fall; naive plug-in estimates from few runs none found State-level intervention value Load vs. abstain at a state; states Activation decisions under a fixed continuation policy The deployed gate’s total effect (Section 3.8) [25] Execution- selection value Run vs. skip the downstream agent; query–bundle pairs Deciding whether to execute, given a utility for skipped runs Effect of loading a skill within a run [26] 7 to choose when total spending is capped. Defining it requires an ex-ante cap B and a rule for allocating B within each arm, because realized spending is itself a post-treatment outcome. The web-agent study of Hajimiri et al. is the clearest example [13]. Its control arm extends the vanilla actor’s horizon from 10 to 15 steps and adds accessibility-tree pruning. The authors state that this step cap approximates, and does not exactly match, the token budgets of the augmented methods; in most tasks the control spends fewer tokens, and in a few it spends slightly more. The finding that augmentation gains often vanish therefore concerns this approximate budget constraint. It does not show that comparisons without budget matching are invalid. A complete report gives the total effect, the costs in each arm, and, where resource efficiency is claimed, a budget-constrained comparison or a cost–success frontier [38, 39]. 3.5 Trigger-Conditioned Paired Contrasts A natural remedy for the retrieved-versus-skipped contrast is to pair each task with a run in which the module is unavailable, and then restrict the comparison to tasks where the module arm triggered retrieval. Skill Following formalizes this as the Retrieval-Invoked Actual-Use Effect (RAE), using the returned-skill event R ret as the trigger [12]. Pairing removes the between-task difficulty gap that makes the retrieved-versus-skipped contrast misleading. Taken alone, it does not make the restricted contrast a causal effect of invocation. Consider a protocol that produces, for each task, one run in the module arm with outcome Y 1 and trigger R 1 , and one run in the control arm with outcome Y 0 . The two runs may be coupled in any way that preserves their marginal distributions: they may be independent given x, share a random seed, or branch from a common prefix. A common prefix is a coupling of the same availability contrast only if it leaves the defined marginal distribution of each arm unchanged; branching from a recorded mid-run state more naturally defines the state-level contrast of Section 3.8. For any coupling that preserves the marginals, the population value of the paired statistic decomposes as D pair =E Y 1 − Y 0 | R 1 = 1 =E μ 1 (x)− μ 0 (x)| R 1 = 1 | z propensity-weighted task effect +E Y 1 − μ 1 (x)| R 1 = 1 | z treated-run selection −E Y 0 − μ 0 (x)| R 1 = 1 | z control-run selection . (2) The first term averages task-level total effects with weights proportional to each task’s trigger probability. It is still an availability effect, not the effect of invoking a skill. The second term is non-zero whenever triggering is correlated with the treated run’s own outcome, for example because an agent retrieves more often after an early mistake. The third term depends on the coupling. If the control run is independent of the treated run given x, the third term vanishes and Equation (2) reduces to the first two terms. Under a shared seed or another dependent coupling, the third term can be non-zero and can offset the second term in part or in full. 8 An analytic counterexample. Suppose all tasks are identical and the module has no effect on the distribution of outcomes: in each arm a run succeeds with probability 0.5, represented by a fair coin U. Let the module arm trigger retrieval exactly when its coin shows U = 1. The true effect is zero for every task. With an independent control run, the paired statistic has expectation 1− 0.5 = 0.5, all of it from the treated-run selection term. If instead both arms read the same coin, the two selection terms cancel and the paired statistic is 0. The example shows that pairing on the task does not remove selection that occurs within the task, and that the size of the resulting bias depends on the coupling. It does not show the direction or size of the bias under any particular shared-seed protocol. Skill Following runs both arms with the same task prompt, generation seed, and decoding configuration (its Section 4.3). This defines a dependent coupling. What the paper does not state is how the shared seed relates the random events that drive triggering in the skill-enabled run to those that drive success in the skill-disabled run, whose prompt lacks the tool definition. Both selection terms in Equation (2) are therefore unknown, and so is their difference. The authors themselves describe RAE as “a protocol-conditional paired outcome signal”, state that it is not an unbiased causal effect over a pre-treatment task population, and list the further ablations that a finer causal decomposition would need, including metadata-only retrieval and oracle retrieval (Limitations). We follow that reading. RAE is a useful diagnostic: under a fixed protocol, did the triggered segment of the chain go with better or worse outcomes than the paired control? It is not a model-level measure of how well skills are used. Two further questions, two different targets. Readers of trigger-conditioned results usually want one of two things, and they require different designs. The first is the availability effect within the triggering stratum: among runs or tasks that would trigger when the module is available, how does having the module change the outcome? Principal stratification defines this target [40], and assumptions such as principal ignorability given observed task features can identify it [41, 42]. For runs rather than tasks, a stated coupling between arms is also needed. The analogy with complier effects in instrumental-variable analysis [43] is instructive but only partial, because the trigger is an event within a stochastic run rather than a stable property of the task. Even when identified, this target remains an availability effect. It still includes every path through which availability acts, such as metadata in the prompt, context occupancy, and behavior before the call. As an analytic example, suppose that listing skill descriptions in the prompt improves every task while the skill bodies themselves are useless. The availability effect among triggering runs is then positive, although loading a body has no effect. The second target is the local invocation effect: at a state where the agent would trigger, with the metadata and remaining budget held fixed, how does loading the skill compare with not loading it, given a stated policy for the rest of the run? This target is defined by intervening on the load decision itself (Section 3.8). The design must state which metadata remain visible in both branches, how the budget is accounted for, and which continuation policy is used. The two targets answer different questions, and neither can be read off the other. 9 3.6 Linking Total and Trigger-Conditioned Contrasts Let π = P(R 1 = 1) be the trigger rate under a given protocol, and let D skip =E[Y 1 − Y 0 | R 1 = 0] be the paired contrast on the complement. For 0 < π < 1, the law of total expectation gives τ pair = π D pair + (1− π)D skip ,(3) where τ pair is the paired mean over all tasks. Skill Following reports the empirical version of this identity as its Equation (4), decomposing the overall skill-access effect into retrieval-invoked and skipped components [12]. We restate it here to make three points explicit. First, the identity holds for the paired statistics of any protocol and any coupling. It says nothing about the causal interpretation of its components. The decomposition of Equation (2) applies to D skip as much as to D pair , with the event R 1 = 0 in place of R 1 = 1. Second, the total and one component do not determine the other unless π is also reported: a complete report gives (π,D pair ,D skip ) or, equivalently, (π,τ pair ,D pair ). When π = 1 the complement is empty and D skip is undefined; when π = 0 the same holds for D pair . Third, a non-zero D skip is consistent with presence effects of the kind described in Section 3.1, but it does not demonstrate them, because it also contains selection terms. Attributing it to metadata in context requires an arm in which only the metadata is present, which is the ablation Skill Following names [12]. The Regression Tax study documents presence-only flips through trajectory evidence, namely regressions in which no skill body was read [14]. This is valuable mechanistic evidence, but the study runs each condition once per task, and one author assigned the mechanism labels; it does not include a metadata-only arm. 3.7 Discordance and Degradation With binary success and one run per arm per task, every task falls into one of four cells. Let ˆp + be the share of tasks that fail without the module and succeed with it, and ˆp − the share that succeed without it and fail with it. Then ˆp + − ˆp − equals the paired difference in success rates. This identity is exact, and reporting both terms rather than their difference is informative [14, 34]. The difficulty is interpretation. It is tempting to read ˆp − as the share of tasks the module harms. That reading is not supported. The marginal success rates of the two arms do not determine their joint distribution. If both arms succeed with probability 0.5 on a task, the chance that the pair shows a regression is 0 when the runs are perfectly coupled in the same direction, 0.25 when they are independent, and 0.5 when they are perfectly anti-coupled. Under a protocol that runs the arms independently given the task, the expected observed regression rate is E[ˆp − ] =E x μ 0 (x) 1− μ 1 (x) ,(4) which is positive even when μ 1 = μ 0 for every task. Repeating runs estimates this protocol-specific 10 rate more precisely, and a same-configuration noise floor, as in the Replay Gap study’s control forks [44], shows how much of it arises without any treatment difference. Neither step identifies how many tasks are causally harmed. We therefore distinguish two quantities. Paired discordance is the observed frequency of gains and regressions under a stated protocol, with its coupling and number of runs. Task-level degradation share is δ ε = P x μ 1 (x)− μ 0 (x) <−ε ,(5) the share of tasks in the population whose expected outcome falls by more than a margin ε. For a finite benchmark of n evaluated tasks, the corresponding quantity is δ ε,n = n −1 P n i=1 1μ 1 (x i )− μ 0 (x i ) < −ε. Under a stated task population and a model of repeated runs, δ ε is estimable. Its estimation error, however, depends on the number of runs per task and on how many tasks have effects close to the margin, and no fixed small number of runs makes a naive estimate reliable. An analytic counterexample. Let both arms succeed with probability 0.5 on every task, so that δ 0 = 0. Run each arm three times per task, independently, and count the tasks whose treated sample mean is below the control sample mean. The count of successes in each arm is Binomial(3, 0.5); the two counts are equal with probability 20/64, so the treated count is lower with probability 22/64≈ 34%. The plug-in estimate of δ 0 is therefore centered near 0.34 rather than 0, and adding tasks makes it more stable without removing the bias. Two reporting options avoid this problem without requiring a stable point estimate from every paired study. One is to construct simultaneous confidence intervals [L x ,U x ] for each task’s difference μ 1 (x)− μ 0 (x) and to report two shares: the share of tasks that are confidently degraded (U x < −ε) and the share that are possibly degraded (L x < −ε). On the event that all intervals cover, these two shares bound δ ε,n for the evaluated benchmark. Extending the bound to δ ε for a wider task population additionally requires a stated sampling design for the tasks and a population- level interval. The other is a hierarchical model with stated assumptions about the distribution of task-level effects. Among the core studies, the Regression Tax study runs each condition once per task and states that run-to-run variance is not estimated [14]. Vasudev et al. report recovery and disruption counts from paired runs, with per-seed results [34]. Both report paired discordance. We found no study in our sample that reports δ ε or either of the two shares. 3.8 State-Level Intervention Value Activation gates and stopping rules decide, at a particular state, whether to load a candidate or to stop acquiring tools [25, 38]. The quantity they aim to approximate must be defined as an intervention, not as a difference between observed groups. Let φ denote a fixed policy for the rest of the trajectory, including its remaining budget, and let C denote total cost from the decision onward. Define Q φ Y (s,a) =E Y a,φ | S = s , Q φ C (s,a) =E C a,φ | S = s ,(6) 11 v φ (s,k) = Q φ Y (s, load k)− Q φ Y (s, abstain) − λ Q φ C (s, load k)− Q φ C (s, abstain) ,(7) where Y a,φ and C a,φ are the outcome and cost when action a is taken at state s and φ is followed afterwards. The cost term is the difference in total subsequent cost, not only the cost of loading k. When v φ is estimated from logged runs, success and cost must each be identified, and the corresponding observed quantity is E[Y | s, load k]−E[Y | s, abstain] − λ E[C | s, load k]−E[C | s, abstain] .(8) A success difference alone corresponds to the first bracket of Equation (7), not to v φ ; with a success difference of 0.1 and a weighted cost difference of 0.2, for instance, v φ = −0.1. Equation (8) equals v φ when the logged runs in both arms follow φ after the decision, and when three further conditions hold: consistency between the logged actions and the interventions; exchangeability of the two actions at the current decision given s; and positivity, meaning that both actions occur at every state of interest. If the logs continue under a different policy, the raw means in Equation (8) estimate the value of that logging continuation, not of φ, and they must be replaced by an off-policy identification formula and estimator for φ. Such formulas require, in addition, that the recorded history suffices for sequential exchangeability at every later decision and that the logging policy supports the actions φ takes. Methods for single-step contextual bandits [45, 46] illustrate the idea of correcting for the behavior distribution but do not by themselves cover these multi-step conditions. Exchangeability at the current decision alone does not remove differences in how the rest of the run was played. Randomizing the action at the decision point secures exchangeability by design, but the continuation condition still applies. Branching both actions from a recorded state and continuing each in closed loop also secures exchangeability, provided that the full state can be restored, the environment is reset validly, and the continuation policy is stated. Among the core studies, SkillApt is the closest approximation of this quantity for skill activation. It defines potential outcomes for an execution state and builds evidence from matched WITH and WITHOUT executions that share task, model, decoding, environment, and evaluator [25]. In its frozen confirmatory evaluation, the state is represented by hashed features of the task question, so the learned utility is conditional on the task rather than on a mid-trajectory state. Correctness is primary and costs serve only as tie-breaks. Pairs in which either arm suffers an infrastructure failure are censored, and two persistent no-skill timeouts left 111 of 113 confirmatory states complete, so the confirmatory result is a complete-case analysis. Confidence-gated retrieval with matched trajectory replay illustrates the difference between this quantity and neighboring ones [36]. It holds candidate answer states, evidence points, budgets, and costs fixed and compares confidence-to- action controllers on these fixed paths, which is a controller swap. Calibrating the probability that the current answer is correct, comparing controllers on fixed paths, and estimating the incremental value of another retrieval are three different targets. The study addresses the first two and concludes that the third requires a separate value-of-information estimate. It therefore shows why calibration cannot substitute for estimating v φ . 12 A different decision is execution selection: whether to run the downstream agent at all on a given query and retrieved bundle. RADEG is of this kind [26]. Its score predicts the probability that executing the agent on the query–bundle pair yields a non-zero verifier reward, and its gate decides whether to launch that execution (its Section 4.1). The gate does not choose between loading and not loading a skill within one run, so it does not estimate v φ (s,k): there is no baseline in which the same state continues without the skill. If a skipped execution is assigned zero reward, the gate targets a well-defined execution-selection utility, and the probability of non-zero reward, the expected reward, and the cost of execution must then be kept apart. The bundle-perturbation study that motivates RADEG is a separate treatment contrast, a comparison of bundle compositions on the same query, and should not be conflated with the gate’s prediction target. Because the gate does not alter the executed agent’s internal policy, its policy value can be evaluated from logged executions when the logs support both decisions, the execution mechanism is fixed, tasks are independent, and the reward feedback follows the stated protocol. RADEG’s evaluation on logged rollouts with query-level held-out splits is of this type. It differs in kind from replaying trajectories in which an intermediate action has been changed (Section 4.5). For all such gates, a state- or query-level estimate must be followed by an evaluation of the total effect, or the policy value, of the policy the gate induces, because accurate local estimates do not guarantee a beneficial policy [34]. 4 Evidence Synthesis Table 3 summarizes the thirteen core studies read in full, and Appendix Table 5 gives the under- lying details. The rest of this section places them, together with the wider literature, along the measurement levels of Section 3.1. 4.1 Exposure and Candidate Construction What the agent can see bounds everything downstream. Four observations recur in this literature. Relevance to the query may be the wrong unit. Several studies argue that the object to retrieve is an executable set rather than a list of individually relevant items. Tool Graph Retriever adds dependency information so that prerequisite tools are not omitted [47]. HYSET scores whole tool sets as hyperedges and reports gains in both retrieval and end-to-end success on ToolBench [48]. Diverse Skill Routing penalizes redundant skills to improve full coverage of multi-skill queries [49]. R3-Skill retains rejected skill combinations as supervision for set compatibility [50], and CaSKG calibrates directed edges between skills with counterfactual probes before graph expansion [51]. State-Path Tool Menus treat the menu as an execution prior over routes from the observable state to the goal. On ToolBench, with one executor and deterministic decoding, the authors report online success rising from 0.737 to 0.898, and their 32-tool menu covers more complete chains than the official 128-tool list [22]. For tasks that require complete tool chains, set-level coverage of the required chain is therefore a useful candidate component metric. Whether it predicts outcomes 13 Table 3: Core studies (full-text reading): what each design compares and the condition that most limits its interpretation. Appendix Table 5 gives arms, populations, runs, censoring, endpoints, versions, and locations in each paper. StudyQuantity (our reading) Condition that limits interpretation Skill Following [12] Overall paired effect; trigger-conditioned paired contrast (RAE) Restricts to tasks where the enabled run returned a skill; shared-seed coupling with unknown selection terms Regression Tax [14] Total availability effect; paired discordance One run per task and condition; variance not estimated Budget study [13] Comparison under an approximate budget constraint Budget matched through a step cap, not exactly; control arm also adds pruning Skill shadowing [28] Composition effect with counterfactual decomposition Population restricted to task–model pairs whose skills each raised pass rate by at least 4 percentage points; bounds need a monotonicity assumption SkillsBench [15] Total availability effect of curated per-task bundles Task-specific bundles supplied in the evaluation; not retrieval from a shared library SkillApt [25]Task-conditional load vs. abstain utility State represented by question features; complete-case analysis (111 of 113 states) RADEG [26]Execution selection: predicted probability of non-zero reward Gate decides whether to run the agent, not whether to load a skill; each pair executed once Confidence- gated retrieval [36] Comparison of confidence-to-action policies Fixed evidence paths; does not estimate the value of another retrieval Replay Gap [44] Validity of replay for per-step model switching Tested for model switching only Failure prevention [34] Total effect of intervention; recovery and disruption counts Counts are paired discordance, not shares harmed State-Path menu [22] Component-swap total effect One executor with deterministic decoding for the main result Capability Pages [35] Component-swap total effect Retrieval and use contributions not separated TabAgent [11]Component-swap total effect with cost One benchmark and decision head for the task-level result 14 better than single-item relevance across settings has not been tested directly. How many candidates to show is itself a decision. Repantis et al. note that no standard metric evaluates whether a shortlist length suits a query. They propose a chance-corrected measure and report, in a downstream check with one model, that shorter adaptive lists improve correct-tool selection [23]. CAM-DF treats tool acquisition as cost-aware stopping over ranked prefixes and reports exposing the agent to 37% fewer tools than full access with comparable task success in live execution [38]. SchemaRouter finds that minimizing the number of selected fields reduced answer accuracy, while a recall-preserving projection restored it [52]. All three show that exposure trades recall against context load and cost, and Recall@k alone does not reveal these trade-offs. Exposure has a risk dimension. Risk-aware reranking annotates 6,108 tools with ordinal risk levels and measures risky-tool exposure in the top-k results [53]. SameCapRisk-Bench shows that public skill retrievers retrieve a helpful skill at Recall@3 between 0.848 and 0.888, yet expose a same-family risky sibling at HSR@3 between 0.346 and 0.372 [54]. A high component metric can therefore coexist with a harmful exposure profile, and the latter needs its own measure. Component metrics do not transfer across query sources. On ToolRet, a retriever fine-tuned on one source-specific slice collapses on another slice of the same benchmark [55]. In a produc- tion router over 34,396 skills, fine-tuning on synthetic data improved in-distribution retrieval but caused catastrophic forgetting on real and out-of-distribution queries [9]. Retrieval benchmarks built from generated queries [33, 56] are especially exposed to this gap between evaluation queries and deployment traffic. 4.2 From Retrieval Gains to Outcome Gains Many retrieval papers now add an end-to-end check. Skill2Query reports that improved retrieval translates into higher agent success across several LLM backends [33]. Capability Pages improve Recall@10 for all five tested retrievers and add 3.62 points of end-to-end success on average when negative-boundary text is included, with both arms sharing the executor [35]. SkillDreamer, SkillDAG, and SkillLens report both retrieval and task-level gains [57–59]. These results mea- sure beyond the retrieval level. In the terms of Section 3.4, each is a component-swap total effect within one agent, one benchmark, and usually one budget. Three limits follow. Curated provision and retriever replacement. A component-swap effect does not show that skills help relative to no skills. A better retriever can raise success over a weaker one while the whole module still lowers success relative to no module. SkillsBench instead compares no skills with curated, task-specific bundles, which raise the macro pass rate from 33.9% to 50.5% across 18 configurations [15]. These bundles are supplied for the evaluated tasks, so the result concerns their availability, not retrieval from a shared library. The same study reports that skills the agent generates for itself fall below the no-skills baseline on the three configurations where this condition was run. Availability effects therefore depend strongly on what the library contains. Read together, SkillsBench and the retrieval studies support distinct decisions. Curated provi- 15 sion tests whether a particular bundle helps when supplied; retrieval from a shared library tests the deployed selection package against its stated baseline; replacing a retriever or its representations tests which component performs better within that package. Capability Pages, for example, com- pares representations with and without negative-boundary text while holding the executor fixed [35]. Its gain supports that replacement in the evaluated setting. It neither estimates the no-skills con- trast in SkillsBench nor transports SkillsBench’s gain to shared-library retrieval. A study addressing both deployment and component choice would need a no-skills arm as well as the alternative retrieval arms on the same task population. The present cross-study evidence motivates those contrasts but cannot supply their missing outcomes. Attribution and metric transfer. The path from better recall to better outcomes is assumed rather than tested. An outcome gain that accompanies a recall gain is consistent with better candidates being used, but also with changes in context length, formatting, or the presence of different text [14]. Capability Pages present their product-form success model explicitly as an attribution sketch rather than an identity [35]. Designs that separate the paths remain rare. The invocation-conditioned decomposition of Song and Wei [28] and the trigger-conditioned pairing of Skill Following [12] are exceptions, subject to the qualifications of Sections 3.5 and 3.6. Where component metrics and outcomes have been compared directly, the link is weak. In multi- turn customer-support workflows with tool calls, next-turn success under gold history improved for every fine-tuned model, yet strict workflow success in autonomous execution reached at most 10.4% [60]. TabAgent illustrates a further outcome profile: it reports maintained task-level success on AppWorld alongside lower shortlisting latency and inference cost [11, 32]. 4.3 Activation and Actual Use A surfaced candidate is not necessarily loaded, and a loaded candidate is not necessarily used well. Recent work separates these levels. Retrieval versus activation. SkillApt treats “which skill is relevant” and “whether loading it is worthwhile” as separate decisions. On its frozen SRA-Bench evaluation it reports the same observed accuracy as BM25 Top-1 (0.838 vs. 0.838) while cutting the activation rate from 100% to 31.5% [25]. SkillApt targets a task-conditional value of loading (Section 3.8). For confidence-gated retrieval in question answering, matched trajectory replay compares confidence-to-action controllers on fixed paths and shows that calibration changes which questions are answered without estimating the benefit of another retrieval [36]; it is a useful counterexample to treating calibrated confidence as a value estimate. RADEG addresses a neighboring decision, whether to run the downstream agent at all on a query and its retrieved bundle; its gate predicts the probability of a non-zero verifier reward, and its motivating study separately shows, with same-query bundle perturbations, that reward is sensitive to bundle composition while relevance scores predict it poorly [26]. Their shared lesson is that the value of a candidate depends on context and must be estimated from contrasts, not read off relevance scores. 16 Activation as a learning problem. When skill selection is learned inside a policy, credit as- signment is itself an attribution problem. SkillGate identifies “selector credit starvation”: under sequence-level outcome reward, the few tokens that name a skill receive a vanishing and increas- ingly wrong-signed share of the credit as trajectories lengthen, and the method adds an action-local advantage for those tokens [61]. Outcome-reward training attributes a trajectory-level contrast to individual choices, which is the same gap that separates total effects from state-level values. Use versus obedience. An agent may call a tool and adopt its output without judgment. Wang and Vemuri found that a ReAct agent given a graph neural network as a tool agreed with the tool’s raw predictions 97.6–99.2% of the time, and that deference increased with backbone capability [27]. A selective-invocation gate recovered part of the gap to an oracle but yielded no net global gain. Conversely, silent tool failures, such as a cached error page in the expected format, are consumed as fact. Outcome Monitors that flag contract violations and name recovery tools raised completion in injected-failure settings, and removing the recovery-tool list eliminated the measured gain [62]. Both findings show that “the tool was used” is not a success criterion; an evaluation must specify what correct use means. Triggered subsets, paired flips, and expected task effects. Skill Following reports, across 17 LLMs on coding and mathematics tasks, that models can have a positive retrieved-versus-skipped lift and a negative paired contrast on the tasks where retrieval occurred [12]. The authors already delimit the paired contrast as protocol-conditional. Our decomposition explains why: both which tasks enter the subset and which stochastic outcomes enter within a task matter, with the latter de- pending on the coupling. The sign of this contrast cannot be substituted for the all-task availability effect or the effect of loading a skill at a fixed state. Regression Tax and Failure prevention retain paired outcome changes as well as aggregate suc- cess [14, 34]. They show why an average can conceal gains on some observed pairs and losses on others. Their treatments and populations differ: one changes skill libraries in document and spreadsheet tasks, while the other adds critic interventions across question-answering and interac- tive tasks. Their repetition also differs: Regression Tax runs each condition once per task, whereas Failure prevention uses two or three seeds depending on the configuration. These are complemen- tary demonstrations of paired discordance, not directly comparable estimates of a common harm rate. More seeds can characterize variation in the reported counts, but the counts alone do not esti- mate the share of tasks with lower expected success. That target requires the per-task uncertainty treatment in Section 3.7. Together, the three studies distinguish a triggered-segment diagnostic, an all-task outcome dif- ference, and a distribution of task-level expected effects. A report can legitimately contain all three only if its design and uncertainty analysis support each. Regression Tax’s trajectory labels also suggest presence, grounding, and verification mechanisms, but the one-run design and absence of a metadata-only arm limit causal attribution to those paths [14]. 17 4.4 Library Scale, Evolution, and Cost Effects of individual skills do not add up to the effect of a library. Song and Wei study SkillsBench task–model pairs in which every authored skill raised the pass rate by at least 4 percentage points over no skills (38 pairs from two models). Moving from these helpful skills to a 202-skill library lowers the pooled pass rate by 21 percentage points (26 for one of the two models), and the authors decompose the drop by conditioning on which skills the agent invoked [28]. The results therefore concern a population selected for skills that help, and the text we read does not state whether the runs used for this selection were separate from those used for estimation. Their decomposition holds either the invocation probabilities or the conditional pass rates fixed, and their upper bounds rely on an assumption that conditional pass rates under the full library do not exceed those under the helpful set. At 202 skills, the selection term (“skill shadowing”) is 0.14 with a 95% interval of 0.06 to 0.26, while the context term is 0.07 with an interval of −0.13 to 0.25 that includes zero; the authors conclude that selection is the main bottleneck. Self-evolving agents that distill skills from their own trajectories show a phase transition past a critical pool size. New skills degrade performance, defective skills seed later ones, and post-hoc removal recovers only a small fraction of the loss [37]. Admission gating before skills enter the pool avoided this in the authors’ experiments. Composition effects are therefore path-dependent: the effect of a library depends on its history, and a total effect measured on an evolving library holds only for the snapshot tested. Total outcomes, observed cost, and budget constraints. TabAgent and the web-agent budget study answer different resource questions. TabAgent replaces the shortlist decision head within an AppWorld agent and reports maintained task-level success alongside lower shortlisting latency and inference cost [11]. Cost is an outcome of that replacement. The comparison can support choosing between the tested configurations on their observed cost–success profile; it does not establish how they would compare after each used a common budget or reinvested its savings. Hajimiri et al. instead strengthen the vanilla web-agent control by extending its actor horizon from 10 to 15 steps and adding accessibility-tree pruning [13]. Under this approximate resource comparison, the control matches or surpasses three online augmentation methods in aggregate success across four WebArena domains and three models, often with fewer total tokens. The authors explicitly note that the step cap does not exactly match token budgets. Because both the horizon and pruning change, this comparison assesses the augmented systems against that specified control package; it does not isolate the effect of extra steps alone. The same study separately ablates horizon extension and pruning under Gemini 3 Flash across the four WebArena domains (its Section 5.3, Table 5), reporting that the longer horizon mainly improves success while pruning mainly reduces token use [13]. Neither result negates a total gain from deploying a module under its original configuration. Together they show why an evaluation should state the deployment contrast, report realized cost alongside success, and separately test an ex-ante resource rule when claiming efficiency under a cap. The different benchmarks, agent configurations, and allocation rules prevent pooling these studies 18 into a general verdict on augmentation efficiency. Liu and Li likewise treat cost and utility jointly when profiling and attributing agent trajectories [39]. 4.5 Validity of Evaluation Protocols The designs above depend on the protocol that generates their counterfactual outcomes. Four protocol-level findings bear directly on tool and skill evaluation. Replay is not intervention. Offline replay evaluates a changed component by substituting its out- put into logged trajectories while assuming that the rest of the trajectory is unaffected. Gonuguntla tested this assumption for per-step model switching on SWE-bench [63] with branching rollouts and same-model control forks. Swaps rewrote 61–94% of post-fork actions, only 3% of replayed states remained valid, and replay mispredicted every success-relevant outcome [44]. The study concerns model switching, but the same logic applies to any component that changes what the agent does next: its counterfactual outcomes should come from executed continuations unless replay has been validated for that component. Local accuracy is not intervention benefit. A critic with an offline AUROC of 0.94 caused a 26- percentage-point collapse on one model and almost no change on another. The authors explain this with a disruption–recovery trade-off and propose a 50-task pilot to decide whether to intervene [34]. For activation gates, a gate’s offline accuracy does not determine the total effect of the policy it induces. Stochastic execution. In the Replay Gap study, temperature-0 serving diverged on over 90% of control forks under one quantization configuration [44], and Hajimiri et al. report that run-to-run variance changed their conclusions [13]. Single-run designs cannot separate treatment differences from this variation at the task level, which is why Section 3.7 separates discordance from degrada- tion. Off-policy estimation needs support. Where logged data rather than new runs are used, classical off-policy estimators apply [45, 46], but only for actions that the logging policy actually took. A working paper on counterfactual tool ranking with exact-propensity logs finds that an initially favorable doubly robust result reversed against full-return direct regression in one setting, and that absolute policy values are not identifiable for unsupported actions [64]. Personalized skill routing takes a different route: a profile-counterfactual benchmark holds the task fixed and varies only the user profile, so the effect of the profile on which skill is correct is built into the data [65]. Finally, the outcome variable must capture the failures that matter. ToolChain-CRC shows that calibrating on the final answer alone can miss retrieval and tool failures, whereas trajectory-level calibration kept accepted-trajectory risk below its target [66]. An outcome that ignores intermediate tool errors can make a harmful component look neutral. 19 5 Discussion 5.1 Cross-cutting Lessons Name the treatment and the question. “Skills”, in current papers, can mean making metadata visible, retrieving with a particular retriever, loading a skill, or following it, and a comparison can ask whether to deploy a module, which component to prefer, or what to do at a given state. These are different contrasts with different answers. A paper should state which contrast it manipulates and which question it answers before reporting results. Treat post-treatment strata with care. Pairing on the task removes between-task selec- tion but not within-task selection (Section 3.5). A trigger-conditioned contrast should be reported together with the trigger event, the trigger rate, and the complementary contrast, and it should be interpreted as a protocol-specific diagnostic unless the coupling between arms and the identi- fying assumptions are stated. Even when identified, an effect within the triggering stratum is an availability effect; a claim about invoking a skill needs a design that intervenes on the load decision. Report discordance as discordance. Counts of gains and regressions are informative [14, 34], but they describe a protocol, not the share of tasks harmed (Section 3.7). Claims about harm to tasks need repeated runs per arm and interval-based summaries, such as the shares of confidently and possibly degraded tasks, or a stated model for task-level effects. Distinguish total effects from budget-constrained comparisons. Both are legitimate. A total effect answers whether to deploy a module as specified; a budget-constrained comparison answers which configuration to choose under a cap. Resource-efficiency claims need the latter, or a cost–success frontier [13, 38]. Match component metrics to task needs. For tasks that require tool chains, set coverage and prerequisite recall are closer to what the task needs than single-item relevance, and risk exposure needs its own measure [22, 48, 54]. Execute counterfactuals, or validate replay. Replay and gold-history scoring answer dif- ferent questions from closed-loop deployment [44, 60]. When full closed-loop evaluation is too costly, a small closed-loop pilot can at least check the sign of an intervention’s effect [34]. 5.2 Reporting Checklist Table 4 lists the information we recommend for papers that claim a tool or skill component improves an agent. Each item states the designs to which it applies. Most items can be reported from runs that authors already perform. Applying the checklist. The following examples use the source locations recorded in Appendix Table 5. They illustrate interpretation, rather than score the studies or validate the checklist. Coupling and identification. Skill Following’s Section 4.3 reports shared task prompts, gen- eration seeds, and decoding configurations; its Limitations explicitly delimit RAE as protocol- conditional [12]. This supports a diagnostic on the triggered subset under that pairing protocol. 20 Table 4: Reporting checklist for tool and skill evaluation ItemWhat to reportApplies to Question and contrast Deployment, component choice, or state-level decision; what each arm sees All designs Target population Task distribution and whether it matches the query distribution of any retrieval benchmark used All designs Unit, coupling, runs Unit of comparison; how arms are coupled (independent, shared seed, branched state); runs per task and arm; run-to-run variance All designs with stochastic agents IdentificationAssumptions under which the reported summary is causal, or a statement that it is protocol-specific Designs that condition on post-treatment events or use logs Trigger stratification Trigger rate π and paired contrasts on triggered and non-triggered tasks, with the trigger definition Designs in which retrieval or invocation is optional Discordance and degradation Gain and regression counts under the stated protocol; if harm to tasks is claimed, confidently and possibly degraded shares from simultaneous per-task intervals, or a stated hierarchical model Paired designs; degradation shares only for harm claims BudgetTokens, calls, and latency per arm; for efficiency claims, an ex-ante cap with allocation rule, or a cost–success frontier All designs; cap only for efficiency claims ProtocolClosed loop, branching, or replay; if replay, evidence that it is valid for the component Designs using logged trajectories Component metrics Set or prerequisite coverage and risk exposure, in addition to single-item recall Retrieval and menu studies Library stateLibrary size, version, and admission rulesStudies of libraries that change 21 A local invocation claim would additionally need an intervention on loading at a fixed state, with metadata, remaining budget, and continuation policy specified. The gap is between targets, not an unacknowledged causal claim by the source authors. Discordance and degradation. Regression Tax’s Sections 3.4–3.6 and 6.2 report one run per condition and task and state that run-to-run variance is not estimated [14]. The paired counts support a description of flips in those executions. A claim about the share of tasks whose expected success decreases would require repeated per-task evidence with simultaneous uncertainty bounds or a stated model; repeating the same discordance summary would not by itself supply that claim. Budget. The web-agent budget study’s Sections 3.2–3.3, 5.1, and 5.3 and Appendices B–D specify the longer actor horizon, pruning, repeated runs, and approximate token comparison [13]. These support a comparison with that control package. An exact equal-budget claim would additionally need a common ex-ante resource cap and allocation rules for every arm. This reporting distinction preserves the study’s own qualification of its budget control. 5.3 Open Problems Invocation effects with run-level triggers. The trigger is an event within a stochastic run, not a fixed property of a task. Principal-stratification methods identify availability effects within strata under assumptions such as principal ignorability [41, 42]. In the studies reviewed here, we found none that states a potential-outcome model for runs, including how triggering varies across runs of the same task and how the arms are coupled, under which a stratum effect or a local invocation effect is identified. Specifying such models for agents, and designs that intervene on the load decision at scale, remains open. Affordable valid counterfactuals. Branching rollouts and paired closed-loop runs are expen- sive [44]. Methods that decide when replay is safe, or that allocate a small number of closed-loop runs to the most informative states, would make valid evaluation cheaper. Benchmarks with built-in identification. Profile-counterfactual [65] and same-family risk [54] benchmarks show that identification can be designed into the data. Benchmarks that include a metadata-only arm, with skills visible in the prompt but not retrievable, would make presence effects measurable as a matter of routine. Evaluating evolving libraries. When libraries change during deployment [37], the natural target is an admission policy rather than a fixed snapshot. How to evaluate an admission policy without re-running the entire evolution is an open problem. From state-level values to policy effects. Activation gates trained on state- or task-conditional utilities [25] and execution gates trained on predicted rewards [26] induce policies whose total effect or policy value, and degradation share, must be evaluated separately [34]. A link between the accuracy of the utility estimates and the degradation share of the resulting policy would connect activation research with evaluation. 22 5.4 Limitations of This Review This review is narrative and not exhaustive, and work posted after 25 September 2026 is not covered. Selection was purposive, and one author read and characterized all studies, so no inter- rater check was possible. The thirteen component evidence studies in Table 3 and Huang’s related methodological study were read in full; the remaining agent studies are characterized by their stated designs and headline results, and a full reading could qualify some of those characterizations. The targeted related-work addition described in Section 2 does not establish exhaustive coverage. Most sources are preprints whose results have not been independently reproduced. We make no quantitative synthesis, because benchmarks, agents, budgets, and outcome definitions differ too much for pooled numbers to be meaningful. The framework of Section 3 is an analytical tool derived from the literature and from standard causal inference, not a validated instrument; the checklist examples illustrate its use but do not validate it. 6 Conclusion Tool and skill components are currently evaluated with a mix of retrieval metrics, component- swap comparisons, retrieved-versus-skipped contrasts, trigger-conditioned paired contrasts, gain and regression counts, and budget-constrained comparisons, often under the same word. We have argued that these designs compare different things. Some are descriptive, some estimate total effects of deploying or swapping a component, and some describe one protocol without identifying a mechanism. Three interpretations in particular need care. Pairing on the task does not turn a trigger-conditioned contrast into an invocation effect. Regression counts from independent runs measure discordance rather than harm. A comparison without budget matching estimates a total effect rather than an invalid one. Stating the contrast, population, unit and coupling, budget, summary, and identification assumptions of each design costs little, and it lets readers compare results that currently appear to conflict. Agents now draw much of their capability from what they retrieve. Evaluating that retrieval therefore requires stating precisely what each design shows. A Study-Level Evidence Table Table 5 records, for each core study, the arXiv version read, the arms and their coupling, the target population and any inclusion or censoring rule, the number of runs, the endpoint, and the location of the relevant material in the paper. “Runs” refers to repeated executions; bootstrap resamples are listed separately where a study uses them. All studies were read in the stated version in September 2026. 23 Table 5: Study-level evidence table for the thirteen core studies Study (version) Arms and coupling Population, inclusion, censoring Runs; endpoint; budget Location in paper Skill Following [12] (v1) Search tool defined vs. removed from the prompt; same task prompt, generation seed, and decoding in both arms 80-task MBPP+ sample (seed 42), HumanEval+, 80-task Math500 partitions; 17 LLMs; RAE restricted to tasks where the enabled run invoked retrieval and a skill was returned One skill-enabled and one skill-disabled execution per task in each cell; counts pooled over config-identical reruns where these exist; harmful-transition annotation and skill-content controls over MBPP+ partition seeds 42–44; correctness; no budget constraint §3.1–3.2 (Eq. 4), §4.1, §4.3, §5.2, §5.5, App. A, Table 17, Limitations Regression Tax [14] (v1) No skills vs. three skill libraries; only the library changes within a stack 486 tasks (OfficeQA-Pro, SpreadsheetBench) × 3 model–harness stacks × 4 conditions; mechanism labels from traces by one author One run per task and condition; variance not estimated; grader pass/fail; no budget constraint §3.4–3.6, §5, §6.2, App. C Budget study [13] (v2) AWM, ASI, ReasoningBank (10-step actor) vs. vanilla actor with 15 steps and accessibility-tree pruning Four WebArena domains, three models; WorkArena-L1 with Qwen 3.6-27B Three runs per domain (three seeds per task type on WorkArena-L1); success and tokens; approximate budget via step cap §3.2–3.3, §4, §5.1, §5.3, App. B–D Skill shadowing [28] (v2) Library of oracle skills vs. expanded libraries of 52, 102, 202 skills SkillsBench task–model pairs kept only if every skill in the authored bundle raised pass rate by at least 4 percentage points over no skills: 38 pairs (21 Haiku 4.5, 17 Sonnet 4.6); the text read does not state whether filtering and estimation used separate runs 2,545 trajectories reported in §4 and App. A.1; the condition counts in Table 2 sum to 2,576, a discrepancy unresolved in the cited version; two-stage clustered bootstrap with 2,000 resamples (resamples, not runs); pass rate; pooled drop at 202 skills 0.21 (21 percentage points), largest single-model drop 0.26 (Haiku 4.5) §3 (Assump- tion 1), §4, App. A.1, Tables 1, 4, 5 SkillsBench [15] (v4) No skills vs. curated per-task bundle; self-generated skills on three configurations 87 tasks, 18 model–harness configurations Three trials per cell, fresh container per run; task-macro pass rate and normalized gain; no budget constraint §3–4, §5.1, Table 2, Fig. 5, App. D.6, App. N 24 Table 5 (continued) Study (version) Arms and coupling Population, inclusion, censoring Runs; endpoint; budget Location in paper SkillApt [25] (v1) WITH vs. WITHOUT a candidate skill; task, model, decoding, environment, evaluator shared Frozen SRA-Bench split: 113 confirmatory states, 111 complete after two no-skill timeouts were censored; infrastructure failures censor the pair One WITH and one WITHOUT execution per candidate–state pair; uncertainty by paired bootstrap (resamples, not runs); correctness first, cost as tie-break §3.1–3.3, §5.2, App. I RADEG [26] (v1) Gate predicts non-zero verifier reward from query–bundle features; separate perturbation study compares bundle compositions 72 queries with bundles and one-skill perturbations; 288 rollouts; query-level held-out splits Each pair executed once; non-zero verifier reward; fixed call budgets in gate evaluation §3, §4.1, §4.4, §5, App. A.2, App. B Confidence- gated retrieval [36] (v1) Raw vs. calibrated confidence-to-action mappings on the same fixed paths HotpotQA and MuSiQue; Mistral, GPT, Qwen models; calibration map fitted on a separate split Traces generated once at temperature 0 and replayed without new generation; source-example counts per split in App. A.4, Table 5; accuracy, coverage, retrieval use; fixed budget §3.2, §4.1, §5, §7, App. A.2, A.4 Replay Gap [44] (v1) Model swap vs. same-model control fork at controlled points of live SWE-bench trajectories Six run pairs, each a 30-instance sweep in one swap direction; about 900 rollouts are branch continuations, not independent tasks Temperature 0; action edit distance and resolution; step budgets §3, §4.1–4.5, Tables 1, 3, §6 Failure prevention [34] (v1) Agent vs. agent with critic intervention on the same tasks HotPotQA (100 tasks), GAIA (30), ALFWorld (202); several models; 50-task pilot Two or three seeds, depending on configuration; App. A lists seeds 42, 123, and 456 for the configurations tabulated there; success, recovery and disruption counts; 15-action cap and at most three critic interventions per episode, no matched total token or cost budget reported §3 (setup), §4, Table 3, §7, App. A (Tables 11–18) State-Path menu [22] (v1) Learned menu vs. baseline menu constructors; agent unchanged ToolBench online tasks; other evaluators, executors, and libraries for transfer One deterministic run per task for the main result (one executor); 8-call budget; online success §4.1–4.2, App. A.2 25 Table 5 (continued) Study (version) Arms and coupling Population, inclusion, censoring Runs; endpoint; budget Location in paper Capability Pages [35] (v1) Skill cards with vs. without negative-boundary text; same executor in both arms SRA-Bench (26,262 skills, 5,400 questions, six datasets); four executors Five solving rounds per executor–dataset cell, same sampling protocol in both arms; end-to-end task success; no budget constraint §3.4, §5.3, Table 5 TabAgent [11] (v1) LLM shortlist head vs. trained classifier; rest of the agent unchanged AppWorldMethods with stochastic behavior run five times per task with distinct seeds; runs of deterministic methods not separately reported; task success, latency, cost §4, §5, App. A References [1] Schick T, Dwivedi-Yu J, Dessì R, et al (2023) Toolformer: language models can teach themselves to use tools. In: Advances in Neural Information Processing Systems 36 [2] Yao S, Zhao J, Yu D, et al (2023) ReAct: synergizing reasoning and acting in language models. In: International Conference on Learning Representations (ICLR) [3] Qin Y, Liang S, Ye Y, et al (2024) ToolLLM: facilitating large language models to master 16000+ real-world APIs. In: International Conference on Learning Representations (ICLR) [4] Patil SG, Zhang T, Wang X, Gonzalez JE (2024) Gorilla: large language model connected with massive APIs. In: Advances in Neural Information Processing Systems 37, p 126544–126565. https://doi.or g/10.52202/079017-4020 [5] Shi Z, Wang Y, Yan L, et al (2025) Retrieval models aren’t tool-savvy: benchmarking tool retrieval for large language models. In: Findings of the Association for Computational Linguistics: ACL 2025, p 24497–24524. https://doi.org/10.18653/v1/2025.findings-acl.1258 [6] Lumer E, Gulati A, Subbiah VK, et al (2025) ScaleMCP: dynamic and auto-synchronizing model context protocol tools for LLM agents. arXiv preprint arXiv:2505.06416 [7] Gaurav N, Akarsh A, Ranjan A, Bajaj M (2025) Dynamic ReAct: scalable tool selection for large-scale MCP environments. arXiv preprint arXiv:2509.20386 [8] Jiang D, Lin E, Cui L, et al (2026) SkillReason: reasoning-enhanced agent skill retrieval for implicit user requests. arXiv preprint arXiv:2608.08640 [9] Murtaza S, Nie Y, Soni U, et al (2026) When synthetic data hurts: on catastrophic forgetting in skill retrieval for LLM agents. In: Proceedings of EMNLP 2026: Industry Track (accepted). arXiv preprint arXiv:2609.10750 26 [10] Qu C, Dai S, Wei X, et al (2025) Tool learning with large language models: a survey. Front Comput Sci 19(8):198343. https://doi.org/10.1007/s11704-024-40678-2 [11] Levy I, Shapira E, Goldshtein Y, et al (2026) TabAgent: a framework for replacing agentic generative components with tabular-textual classifiers. arXiv preprint arXiv:2602.16429 [12] Cho S, Park C (2026) Skill following: evaluating actual skill use in retrieval-enabled LLM agents. In: Findings of EMNLP 2026 (accepted). arXiv preprint arXiv:2609.00549 [13] Hajimiri S, Aminbeidokhti M, Dolz J, et al (2026) Are online skill and memory modules always worth their tokens? A budget-constrained study of web agents. In: Proceedings of EMNLP 2026 (accepted). arXiv preprint arXiv:2606.15017 [14] Tank D, Nama B (2026) The regression tax: decomposing why skills help and hurt LLM agents. arXiv preprint arXiv:2607.22520 [15] Li X, Liu Y, Chen W, et al (2026) SkillsBench: benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670 [16] Yehudai A, Eden L, Li A, et al (2026) A survey on evaluation of LLM-based agents. In: Findings of the Association for Computational Linguistics: ACL 2026. https://aclanthology.org/2026.findings -acl.1330/ [17] Mohammadi M, Li Y, Lo J, Yip W (2025) Evaluation and benchmarking of LLM agents: a survey. In: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p 6129–6139. https://doi.org/10.1145/3711896.3736570 [18] Nageshwaran V, Ezekiel S, Tran T, Lakshmi Narasimhan V (2026) Large language model agent eval- uation and benchmarking: a systematic survey, meta-taxonomy, and critical research roadmap. Artif Intell Rev. https://doi.org/10.1007/s10462-026-11678-4 [19] Kehkashan T, Abdullah M, Al-Shamayleh AS, et al (2026) From benchmarks to deployment: a com- prehensive review of agentic AI evaluation. Artif Intell Rev 59(8). https://doi.org/10.1007/s10462 -026-11571-0 [20] Wang J, Wang Y, Chen M, et al (2026) A survey for LLM agent trajectory analysis: from failure attribution to enhancement. IEEE Trans Softw Eng. https://doi.org/10.1109/TSE.2026.3717765 [21] Huang W-J (2026) What does an LLM-agent leaderboard rank actually compare? arXiv preprint arXiv:2609.07785v1. https://arxiv.org/abs/2609.07785v1 [22] Yan B, Lin W, Wang S (2026) The menu is an execution prior: state-path tool menus for online agents. In: Proceedings of EMNLP 2026 (accepted). arXiv preprint arXiv:2609.09395 [23] Repantis V, Gawde A, Singh H, Blackwell J I (2026) How many tools should an LLM agent see? A chance-corrected answer. arXiv preprint arXiv:2605.24660 [24] Goulart P, Wu L, Wan K, et al (2026) Field-aware agent skill retrieval. arXiv preprint arXiv:2608.02880 [25] Guo S (2026) SkillApt: learning when to activate agent skills from counterfactual evidence. arXiv preprint arXiv:2609.26863 27 [26] He L, Wen J, Gu H, et al (2026) From relevance to execution utility: reward-aware dynamic execution gating for skill-based LLM agents. arXiv preprint arXiv:2608.09168 [27] Wang Z, Vemuri P (2026) When the tool decides: LLM agents defer blindly to graph neural network tools, and stronger backbones defer more. arXiv preprint arXiv:2606.14476 [28] Song H, Wei S (2026) More skills, worse agents? Skill shadowing degrades performance when expanding skill libraries. arXiv preprint arXiv:2605.24050 [29] Rubin DB (1974) Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol 66(5):688–701. https://doi.org/10.1037/h0037350 [30] Imbens GW, Rubin DB (2015) Causal inference for statistics, social, and biomedical sciences: an intro- duction. Cambridge University Press, Cambridge [31] International Council for Harmonisation (2019) ICH E9(R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. Step 4 version, 20 November 2019 [32] Trivedi H, Khot T, Hartmann M, et al (2024) AppWorld: a controllable world of apps and people for benchmarking interactive coding agents. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p 16022–16076. https://doi.org/10.186 53/v1/2024.acl-long.850 [33] Ding L, Guo Z, Lu B, et al (2026) Skill2Query: exploiting skill structure to generate pseudo-queries for agent skill retrieval. arXiv preprint arXiv:2608.16071 [34] Vasudev R, Russak M, Bikel D, Alshikh W (2026) Accurate failure prediction in agents does not imply effective failure prevention. arXiv preprint arXiv:2602.03338 [35] Wang Z, Wen W, Ji Q, Qiao R (2026) Skills know their neighbors: cluster-contrastive capability pages for skill retrieval. arXiv preprint arXiv:2608.04482 [36] Chhikara P (2026) Evaluating confidence-gated retrieval with matched trajectory replay. arXiv preprint arXiv:2608.26846 [37] Shang L, Xu M, Sun Y, et al (2026) When self-evolution backfires: pre-commit gating against skill contamination in LLM agents. arXiv preprint arXiv:2608.05810 [38] Feng Y, Zhang Y, Cheng Y, Qi W (2026) Scores are not decisions: cost-aware stopping for tool acqui- sition in LLM agents. arXiv preprint arXiv:2607.27083 [39] Liu D, Li J (2026) Cost-utility alignment in LLM agent trajectories: profiling, attribution, diagnosis, adaptation, and evaluation. arXiv preprint arXiv:2608.26195 [40] Frangakis CE, Rubin DB (2002) Principal stratification in causal inference. Biometrics 58(1):21–29. https://doi.org/10.1111/j.0006-341X.2002.00021.x [41] Ding P, Lu J (2017) Principal stratification analysis using principal scores. J R Stat Soc Ser B Stat Methodol 79(3):757–777. https://doi.org/10.1111/rssb.12191 28 [42] Lu S, Jiang Z, Ding P (2026) Principal stratification with continuous post-treatment variables: nonpara- metric identification and semiparametric estimation. J R Stat Soc Ser B Stat Methodol 88(1):239–260. https://doi.org/10.1093/jrsssb/qkaf049 [43] Angrist JD, Imbens GW, Rubin DB (1996) Identification of causal effects using instrumental variables. J Am Stat Assoc 91(434):444–455. https://doi.org/10.1080/01621459.1996.10476902 [44] Gonuguntla A (2026) The replay gap: static evaluation of model switching in LLM agents scores the wrong world. In: Conference on Language Modeling (COLM 2026). arXiv preprint arXiv:2608.08239 [45] Li L, Chu W, Langford J, Wang X (2011) Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In: Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, p 297–306. https://doi.org/10.1145/1935826.1935878 [46] Dudík M, Langford J, Li L (2011) Doubly robust policy evaluation and learning. In: Proceedings of the 28th International Conference on Machine Learning, p 1097–1104 [47] Gao L, Wang Y, Peng M, et al (2025) Tool graph retriever: exploring dependency graph-based tool retrieval for large language models. arXiv preprint arXiv:2508.05152 [48] Hong X, Dong P, Yu X, Jiang B (2026) Tools are not islands: set-level tool retrieval for LLM agents via query-conditioned hyperedge prediction. arXiv preprint arXiv:2607.25718 [49] Wei W, Yang T, Basu S, et al (2026) Beyond top-k skill retrieval: diversity-aware skill routing for LLM agents. arXiv preprint arXiv:2609.05824 [50] Wang Z, Wen W, Ji Q, et al (2026) Skill is not document: query-conditioned compatibility for LLM agent skill routing. arXiv preprint arXiv:2606.03565 [51] Li Z, Gao L, Ding X, et al (2026) CaSKG: counterfactual-causal skill graphs for scalable agent skill retrieval. arXiv preprint arXiv:2608.25500 [52] Cho Y (2026) SchemaRouter: field-aware tool routing for efficient heterogeneous agentic RAG. arXiv preprint arXiv:2608.21375 [53] Li Q, Dong X, Zhang J, et al (2026) Risk-aware reranking for agentic tool retrieval. In: Proceedings of CIKM 2026 (accepted). arXiv preprint arXiv:2608.22751 [54] Ding J, Ji H, Liu M, Duan T (2026) Right family, wrong skill: evaluating risk exposure in agent skill retrieval. arXiv preprint arXiv:2606.10388 [55] Liu Y, James J, Wang Y, et al (2026) When tool-backed skill retrieval fails: source-style collapse in executable capability retrieval. arXiv preprint arXiv:2608.16502 [56] Chen Y, Yoon J, Sachan DS, et al (2024) Re-Invoke: tool invocation rewriting for zero-shot tool re- trieval. In: Findings of the Association for Computational Linguistics: EMNLP 2024. arXiv preprint arXiv:2408.01875 [57] Liu S, Yang Y, Xiao H, et al (2026) Imagine before retrieval: prospective skill retrieval for LLM agents. arXiv preprint arXiv:2609.01642 29 [58] Bai T, Wan Z, Zhou P, et al (2026) SkillDAG: self-evolving typed skill graphs for LLM skill selection at scale. arXiv preprint arXiv:2606.03056 [59] Miao Y, Yu Z, Zhao L, et al (2026) SkillLens: adaptive multi-granularity skill reuse for cost-efficient LLM agents. arXiv preprint arXiv:2605.08386 [60] Laskar MTR, Fu XY, Singh G, et al (2026) When better turns do not make better agents: diagnosing the gap between next-turn metrics and workflow success. In: REALM Workshop at EMNLP 2026 (accepted). arXiv preprint arXiv:2609.21187 [61] Li Q, Jiao W, Shao S, et al (2026) SkillGate: training in-policy skill selection in long-horizon agents. arXiv preprint arXiv:2608.18852 [62] Panthi S, Abdelfattah R (2026) Outcome monitors: recovery affordances for silent tool failures. arXiv preprint arXiv:2608.19303 [63] Jimenez CE, Yang J, Wettig A, et al (2024) SWE-bench: can language models resolve real-world GitHub issues? In: International Conference on Learning Representations (ICLR) [64] Li J (2026) Counterfactual tool ranking under utility, cost, and privilege constraints. arXiv preprint arXiv:2609.22819 [65] Wang T, Zou Y, Liu X, et al (2026) Beyond task-only matching: personalized skill routing with coun- terfactual evaluation. arXiv preprint arXiv:2608.28241 [66] Opoku J, Banahene D (2026) ToolChain-CRC: conformal risk control for agentic AI under retrieval and tool-use drift. arXiv preprint arXiv:2606.18467 30

Instant research discovery