Paper deep dive
Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments
Yangjie Xu, Lujun Li, Lama Sleem, Niccolo Gentile, Yewei Song, Yiqun Wang, Siming Ji, Wenbo Wu, Radu State
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/21/2026, 1:20:49 AM
Summary
This paper investigates the effectiveness of Agent Skills for small to medium-sized open-source language models (270M-80B) in resource-constrained industrial settings. It finds that very small models (<4B) struggle with skill selection, while models around 30B-80B benefit substantially. Thinking variants incur high GPU costs with minimal performance gains, highlighting a trade-off between computational cost and agent performance.
Entities (12)
Relation Signals (8)
Agent Skill Framework → evaluates → Small Language Models
confidence 95% · We systematically study the Skill paradigm in resource-constrained industrial settings... for small and medium-sized open source language models
Qwen3-80B-Thinking → hashighcost → GPU Cost
confidence 92% · Qwen3-80B-Thinking exhibits severely uncontrolled resource consumption... VRAM·Time reaches 94.802 GB·min
Qwen3-80B-Instruct → performswellwith → Agent Skill Framework
confidence 92% · Qwen3-80B-Instruct improves from 0.303 to 0.618 compared with Direction Instruction
Gemma-3-270m-it → struggleswith → Skill Selection
confidence 90% · Gemma-3-270m-it appears to largely miss the objective of skill retrieval
Agent Skill Framework → uses → InsurBench
confidence 88% · Across two open-source tasks and a real-world insurance claims classification task... InsurBench
Agent Skill Framework → uses → FiNER
confidence 88% · Across two open-source tasks... FiNER dataset is used
Agent Skill Framework → uses → IMDB
confidence 88% · Across two open-source tasks... IMDB dataset derived from the Large Movie Review Dataset
Agent Skill Instruction (ASI) → outperforms → Full-Skill Instruction (FSI)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agent skills are widely supported by major agentic frameworks and perform well with proprietary models, yet their effectiveness for small and medium-sized open source language models (270 M-80B) remains underexplored. We systematically study the Skill paradigm in resource-constrained industrial settings, where reliance on proprietary APIs is impractical due to data security and budget constraints. Across two open-source tasks and a real-world insurance claims classification task, we find that very small models struggle with reliable skill selection, while models around 30B-80B benefit substantially. Thinking variants do not show major levels of improvement from skills, also considering GPU usage increases due to overthinking. These findings reveal a trade-off between GPU cost and agent performance, and provide actionable insights for effective Skill configuration and SLM deployment in real world settings.
Tags
Links
- Source: https://arxiv.org/abs/2602.16653v3
- Canonical: https://arxiv.org/abs/2602.16653v3
Trouble viewing inline? Open PDF directly →
Full Text
45,282 characters extracted from source content.
Expand or collapse full text
Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments Yangjie Xu ♠1 , Lujun Li ♠1 , Lama Sleem 1 , Niccolo’ Gentile 2 , Yewei Song 1 , Yiqun Wang 1 , Siming Ji 3 , Wenbo Wu 4 , Radu State 1 , 1 University of Luxembourg, 2 Foyer S.A., 3 Princeton University, 4 Université Paris-Saclay Abstract Agent skills are widely supported by major agentic frameworks and perform well with proprietary models, yet their effectiveness for small and medium-sized open source language models (270M–80B) remains underexplored. We systematically study the Skill paradigm in resource-constrained industrial settings, where reliance on proprietary APIs is impractical due to data security and budget constraints. Across two open-source tasks and a real-world insur- ance claims classification task, we find that very small models struggle with reliable skill selection, while models around30B–80Bben- efit substantially. Thinking variants do not show major levels of improvement from skills, also considering GPU usage increases due to overthinking. These findings reveal a trade- off between GPU cost and agent performance, and provide actionable insights for effective Skill configuration and SLM deployment in real world settings. 1 Introduction Agent skills, initially introduced by a leading code- oriented unicorn, are widely regarded as an ef- fective framework for agent-centric context engi- neering. This approach can be viewed as a well- designed “static cheat-sheet” (Suzgun et al., 2025), enabling Large Language Models (LLMs) to fo- cus on salient information and instructions through progressive context management. This approach substantially optimizes both the length and qual- ity of context windows while improving LLMs’ use of tools, documents, and external knowledge resources. Unlike Retrieval Augmented Genera- tion (RAG) systems (Lewis et al., 2020), which rigidly encode textual or visual data into vector- space databases, the Skills design directly lever- ages the In-Context Learning (ICL) capabilities of LLMs (Dong et al., 2024a; Brown et al., 2020a) and emergent reasoning to dynamically select the most pertinent information, instructions, and contextual knowledge. However, the capacity to leverage reasoning ca- pabilities is heavily dependent on the inherent per- formance of huge foundation models, such as GPT or Claude, and are primarily tailored to models specifically optimized for code-related tasks(Zhang et al., 2024a). Moreover, concerns regarding data security have been raised (Yan et al., 2025), along with the substantial computational and financial costs associated with invoking proprietary APIs. Finally, it remains largely unclear whether Agent Skills can deliver comparable benefits when de- ployed on open-source models, especially Small Language Models (SLMs). Consequently, this pa- per systematically investigates the integration of Skills for reports deployment-oriented tasks, hence assessing whether the Skill mechanisms can be ben- eficial also for SLMs. We also present and discuss exploratory experiments examining the effects of different Skill settings. 2 Related Work 2.1 Context Engineering (CE) The concept of CE has already garnered substan- tial research and engineering practices across mul- tiple directions. With the advent of zero-shot or few-shot generalization capabilities in LLMs (Brown et al., 2020b; Dong et al., 2024b), post- training paradigms have gradually been challenged by more convenient, efficient, and cost-effective post-deployment CE approaches for domain adap- tation and agent behavior improvement. More- over, a growing body of research has validated that LLMs exhibit human-like attention limitations and the "Lost in the Middle" / "Context rot" (An- thropic, 2025) phenomenon when confronted with excessively long contexts (Kou et al., 2024; Dai et al., 2024; Du et al., 2025), demonstrating the importance of curated CE practices. Contempo- 1 arXiv:2602.16653v3 [cs.AI] 18 Jun 2026 rary LLMs as such employ various CE designs, in- cluding but not limited to hierarchical multi-agent systems to handle complex tasks(Luo et al., 2025), routing steps directly fed to designated agents (Yue et al., 2025), and sophisticated management ap- proaches to multi-turn dialogue histories. These CE techniques are integrated within heterogeneously long and short term agent memory modules (Zhang et al., 2024b; Salama et al., 2025; Hu et al., 2025) leveraging file systems, vector databases, knowl- edge graphs (Edge et al., 2025), and memory buffers (Xu et al., 2025) to achieve comprehensive enhancements in contextual coherence, personal- ized learning, and complex task decision-making. 2.2 Research in Agent Skills Ye et al. conceptualizes CE as an evolving skill, proposing a two-layer framework termed "Meta CE" to automatically rewrite and optimize skill descriptions. Li demonstrates that, for many rea- soning tasks, constructing a single-agent system equipped with a skill library achieves accuracy comparable to multi-agent systems while reducing token consumption and latency by approximately half. (Li et al., 2026) introduces SkillsBench to systematically evaluate whether endowing agents with “skills” is truly worthwhile, but it focuses primarily on closed-source large models. Numer- ous open-source libraries for Agent Skills, such as DeepAgents and the Agent Skill Collections 1 , have grown rapidly in popularity (Chen et al., 2026), while also sparking discussions on skill safety and permissions Liu et al.. When using Agent Skills, model selection proves crucial for skill routing from SKILL.md, where indeed SLMs (Li et al., 2025a) often exhibit suboptimal success rates and low performance than larger counterpart in choos- ing the right skill (Belcak et al., 2025; Li et al., 2025b). 3 Problem and Experiments 3.1 Research Focus As a reminder, in applying Agent Skills, when con- fronted with a complex task, instead of winging it, the LLM reads a file like SKILL.md from skill hub, which describe exactly what the to do next, how to use specific tools, and where to find extra help. This setup is perfect for “ReAct’s style agent system (Yao et al., 2023) that solves problems by thinking, taking an action, and adjusting based on 1 https://github.com/heilcheng/awesome-agent-skills Request LLM Loops - Skill Selection - Task Planning - Interaction - Reasoning Response Human User - Skill Pre-Selection - Task Definition - Interactive Instruction Skill Content Exposure & Reference A skill can guide an agent to another skill or reference. Skill Description Exposure Guidance Skill Hub - Skill Installation Skill to be good coder . - Skill Organization Skill to do web search . - Interaction with Agents Registry External Env - Persistence Authorization - MCP - Tools Calling - Additional Infos (RAGs) - Referenced Skills Skill Retrieval External Calling Infos Feedback ReAct Figure 1: Agent Framework with Skill the results, as shown in Figure 1. First, the agent searches its skill library and selects the best skill for the task. It then plans, executes tools step by step, and adapts on the basis of tool output or new data. After each step, the agent decides: “Should I con- tinue, gather more information via Model Context Protocol (MCP) connections or other references, switch skill, or finish?” This process prevents con- text explosion and enables more efficient use of context windows, which is the most precious re- source in agentic frameworks (Liu et al., 2026a). As illustrated, the core capability lies in select- ing appropriate skills from a “skill hub”, and sub- sequently performing planning, decision-making, and execution based on the selected skill content. 3.2 Methods In order to properly evaluate the skill-selection per- formance of a model, we need to isolate the con- tribution of the skill framework from confounding factors such as tool availability and MCP latency. To do so, we as such deliberately exclude live tool execution in the main experiments. Section 4.3 extends the evaluation to a ReAct agentic settings for coding agents with real tool calling to validate transferability. To evaluate skill-enabled agents, a temporary skill repository is constructed for each task by sampling 4-5 distractor Skill entries from a publicly collected skill hub and combining them with ground-truth skill. Then, we assess check how three CE strategies affect agent performance and efficiency in complex decision-making. (1) Direct Instruction (DI) uses a minimal prompt to mimic raw user input. (2) Full-Skill Instruction (FSI) provides a fixed context containing the entire tem- porary Skill repository, requiring the model to iden- tify the correct Skill among multiple specifications. (3) Agent Skill Instruction (ASI) loads Skill infor- mation on demand: the model determines whether additional Skill details are needed, retrieves the rel- 2 evant Skill, and generates an answer conditioned on that information. 3.3 Datasets Table 1: Dataset overview for the evaluations: average length (words/items), number of labels, domain/topic, and evaluation set size. DatasetWord / Item# LabelsTopics# Eval IMDB74.052Film reviews300 FiNER50.43139Financial Tags403 InsurBench710.522Insurance Claims200 As shown in Table 1, we use a subset of the IMDB dataset derived from the Large Movie Re- view Dataset v1.0 (Maas et al., 2011) as a sim- ple benchmark specifically designed for very small SLMs, which often struggle even on this relatively easy dataset. Specifically, reviews are filtered to retain only those with a string length between 300 and 500 characters so to control for review length. This dataset is used to evaluate the agent’s ability to perform binary sentiment classification (posi- tive vs. negative) on movie reviews. In addition, the FiNER dataset (Loukas et al., 2022) is used, representing comparatively challenging XBRL tag- ging benchmark. Specifically, FiNER contains 139 tag types and requires strong domain knowl- edge in finance as well as robust logical reason- ing capabilities. In addition to public benchmarks, a proprietary industry dataset, InsurBench, is employed to examine the applicability and perfor- mance for SLMs in real industry settings, which requires an LLM to issue a recommendation based on the full thread: whether to continue engaging with the claim, initiate further actions, or close the ticket and terminate the process. InsurBench is built from authentic insurance-claim email histo- ries, which are typically long, noisy, and highly challenging, with SLMs achieving less than 40% average accuracy without fine-tuning. 3.4 Small Language Models What qualifies as a “small” model remains debat- able. We adopt the definition that the upper bound for SLMs should be determined by the target task (Wang et al., 2024) — in agentic contexts, models are considered large at sizes of Claude Sonnet and Fable, likely extending the 1T parameters. Follow- ing this convention, we treat models smaller than 30B parameters as SLMs throughout this paper, and also explore 80B models as a medium-size compar- ison point. Models of the same scale but different variants are also included in scope. Code variants of LLMs are specialized for programming-centric behaviors such as code generation, completion, and repair, whereas reasoning models are optimized for multi-step deliberation, planning, and verification, and tend to excel at tasks requiring decomposition and derivation. Accordingly, our evaluations are restricted to open-source models spanning a wide range of scales, from 270M to 80B parameters, with one =closed-source model GPT-4o-mini, as shown in Table 2. Table 2: Model inventory for evaluation. ModelSizeVRAM (GB)Release Gpt-4o-mini--07/2024 (OpenAI) Gemma-3-270m-it0.27B107/2025 (Google, 2025b) Gemma-3-4b-it4B1003/2025 (Google, 2025c) Gemma-3-12b-it12B2903/2025 (Google, 2025a) Qwen3-30B-Instruct30B7207/2025 (Qwen Team, 2025a) Qwen3-80B-Instruct80B19209/2025 (Qwen Team, 2025b) Qwen3-80B-Thinking80B19209/2025 (Qwen Team, 2025c) Qwen3-80B-Coder80B19201/2026 (Qwen Team) 3.5 Experimental Settings The experiments are structured to emphasize two core aspects: (i) skill selection: selecting (routing to) the appropriate skill for a given task and (i) the agent’s execution correctness after the selected skill is obtained, thereby assessing the validity of the Agent Skill framework. For evaluation, Cls ACC (CLassification ACCuracy) and Cls F1 (F1 score) are used to quantify classification perfor- mance, together with Skill ACC (skill-selection accuracy) which measures routing quality. To ac- count for practical efficiency in industrial settings, Avg GT (min) is added which is defined as the average processing time per task in minutes. Fi- nally, Avg VRAM Time (GB·min), defined as the average GPU memory, represents instead the time cost per task. This metric design is motivated by common production billing practices based on GPU-hours, under which both wall-clock latency and memory residency translate directly into op- erational cost. Furthermore, given fixed VRAM budgets, memory occupancy can constitute a pri- mary throughput bottleneck: once GPU memory is saturated by a workload, other jobs may be pre- vented from running concurrently, an effect not adequately reflected within conventional compute- centric measures such as FLOPS (floating-point operations per second). 3 Table 3: Main performance on IMDB, FiNER, and InsurBench. gpt-4o-mini was not evaluated on InsurBench due to data privacy and security constraints; the corresponding entries are left blank. Model NameMethod IMDBFiNERInsurBench ClsClsSkillAVGAVG VRAMClsClsSkillAVGAVG VRAMClsClsSkillAVGAVG VRAM ACC.(↑) F1(↑) ACC.(↑) GT (min)(↓) Time (GB·min)(↓) ACC.(↑) F1(↑) ACC.(↑) GT (min)(↓) Time (GB·min)(↓) ACC.(↑) F1(↑) ACC.(↑) GT (min)(↓) Time (GB·min)(↓) Gpt-4o-miniDI0.8500.603---0.4840.298-------- Qwen3-80B-Instruct DI0.3070.311-0.0275.2590.1940.139-0.05410.4250.5250.305-0.0499.379 FSI0.2870.295-0.0265.0220.1960.134-0.05510.5470.5300.335-0.0438.201 ASI0.9500.6340.9970.0224.1950.6480.5090.9830.0275.2420.5300.3860.9500.0285.321 Qwen3-80B-Thinking DI0.1940.218-0.08816.8220.2410.249-0.10920.9480.3000.210-0.20333.893 FSI0.3380.338-0.21240.8110.3270.280-0.18836.0760.3300.202-0.20840.017 ASI0.4920.43991.0000.49494.8020.6650.5620.9730.49294.5030.2300.2180.9450.943181.003 Qwen3-80B-Coder DI0.6030.489-0.0203.8330.3030.205-0.0478.9280.5600.363-0.05811.176 FSI0.5530.469-0.17333.2890.3470.225-0.0499.3810.5150.331-0.05510.555 ASI0.9230.6300.9960.0254.7290.6180.4821.0000.0336.3590.6050.4290.9950.05710.975 Qwen3-30B-Instruct DI0.4870.429-0.0231.6460.0940.100-0.0231.6830.5100.282-0.0513.680 FSI0.4030.378-0.0332.3860.0090.025-0.15711.3130.5200.324-0.0412.923 ASI0.9500.9501.0000.0151.0830.5360.4210.9950.0231.6780.3950.3060.9950.0302.153 Gemma-3-12b-it DI0.6430.514-0.44112.6930.4310.314-0.45713.1470.4800.276-0.0340.964 FSI0.4170.386-0.87425.1780.2880.218-0.67619.4550.4950.318-0.0401.149 ASI0.8970.6190.8970.2457.0690.5010.3810.5010.1494.2980.5550.4910.9900.0250.713 Gemma-3-4b-it DI0.2020.221-0.0370.3540.1090.088-0.0130.1230.5200.297-0.0340.333 FSI0.2400.253-0.0530.5060.0080.008-0.0580.5530.4500.261-0.0410.389 ASI0.0230.0290.0230.0150.1450.1140.1090.1140.0080.0810.4350.2210.8450.0160.150 Gemma-3-270m-it DI0.6030.394-0.0020.0060.0050.002-0.0280.0670.2750.1915-0.0070.016 FSI0.0170.021-0.0060.0150.0000.000-0.0610.1470.1400.127-0.0170.041 ASI0.6670.6270.6670.0600.1440.0170.0110.0170.0510.1230.4200.2230.5700.0210.051 gpt-4o-mini Qwen3-Next-80B-A3B- Instruct Qwen3-Next-80B-A3B- Thinking Qwen3-Coder-Next Qwen3-30B-A3B- Instruct-2507 gemma-3-12b-it gemma-3-4b-it gemma-3-270m-it 0% 20% 40% 60% 80% 100% Error Composition 5% 8% 100%100%100% 21% 15% 16% 11% 77% 92% 77% 84% 88% FINER gpt-4o-mini Qwen3-Next-80B-A3B- Instruct Qwen3-Next-80B-A3B- Thinking Qwen3-Coder-Next Qwen3-30B-A3B- Instruct-2507 gemma-3-12b-it gemma-3-4b-it gemma-3-270m-it 0% 20% 40% 60% 80% 100% Error Composition 7% 4% 100%100%100% 11% 97% 57% 89% 93% 39% 100% IMDB Qwen3-Next-80B-A3B- Instruct Qwen3-Next-80B-A3B- Thinking Qwen3-Coder-Next Qwen3-30B-A3B- Instruct-2507 gemma-3-12b-it gemma-3-4b-it gemma-3-270m-it 0% 20% 40% 60% 80% 100% Error Composition 11% 7% 27% 74% 37% 72% 30% 45% 12% 18% 52% 21% 68% 55% 98% 60% 8% INSURBENCH Skill selection errorParsing errorReasoning error Figure 2: Error composition by model and benchmark. Each bar is normalized over incorrect cases and decomposed into three failure stages: Skill Selection Error (wrong skill chosen), Parsing Error (correct skill selected but output malformed), and Reasoning Error (correct skill and parsing, but incorrect final answer). 4 Results 4.1 Main Performance Skill Returns in SLMs As shown in Table 3, most SLMs exhibit clear performance improvements while maintaining a high skill-selection accuracy. The gains are particularly pronounced for mid- sized models; for example, on FiNER, Qwen3-80B- Instruct improves from 0.303 to 0.618 compared with Direction Instruction. In contrast, smaller models such as Gemma-3-4B-IT and Gemma-3- 270M-IT show more limited improvements using Agent Skill. It is also observed that for simpler tasks (e.g., IMDB), the benefits of Agent Skills are modest. For more challenging benchmarks such as FiNER and InsurBench, however, the results highlight the necessity of Agent Skills, hence high- lighting the importance of CE. InsurBench further strengthens this conclusion, as its closed-source nature reduces the likelihood of dataset contam- ination (i.e., the benchmark itself, or just highly comparable ones, having already been seen during training). Tiny Models Fail at Skill Routing Across these three datasets, each assessment includes 4–6 distracting (irrelevant) skills, which presumably would make skill identification relatively straight- forward for the model. However, we find that ex- tremely small models such as Gemma-3-4B-it and Gemma-3-270M-it still struggle to retrieve the ap- propriate skill. In particular, Gemma-3-270M-it ap- pears to largely miss the objective of skill retrieval, and Gemma-3-4B-it achieves only a 0.78 success rate on InsurBench. These results suggest that, within an Agent Skill framework, models below 4B parameters often lack even the basic capability to identify the correct skill, further compromising the capacity to carry out the subsequent, more com- plex execution steps. Error Analysis As shown in Figure 2, the dom- inant failure mode for SLMs (<12B) is high re- trieval error, indicating that the core bottleneck remains at the task-to-skill alignment stage. Our results as such suggest that optimization efforts should therefore prioritize the retrieval pipeline. For most medium-sized to large models(>=30B), 4 retrieval errors decrease substantially, and the error mass shifts downstream toward reasoning bound- aries and instruction/format compliance. It is also worth noting that Qwen 80B Thinking exhibits a notably high error rate on IMDB/InsurBench, sug- gesting late-stage format breakdowns, which high- lights a key vulnerability of instruction-following from thinking-type models. 0.00.20.40.60.81.0 Normalized Cls ACC. 0.0 0.2 0.4 0.6 0.8 1.0 Normalized VRAM Time (GB·min) IMDB 0.00.20.40.60.81.0 Normalized Cls ACC. 0.0 0.2 0.4 0.6 0.8 1.0 FiNER Qwen3-Next-80B-A3B-Instruct Qwen3-Next-80B-A3B-Thinking Qwen3-Coder-Next Qwen3-30B-A3B-Instruct-2507 gemma-3-12b-it gemma-3-4b-it gemma-3-270m-it 0.00.20.40.60.81.0 Normalized Cls ACC. 0.0 0.2 0.4 0.6 0.8 1.0 InsurBench Figure 3: Normalized average VRAM-time vs. task performance across different model variants on three datasets. Lower-left region indicates superior perfor- mance; both axes are normalized. High Cost, Low Return Qwen3-80B-Thinking exhibits severely uncontrolled resource consump- tion: VRAM·Time reaches 94.802 GB·min on IMDB, 94.503 GB·min on FiNER, and 181.003 GB·min on InsurBench — representing22×,18×, and 34× the cost of Instruct (4.195 / 5.242 / 5.321 GB·min) under the same method, respectively. Yet accuracy does not scale proportionally: InsurBench Cls ACC is only 0.230, the lowest among the three models, and IMDB Cls ACC reaches merely 0.492, far below Instruct’s 0.950 and Coder’s 0.923. This indicates that the reasoning mechanism of the Thinking model generates substantial redundant reasoning tokens while despite solid accuracy in identifying the correct Agent Skills description, finally failing on improving task accuracy while incurring extreme computational waste. This struc- tural conflict between high cost and low return war- rants careful consideration before deployment in such settings. 4.2 Hitting Paradigm? Small Vs Large Prior experiments indicate that tiny SLMs exhibit noticeable performance degradation in model se- lection tasks for Gemma-3-4b-it, even when inter- ference is limited to only 4–6 competing skills. To further investigate this phenomenon, the robustness of SLMs is evaluated under larger skill hubs, reflect- ing more realistic autonomous agent development scenarios that require extensive skill repertoires (e.g., exceeding 50 skills to achieve full project au- tonomy). As shown in Figure 4, tiny models exhibit rapid accuracy decline beyondN = 10–20skills, whereas models exceeding 12B parameters demon- strate exceptional robustness, maintaining high pre- cision even atN = 100skills to choose from. In particular, we observe that the code-specialized variant outperforms its counterparts in skill selec- tion tasks. SLMs struggle to capture hierarchi- cal skill-revealing structures, whereas medium-to- large-scale models reliably handle nested depen- dencies within a single SKILL.md. Even propri- etary models like GPT-4o-mini occasionally falter in interpreting these relationships accurately, as evidenced in several LangChain DeepAgent CLI experiments, where solely Claude-Opus models consistently achieved near-100% success rate in identifying referenced skills within SKILL.md de- scriptions. 0102030405060708090100110 Skill Numbers 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Skill Selection Accuracy Fitted Performance Curves According to Skill Numbers google/gemma-3-270m-it google/gemma-3-4b-it google/gemma-3-12b-it Qwen/Qwen3-30B-A3B-Instruct-2507 Qwen/Qwen3-Next-80B-A3B-Instruct Qwen3-Next-80B-A3B-Thinking Qwen/Qwen3-Coder-Next Figure 4: Fitted decay curves (solid lines) and empirical data points (markers) for skill selection accuracy across skill number N = 5 to 100. 4.3 Post-hoc Exploration Chat History Matter? In Table 4, we evalu- ate the impact of incorporating conversational his- tory on model performance in InsurBench. To avoid context-window overflow, the dialog before each call is truncated, retaining the system prompt and the most recent 3–4 turns. The results in- dicate that history benefits are largest for very small models (e.g. Gemma-3-4b-it and Gemma- 3-270m-it), while for medium models improve only slightly. However, chat history increases VRAM-time usage—for Qwen3-80B-Instruct, the cost increases from 5.321 to 10.035 GB·min per item. Therefore, enabling chat-history processing 5 is mainly recommended for lightweight SLMs in Agent Skills deployments. Model NameMethods Cls ACC.↑ Avg VRAM Time (GB·min)↓ Qwen3-80B-Instruct ASI0.6205.321 ASIH0.53510.035 Qwen3-30B-Instruct ASI0.4502.153 ASIH0.5002.243 Gemma-3-12b-it ASI0.5750.713 ASIH0.5850.7521 Gemma-3-4b-it ASI0.5250.150 ASIH0.6600.152 Gemma-3-270m-it ASI0.4150.051 ASIH0.5250.058 Table 4: Performance of Various SLMs on InsurBench: ASI vs. ASIH (ASI with Chat History) on Qwen3-80B- Instruct. Method Keyword Name Cls ACC.↑ Cls F1↑ Skill ACC.↑ AVG GT (min)↓ Avg VRAM Time (GB·min)↓ ASI Skill0.6200.6010.9150.0285.321 Capability0.5950.5940.9150.0326.147 Expertise0.6100.6080.9300.0326.184 Proficiency0.5800.5780.9250.0315.946 Know-how0.5800.5790.9100.0224.302 FSI Skill0.5300.506-0.0438.201 Capability0.4850.436-0.05510.458 Expertise0.5700.539-0.05610.739 Proficiency0.5100.462-0.05510.486 Know-how0.5350.497-0.0407.459 Table 5: Performance comparison of different skill syn- onyms under ASI and FSI frameworks on Qwen3-80B- Instruct tested on InsurBench. Bold values indicate the best performance within each method group. Replacing "Skill" with "others"? We conduct one final experiment to also investigate whether replacing the keyword "Skill" with its synonyms affects the efficiency and accuracy of agent tasks. As shown in Table 5, four synonyms were tested. Their impact is shown to be minimal on the perfor- mance. Notably, "Expertise" consistently outper- formed "Skill" across metrics, suggesting it as a po- tentially superior alternative. Additionally, "Know- how" demonstrated substantial improvements in GPU memory efficiency with negligible perfor- mance degradation. Agent Skill Calling in the Wild The main ex- periments above focus on assessing the agent in se- lecting the appropriate skills and perform complex reasoning based on them, with the design inten- tionally isolating the effect of skill usage. Indeed, incorporating real-world tool interactions would introduce additional complexity and reduce exper- imental reliability. In this spirit, agents were not allowed to actually execute additional tools in those Qwen3-Next-80B-A3B-InstructQwen3-30B-A3B-Instruct-2507gemma-3-12b-itgemma-3-270m-it 0 2 4 6 8 10 12 Average Elapsed Time (s) 0.59 0.36 1.91 0.20 1.18 0.87 8.23 1.40 1.45 1.27 3.46 1.01 0.0 0.2 0.4 0.6 0.8 1.0 Pass Rate 0.90 0.89 0.93 0.86 0.85 0.88 0.83 0.82 0.82 0.10 0.02 0.00 DIFSIASI_WITH_TOOL Figure 5: Coding agents performance: bars show aver- age runtime, lines show the average pass rate. settings. To further study this aspect, we also eval- uate agents in a ReAct-style environment within a tool calling setting of an integrated Python inter- preter, where the agent is required to choose among five distractor skills, invoke the selected skill, and following its description for calling the tool to get feedback until reaching a final solution. We con- ducted experiments on the Mostly Basic Python Problems (MBPP) dataset, reporting the pass rate and the average execution time in Figure 5. The results are consistent with those observed in the no-tool setting: very small models (e.g., 270M) struggle to benefit from the Agent Skill framework, and similar behavior is observed for 12B models (DI: 0.83, ASI: 0.82), whereas larger models (e.g., 30B and 80B) show clear performance gains under ASI. 5 Conclusion This paper presents the first systematic study of the Agent Skill framework under realistic industrial deployment constraints, evaluating open-source models from 270M to 80B parameters across pub- lic benchmarks and a proprietary insurance-claims dataset. Results show that ASI consistently outper- forms direct-instruction and full-skill baselines for mid-to-large models (≥30B), approaching closed- source GPT-4o-mini performance without propri- etary APIs. A practical deployment threshold emerges at∼12B–30B parameters, below which skill routing breaks down even under minimal in- terference. The 30B to 80B scale is well suited for agent-skill pipelines that has acceptable accu- racy with budget constraints. These findings of- fer actionable guidance on model selection, skill- keyword configuration, and chat-history manage- ment for organizations with resource-constrained, security-sensitive industrial environments. 6 Limitations Our evaluation is limited to a narrow range of task types, primarily classification and tagging, along with an exploratory coding task. Evaluating full agent systems introduces multiple confounding fac- tors (e.g., MCP server latency and tool effective- ness), and thus we adopt a simplified experimental design, focusing on skill selection and skill pow- ered reasoning. In addition, our study covers mod- els ranging from 270M to 80B parameters and does not include a broader set of open-source models. The underlying causes of SLM limitations in sus- tained or recursive reasoning under Progressive Dis- closure remain unclear, as does the observed advan- tage of code-oriented LLMs in terms of accuracy and VRAM efficiency. Furthermore, the optimal structure and representation ofSkill.mdremain open questions. Recent evidence also suggests that automatically generated skills from LLMs may of- fer limited benefits compared to human-designed ones; accordingly, our experiments primarily rely on existing skills curated from sources such as OpenClaw and other open-source repositories. 7 References Anthropic. 2025. Effective context engineering for ai agents. Accessed: February 6, 2026. Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Ce- line Lin, and Pavlo Molchanov. 2025. Small lan- guage models are the future of agentic ai. Preprint, arXiv:2506.02153. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020a.Lan- guage models are few-shot learners.CoRR, abs/2005.14165. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020b.Lan- guage models are few-shot learners.Preprint, arXiv:2005.14165. Tianyi Chen, Yinheng Li, Michael Solodko, Sen Wang, Nan Jiang, Tingyuan Cui, Junheng Hao, Jongwoo Ko, Sara Abdali, Leon Xu, Suzhen Zheng, Hao Fan, Pash- mina Cameron, Justin Wagle, and Kazuhito Koishida. 2026. Cua-skill: Develop skills for computer using agent. Preprint, arXiv:2601.21123. Hui Dai, Dan Pechi, Xinyi Yang, Garvit Banga, and Raghav Mantri. 2024. Deniahl: In-context features influence llm needle-in-a-haystack abilities. Preprint, arXiv:2411.19360. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024a. A survey on in-context learning. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA. Association for Computational Linguistics. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024b. A survey on in-context learning. Preprint, arXiv:2301.00234. Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, and Hao Peng. 2025. Context length alone hurts llm performance de- spite perfect retrieval. Preprint, arXiv:2510.05381. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. From local to global: A graph rag approach to query-focused summarization. Preprint, arXiv:2404.16130. Google. 2025a. google/gemma-3-12b-it.https:// huggingface.co/google/gemma-3-12b-it . Hug- ging Face model card. Accessed: 2026-02-15. Google. 2025b. google/gemma-3-270m-it.https: //huggingface.co/google/gemma-3-270m-it. Hugging Face model card. Accessed: 2026-02-15. Google. 2025c.google/gemma-3-4b-it.https:// huggingface.co/google/gemma-3-4b-it. Hug- ging Face model card. Accessed: 2026-02-15. Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. 2025. HiAgent: Hier- archical working memory management for solving long-horizon agent tasks with large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32779–32798, Vienna, Austria. Association for Computational Linguistics. Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma, and Tianyi Zhang. 2024. Do large language models pay similar attention like human programmers when gen- erating code? Proceedings of the ACM on Software Engineering, 1(FSE):2261–2284. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9459– 9474. Curran Associates, Inc. Lujun Li, Lama Sleem, Niccolo’ Gentile, Geoffrey Nichil, and Radu State. 2025a. Small language mod- els in the real world: Insights from industrial text classification. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lin- guistics (Volume 6: Industry Track), pages 971–982, Vienna, Austria. Association for Computational Lin- guistics. Lujun Li, Lama Sleem, Niccolo’ Gentile, Geoffrey Nichil, and Radu State. 2025b. Exploring the im- pact of temperature on large language models: Hot or cold? Procedia Computer Science, 264:242–251. International Neural Network Society Workshop on Deep Learning Innovations and Applications 2025. Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Ky- oung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiao- jun Wu, Junwei Zhou, Xiaokun Chen, Hanwen Xing, Yubo Li, and 59 others. 2026. Skillsbench: Bench- marking how well agent skills work across diverse tasks. Preprint, arXiv:2602.12670. 8 Xiaoxiao Li. 2026.When single-agent with skills replace multi-agent systems and when they fail. Preprint, arXiv:2601.04748. Jiacheng Liu, Xiaohan Zhao, Xinyi Shang, and Zhiqiang Shen. 2026a.Dive into claude code: The de- sign space of today’s and future ai agent systems. Preprint, arXiv:2604.14228. Yi Liu, Weizhe Wang, Ruitao Feng, Yao Zhang, Guangquan Xu, Gelei Deng, Yuekang Li, and Leo Zhang. 2026b. Agent skills in the wild: An empirical study of security vulnerabilities at scale. Preprint, arXiv:2601.10338. Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion An- droutsopoulos, and Paliouras George. 2022. FiNER: Financial Numeric Entity Recognition for XBRL Tag- ging. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Asso- ciation for Computational Linguistics. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Jun- wei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, and 7 others. 2025. Large language model agent: A survey on methodology, applications and challenges. Preprint, arXiv:2503.21460. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Lin- guistics. OpenAI.Gpt-4o mini model | openai api. https://developers.openai.com/api/docs/ models/gpt-4o-mini. Accessed: 2026-02-15. Qwen Team. Qwen3-coder-next technical report. Tech- nical report. Accessed: 2026-02-03. QwenTeam.2025a.Qwen/qwen3-30b-a3b- instruct-2507.https://huggingface.co/ Qwen/Qwen3-30B-A3B-Instruct-2507. Hugging Face model card. Accessed: 2026-02-15. Qwen Team. 2025b.Qwen/qwen3-next-80b-a3b- instruct.https://huggingface.co/Qwen/ Qwen3-Next-80B-A3B-Instruct. Hugging Face model card. Accessed: 2026-02-15. Qwen Team. 2025c.Qwen/qwen3-next-80b-a3b- thinking.https://huggingface.co/Qwen/ Qwen3-Next-80B-A3B-Thinking. Hugging Face model card. Accessed: 2026-02-15. Rana Salama, Jason Cai, Michelle Yuan, Anna Currey, Monica Sunkara, Yi Zhang, and Yassine Benajiba. 2025. MemInsight: Autonomous memory augmenta- tion for LLM agents. In Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pages 33136–33152, Suzhou, China. As- sociation for Computational Linguistics. Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. 2025. Dynamic cheat- sheet: Test-time learning with adaptive memory. Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Jun- jie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. 2024. A comprehensive survey of small language models in the era of large lan- guage models: Techniques, enhancements, applica- tions, collaboration with llms, and trustworthiness. Preprint, arXiv:2411.03350. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. Preprint, arXiv:2502.12110. Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. 2025. On protecting the data privacy of large language models (llms) and llm agents: A literature review. High-Confidence Computing, 5(2):100300. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. Preprint, arXiv:2210.03629. Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song. 2026. Meta context engineering via agentic skill evolution. Preprint, arXiv:2601.21557. Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. 2025. MasRouter: Learning to route LLMs for multi-agent systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 15549–15572, Vienna, Austria. Association for Computational Linguistics. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024a. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo- level coding challenges. Preprint, arXiv:2401.07339. Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji- Rong Wen. 2024b. A survey on the memory mecha- nism of large language model based agents. Preprint, arXiv:2404.13501. A Progressive Disclosure Difficulty The feasibility of progressive disclosure for small- scale models was also evaluated in several trial, but poor results were obtained. Cross-Skill references within Skill descriptions were often not detected unde SLMs senarios, which prevented the system 9 from triggering intra-Skill calls (e.g., when a Skill description inSKILL.mdreferenced another Skill that needed to be disclosed). Even with GPT-4o- mini, a low hit rate for such references was ob- served. The official LangChain CLI was also tested with its default system-prompt configuration, and higher hit rates were observed only with Claude Sonnet 4.5 or Claude Opus 4.5. Therefore, intra- Skill invocation (i.e., calling one Skill from within another) was excluded from our experiments, since skill selection rates yielded by open-source mod- els were too low for meaningful comparison; this exclusion is also recommended by Anthropic. B Additional Experimental Settings In all experiments, the models’ default decoding configurations (e.g., temperature, top-p, and top-k) were used to avoid confounding effects from addi- tional hyperparameter tuning. All methods were implemented using the LangChain agent frame- work to ensure a consistent agentic workflow across settings. For inference, vLLM was used as the serv- ing engine, and the context length was fixed to 10240 tokens. To manage conversational history under the context-window constraint, a deterministic message-trimming policy was applied. Specifically, the first message (typically the system prompt) was always retained, and only the most recent3 or4messages were kept depending on the parity of the total message count. Core instructions were preserved while the prompt length was bounded, thereby reducing truncation risk and stabilizing inference cost in longer interactions. For formatting and metadata extraction from SKILL.md, the publicly available codebase is primarily used. 2 C Prompts C.1 Direct Instruction Prompts This is the Direct Instruction prompt setting on the IMDB dataset, where the task is to classify a given review as either positive or negative. Since the task is relatively simple, introducing the skill- based mechanism does not yield a significant per- formance improvement. 2 https://github.com/agentskills/agentskills/ tree/main/skills-ref Direct Instruction Of IMDB: Task: Classify the following movie review as positive or negative sentiment. Review: <<<Review Content>>> This is the Direct Instruction prompt setting for the FiNER dataset. The objective is to identify and classify XBRL tags. As shown, we provide not only the sentence but also the target numerical value appearing in that sentence. Because the139 candidate tags correspond to specialized financial terminology, this task demands **strong** logical reasoning as well as substantial domain knowledge in finance. Direct Instruction Of FiNER: Given a sentence from financial documents, a target numeric entity from that sentence, and a list of candidate XBRL tags, choose the single best-matching XBRL tag. Tag List (candidates): [InterestExpense, ....] Inputs: Sentence:<<<Sentence Content>>> Target entity: <<<Numeric Entity>>> This is the Direct Instruction prompt setting for InsurBench. The task requires the agent to identify the key email(s) within a very long email thread and to make decisions on behalf of an insurance company. This setting evaluates the agent model’s ability to localize salient information and to per- form complex decision-making under long-context conditions. Direct Instruction Of InsurBench: Task: Given the full email thread in Email History, decide whether the insurance company must take action to reply. Email History: <<<Email History>>> C.2 Skill Selection Prompts In this setting, we configure the model at initial- ization via the system prompt to prioritize skill selection rather than answering the user query di- 10 rectly. This system prompt is adapted from the Agent Skill system-agent implementation in the LangChain DeepAgent CLI, and we further refine it to better suit our use case. Skill Selection System Prompt (Part 1): In order to complete the objective that the user asks of you, you have access to a number of skills. ## Skills System **Available Skills:** Skill Context ## Step-by-Step Process **ALWAYS follow these exact steps:** Skill Selection System Prompt (Part 2): 1. **Think step-by-step** about the user’s request: - What is the main task? - Which skill domain does it match? 2. **Skill matching rules:** - **Multiple skills**: List names separated by commas (e.g., "langgraph-docs, sales- analytics") - **No skill matches**: Use empty list ‘[]‘ 3. **Generate response based on skills found:** | Skills Found | Message Content | |—–|—–| | 1+ skills | "Yes I need to read the skill information first because ..." | | No skills | "I didn’t find the right skill." | Skill Selection System Prompt (Part 3): **Final Output Format (Strict JSON)** Your final response must **always** follow this JSON structure: "Message": "Your complete response to the user query goes here.", "Skills":["List of skills selected to complete the request,e.g.,[’sales- analytics’,’sentiment-analytics’]"] ## Examples Skill Selection System Prompt (Part 4): ## Examples **User:** "How do I use LangGraph StateGraph?" **Skills:** ["langgraph-docs"] **JSON:** ‘"Message": "es I need to read the skill information first because I need details on StateGraph from LangGraph docs.", "Skills": ["langgraph-docs"]‘ **User:** "Write a poem about cats" **Skills:** [] **JSON:** ‘"Message": "I didn’t find the right skill.", "Skills": []‘ **User:** "Write a report about LangGraph usage in sales." **Skills:**["langgraph-docs","sales- analytics"] **JSON:** ‘"Message": "Yes I need to read the skill information first because it in- volves LangGraph documentation and sales analytics.", "Skills": ["langgraph-docs", "sales-analytics"]‘ 11 Skill Execution System Prompt (Part 1): In order to complete the objective that the user asks of you, you have access to a number of skills description. ## Skills System You have access to a skills library that provides specialized capabilities and domain knowledge. **How to Use Skills:** 1. **Read the skill’s full instructions**: 2.**Follow the skill’s instructions**: contains step-by-step workflows, best practices, and examples **When to Use Skills:** - User’s request matches a skill’s domain (e.g., "research X" -> web-research skill) - A skill provides proven patterns for complex tasks Skill Execution System Prompt (Part 2): **Example Workflow:** User: "Can you research the latest develop- ments in quantum computing?" 1. Check available skills description 2. Read the skill 3. Follow the skill’s research workflow (search -> organize -> synthesize) 4. Make the final decision. **Skill Information Collected** Skill Context Remember: Skills make you more capable and consistent. When in doubt, check if a skill exists for the task! ## Output instructions **Final Output Format (Strict JSON)** "Message": Your message here. 12