Paper deep dive
SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications
Zhengxi Li, Junyi Zhou, Qiyuan Zhang, Chuhan Wu, Fuyuan Lyu, Le Tian, Xiao Zhou, Chen Ma, Yufei Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present SkillForgeBench, which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, SkillForgeBench reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 ($40.9\%$ strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates ($70.0\%$ to $85.7\%$ vs. $12.6\%$ to $40.9\%$). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., $\Delta 19.2\%$ for Opus 4.8 vs. $\Delta 5.8\%$ for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under $3\%$ for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability.
Tags
Links
- Source: https://arxiv.org/abs/2610.14918v1
- Canonical: https://arxiv.org/abs/2610.14918v1
Trouble viewing inline? Open PDF directly β
Full Text
118,290 characters extracted from source content.
Expand or collapse full text
2026-10-10 SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications Zhengxi Li 1,* , Junyi Zhou 2,3,* , Qiyuan Zhang 2,3 , Chuhan Wu 3 , Fuyuan Lyu 4 , Le Tian 3 , Xiao Zhou 3 , Chen Ma 2 , Yufei Wang 3,β 1 MBZUAI 2 City University of Hong Kong 3 WeChat AI, Tencent 4 McGill β Mila & Quebec AI Institute * Equal contribution β Corresponding author Abstract Evaluating an agentβs ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present SkillForgeBench, which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, SkillForgeBench reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 (40.9% strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates (70.0% to 85.7% vs. 12.6% to 40.9%). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., β19.2% for Opus 4.8 vs. β5.8% for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under 3% for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability. 1 Introduction As large language model (LLM) agents take on increasingly complex professional work across diverse do- mains (Merrill et al., 2026; Ye et al., 2026), establishing rigorous benchmarks to evaluate their reliability across varied workflows has become essential. In practical deployments, a reliable agent must strictly satisfy all operational constraints (e.g., procedural steps and formatting specifications) across diverse scenarios, rather than produce superficially plausible deliverables. Every unfulfilled constraint forces human intervention, invalidating the practical utility of autonomous agents. However, existing agentic benchmarks struggle to evaluate reliability across diverse workflows due to a fundamental dilemma in task curation. Expert-curated benchmarks pair complex tasks with strict verification scripts, offering high task fidelity, yet they remain strictly bottlenecked by manual authoring costs and confined to narrow domains (Yang et al., 2026; Wu et al., 2026). Zhengxi Li <zhengxi.li@mbzuai.ac.ae>, Junyi Zhou <junyizhou8-c@my.cityu.edu.hk>, Yufei Wang <garyyfwang@tencent.com> 1 arXiv:2610.14918v1 [cs.AI] 10 Oct 2026 Conversely, crowdsourced benchmarks achieve domain diversity, but their evaluation signals are frequently decoupled from workspace contexts, ultimately yielding shallow tasks that fail to probe multi-dimensional reliability (Hua et al., 2026; Han et al., 2026). To bridge this gap, we leverage agent skills, the native structural primitives of agent ecosystems, as execution- grounded blueprints for benchmark design. Authored by human experts, skills encode rich procedural workflows, domain-specific knowledge, and professional conventions that naturally reflect user demands across diverse domains. Building on this insight, we present SkillForgeBench, an agentic evaluation benchmark built through a three-stage skill-centric design architecture that directly fulfills these essential properties: First, during Resource Preparation (Β§3.1), we curate a domain-balanced skill set and deploy web-enhanced retrieval to harvest artifacts and user queries, establishing broad workflow coverage. Second, during Skill-driven Task Synthesis (Β§3.2), skills guide the functional partitioning of artifacts into editable targets, reference evidence, and read-only contexts to construct concrete task scenarios. We then leverage retrieved user queries to compose naturalistic task instructions and package each scenario into a sandboxed environment, achieving deep procedural instantiation. Third, during Holistic Evaluation Design (Β§3.3), we evaluate requirement satisfaction across both final deliverables via completion rubrics and execution trajectories via process rubrics. To establish truly rigorous reliability, we prioritize strict scoring that requires full compliance with constraints, complemented by loose scoring for fine-grained capability diagnosis, while conducting all evaluations in both skill-conditioned and unconditioned settings. In total, SkillForgeBench currently comprises 842 rigorously verified tasks across diverse domains, uniquely combining real-world demand, authentic environments, fine-grained rubrics, execution evaluation, and explicit skill use (Table 1), and establishing a challenging testbed for evaluating agent capabilities under realistic constraint adherence. Evaluating eight mainstream agents on SkillForgeBench reveals a steep capability hierarchy across leading proprietary and open-weight models. Frontier models, led by Opus 4.8 (40.9% strict completion), substantially outpace low-tier models such as Qwen3.5-397B , which drops to near 12.6%. Crucially, while loose scoring yields deceptively high rubric pass rates (70.0% to 85.7%), strict scoring requiring total constraint compliance exposes severe capability bottlenecks across all model tiers. Beyond leaderboard rankings, our diagnostic analyses yield three core insights: First, strict evaluation defines the true boundary of agent reliability, while loose scoring mask hidden process failures and create an illusion of competence. Second, skill conditioning improves individual rubric pass rates similarly across models (β5.1% to 7.4%), and frontier models convert skills into far larger strict completion gains (β19.2% for Opus 4.8 vs. β5.8% for Haiku 4.5). Third, performance bottlenecks stem from process discipline rather than deliverable production: agents satisfy process rubrics far less frequently than completion rubrics (i.e., Haiku 4.5 meeting all process constraints on fewer than 3% of tasks). In summary, these findings confirm that real-world agent reliability hinges on strict constraint adherence across the entire workflow execution. SkillForgeBench thus provides a rigorous measurement standard to evaluate this essential capability. 2 Related Work 2.1 Agentic Benchmarks Agent evaluation has evolved alongside expanding LLM agentsβ capabilities. Early benchmarks target specialized, single-domain workflows, such as code repair (Jimenez et al., 2024), web navigation (Zhou et al., 2024), multi-app API coordination (Trivedi et al., 2024), and policy-adherent customer service (Yao et al., 2025). Recent benchmarks target autonomous agents operating in general-purpose environments. OSWorld (Xie et al., 2024) provisions full desktop VMs; Terminal-Bench (Merrill et al., 2026) evaluates system administration in Docker containers; ClawBench (Zhang et al., 2026) and Claw-Eval (Ye et al., 2026) define rubric-scored tasks across productivity domains for Claw-style agents. PinchBench (Kilo AI, 2026) combines automated checks with LLM-judge rubrics. These benchmarks span more diverse scenarios but remain manually curated and static in scale. To bypass manual authoring constraints, recent works explore automated task harvesting and synthesis. RealClawBench (Lv et al., 2026) reconstructs environments from production agent sessions; WildClawBench (Ding et al., 2026) runs tasks inside real CLI harnesses; LiveClawBench (Long et al., 2026) continuously 2 Table 1Comparison of SkillForgeBench with representative agent benchmarks. Real Demand and Authentic Env. measure task and environment realism; Fine Rubrics and Process Eval. characterize evaluation granularity; and Skill Use indicates whether skills are explicitly evaluated. A green check, orange dot, and gray cross denote full, partial, and no support. BenchmarkTask SourceScale Real Demand Authentic Env. Fine Rubrics Process Eval. Skill Use WebArena (Zhou et al., 2024)Authored web intents812 β’ β’ β AppWorld (Trivedi et al., 2024)Authored app workflows 750 β’ β’ β Terminal-Bench 2.0 (Merrill et al., 2026)Expert workflows89 β’ ββ ClawBench (Zhang et al., 2026)Everyday assistant tasks 153 β’ β β’ ββ LiveClawBench (Long et al., 2026)Real usage cases134 β’ β’ ββ RealClawBench (Lv et al., 2026)Production sessions281ββ Claw-Eval (Ye et al., 2026)Human-curated tasks300 β’ β’ ββ SkillsBench (Li et al., 2026a)Curated skill tasks86β β’ ββ Agentic Skills at Scale (Shaposhnikov et al., 2026) LLM-synthesized tasks 1,000ββ SkillForgeBenchSkills + real artifacts 842β refreshes tasks against live services to prevent contamination. Automated synthesis pipelines address scale and cost simultaneously. ClawEnvKit (Li et al., 2026b) generates environments from natural-language descriptions at orders-of-magnitude lower cost. ClawForge (Lai et al., 2026) templates state-conflict scenarios with grounded slots. However, current synthesis pipelines lack a principled framework to reconcile broad domain diversity with deep procedural complexity, often generating tasks that lack structured requirement grounding and rigorous evaluation rubrics. Scaling evaluation reliability remains their central limitation. Table 1 summarizes these comparisons. 2.2 Synthesis from Skills Synthetic data generation has emerged as a key paradigm for scaling agent training and evaluation. Early approaches synthesize function-calling instructions, and interaction traces over real or mocked API endpoints, validating trajectory executability programmatically (Liu et al., 2024; Wang et al., 2026). Subsequent methods ground trajectory synthesis in live web or desktop environments to produce realistic execution traces (Xu et al., 2025; Wang et al., 2024), while environment synthesis techniques automatically generate or expand sandbox setups to diversify execution contexts (Zhu et al., 2026; Dong et al., 2026). Notably, Terminal-World (Cheng et al., 2026) also leverages skills for data synthesis, but treats them merely as topic seeds for task generation. This approach discards the rich procedural step definitions and execution constraints embedded within agent skills. In contrast, SkillForgeBench leverages skills as unifying blueprints across the task lifecycle, guaranteeing domain diversity, procedural complexity, and structurally aligned evaluation rubrics. 3 SkillForgeBench Construction SkillForgeBench is designed to evaluate an agentβs ability to holistically satisfy diverse and procedurally complex constraints in long-horizon workflows. This requires benchmark to fulfill three properties simulta- neously: broad workflow coverage, deep procedural instantiation, and fine-grained yet holistic evaluation. We achieve these through three components (Figure 1): 1. Resource Preparation (Β§3.1): We curate a domain-balanced, procedurally rich skill set and retrieve authentic open-web materials to establish broad workflow coverage; 2. Skill-driven Task Synthesis (Β§3.2): We leverage skills to partition artifact roles, formulate scenario tasks, and compose naturalistic instructions, deeply instantiating procedural logic into executable environments; 3. Holistic Evaluation Design (Β§3.3): We derive calibrated completion and process rubrics through contrastive rollouts and human expert audits, evaluating agents across fine-grained constraints and dual scoring regimes. 3 a Resource Preparation b Skill-Driven Task Synthesis c Holistic Evaluation Design Coverage-oriented skill curation Filtering Mathematics Code generation Data analysis Scientific writing Citation verification Check citations Cross-reference Find discrepancies Suggest fixes Authentic resource search Resource Requirements .bib @article... author = ... title = ... .tex PDF Metadata title: ... authors: ... venue: ... year: ... DOI: ... Retrieved artifacts User query search Verify and repair academic citations User intent Selected skill Naturalize wording (1) Skill-guided artifact role assignment Editable targets Read-only context Synthesis-only evidence .bib @article... author = ... title = ... .texPDF Metadata title: ... authors: ... venue: ... year: ... DOI: ... Excluded from workspace (3) Instruction composition (4) Packaging & Validation Selected senario .bib User query Instruction Check and repair the bibliography. Keep unresolved cases separate. Initial workspace .bib.texPDF Install dependencies Setup Executable environment Skill-guided contrastive agent rollouts With skill q Check and repair the bibliography. Keep unresolved cases separate. .bib.texPDF Linux sandbox (isolated environment) Without skill q Check and repair the bibliography. Keep unresolved cases separate. .bib.texPDF Linux sandbox (isolated environment) Trajectories + final workspaces log log .bib log log ? .bib Compare outcomes & processes Completion rubrics Correctness Β· validity Process rubrics Verification Β· constraints Verified benchmark tasks Includes rubrics β’ LaTeX project files β’ Bibliography (.bib) β’ Paper PDFs β’ Metadata Skill ----- ----- ------ ----- Skill ----- ----- ------ ----- (2) Role-driven task formulation Check bibliography .bib Match citations .texPDF Verify and repair citations .bib.texPDF Retrieved Queries How do I fix incorrect citations? Why is this citation shown as [???]? Are all references actually cited? Skill ----- ----- ------ ----- Discard failed setups Skill outside workspace Specify content Preserve details Propose candidates and select High-quality skills Skill pool Human expert calibration Independently review and jointly resolve disagreements Atomic Precise Executable Figure 1Overview of the SkillForgeBench construction pipeline, demonstrated using a citation verification skill as a running example: (a) Skills guide retrieval of real artifacts and user queries; (b) These inputs support task formulation, instruction composition, and environment packaging; and (c) Contrastive rollouts with and without skill access inform completion and process rubrics, followed by human task validation. 3.1 Resource Preparation In this stage, our primary objective is to establish broad workflow coverage. Since skills originate from diverse real-world domains and developer communities, they collectively provide an expansive pool of realistic workflows. Accordingly, we design a two-fold resource preparation strategy: first, we implement a rigorous skill curation pipeline to select a domain-balanced, procedurally complex skill setS=s 1 ,...,s N ; second, we deploy a web-enhanced retrieval mechanism to harvest real-world artifacts and user queries that ground each skill workflow with authentic materials. Coverage-oriented skill curation. To establish diverse and complex workflows, we curate skills through a three-stage process. (1) Broad skill collection and filtering. We collect over 50,000 candidate skills from major public Agent Skill ecosystems and repositories, includingSkillsMP,SkillHub, and public skill repositories maintained by Anthropic, OpenAI, and other developer communities. A rule-based filter removes malformed entries, duplicates, and skills that rely on GUI interaction, private services, or non-local execution. (2) Procedural quality assessment and domain tagging. We employ a frontier LLM to evaluate each remaining skill across workflow complexity, required domain knowledge, and resource richness, averaging these dimension scores into a unified quality scoreQ(s). Concurrently, the model assigns a domain tag to each skill to facilitate balance. (3) Domain-balanced selection. We rank skills byQ(s) within each domain and select them uniformly across domains until reaching the total budgetN, yielding the final skill setS. The skills behind the final 842 tasks span five domainsβTechnology, Business, Creative, Harness Behavior, and Knowledge. Table 5 reports the full distribution. Authentic resource and demand search. Beyond skill curation, we further leverage each skills β Sas a focal seed to retrieve diverse, authentic open-web resources across two parallel search streams. Specifically, we analyze each skill specification to extract its core resource requirementsR s =r 1 ,...,r m and user intent summaryG s , where eachr i defines the expected content of one required material, andG s is a concise sentence describing what a user aims to accomplish. To harvest realistic grounding materials, we query the open web usingR s to collect diverse retrieved artifactsA s = Ξ¨ A (R s ) matching each structural requirement. 4 Concurrently, we search 20 predefined public forums 1 usingG s to gather real-world community discussions, producing matching user queriesD s = Ξ¨ D (G s ). These queries serve as naturalistic demonstrations for instruction composition, providing realistic vocabulary and authentic background knowledge. By grounding each skill in authentic web artifacts and user expressions, this retrieval process significantly expands the diversity and realistic depth of our benchmark input resources. 3.2 Skill-driven Task Synthesis With curated skills and authentic materials in hand, our next step is to achieve deep procedural instantia- tionβfaithfully injecting skillsβ complex procedural logic into executable benchmark tasks. To accomplish this, we design a skill-driven synthesis framework that transforms raw materials into skill-grounded scenarios, fuses procedural constraints into naturalistic user instructions, and packages them into executable environments. Skill-guided task formulation. While retrieval yields relevant artifacts, directly formulating tasks without constraints risks shallow edits or evidence leakage. Since skills operate on specific input resources and produce outcomes, we partition the retrieved artifactsA s into three functional roles to constrain task formulation: (1) editable targetsA edit s corresponding to the required inputs; (2) synthesis-only evidence A syn s providing ground-truth reference for expected outcomes; and (3) read-only contextA ctx s embedding realistic background constraints. Using these roles, an agent groups related artifacts into candidate scenarios C s =c 1 ,...,c K , where eachc i defines a coherent artifact collection by associating editable targets with evidence for expected outcomes and relevant contextual resources. The agent then evaluates these candidates for skill alignment, procedural depth, and executability to select the optimal task scenarioT s . For instance, in a citation verification skill, a flawed BibTeX file serves asA edit s , a metadata dump of verified citations as A syn s , and paper PDFs asA ctx s βtogether instantiating a task scenarioT s where the agent audits and repairs the bibliography against ground-truth metadata. Naturalistic instruction composition. Given the task scenarioT s and skills, we leverage retrieved user queriesD s as in-context demonstrations to compose a naturalistic task instructionq s . Notably, this retrieval-guided grounding preventsq s from collapsing into rigid synthetic templates inspired by Natural Questions (Kwiatkowski et al., 2019). Specifically,T s defines what should be accomplished,D s shapes the natural phrasing and vocabulary, and s ensures the essential details from the skill to produce q s : q s = Ξ¦(T s ,D s ,s).(1) where the agent Ξ¦ rewrites the structured task T s into a natural user query conditioned on D s . Packaging and execution validation. To ensure reproducible execution, we construct an initial workspace W 0 s containingA edit s andA ctx s , while strictly withholding ground-truthA syn s to prevent answer leakage. We then packageW 0 s into a sandboxed Linux container and automatically resolve its dependencies to form the executable environmentE s . Any task requiring persistent external services or failing sandbox build validation is discarded. 3.3 Holistic Evaluation Design With executable tasks in place, our final goal here is to establish a fine-grained yet holistic evaluation that assesses an agentβs capability to satisfy diverse requirements and procedural constraints in long-horizon workflows. In real-world environments, an agent may produce a superficially correct output while violating critical process rules and skipping mandatory safety verifications. Thus, SkillForgeBench evaluates individual requirements in detail while verifying holistic compliance across the entire workflow. 1 The list of 20 forums is shown in Table 4. 5 Completion and process rubric construction. To cover both outcome and execution constraints, we structure evaluation into two fine-grained rubric dimensions: Completion Rubrics (R comp ) evaluate outcome constraints on final workspace, focusing on the structural correctness and content validity of skill-specified deliverables; Process Rubrics (R proc ) evaluate execution constraints along agent trajectories, covering adherence to procedural steps, intermediate verification checks, and safety bounds in the skill workflow. However, directly prompting language models to synthesize rubrics from task descriptions often yields abstract and superficial rules (Zhou et al., 2026), missing critical execution details and specific environment changes. To derive precise and executable rubrics, we combine skill-guided contrastive agent rollouts with human expert calibration. Specifically, a reference agentΟ ref executes task instructionq s twice on identical environments E s : (Ο + s ,W + s ) = Rollout(Ο ref ,q s ,E s ,s),(Ο β s ,W β s ) = Rollout(Ο ref ,q s ,E s ,β ),(2) where (Ο + s ,W + s ) and (Ο β s ,W β s ) denote the resulting trajectories and workspace states generated with skill (s) and without skill (β ), respectively. An evaluation agent then contrasts the trajectory deltas (Ο + s vs. Ο β s ) and state deltas (W + s vs. W β s ) to isolate the procedural constraints introduced by skills, synthesizing draft rubrics forR comp s andR proc s . Finally, two human experts calibrate these draft rubrics through three explicit steps: atomic decomposition splits complex rules into single atomic rubrics, filtering removes superficial formatting noise while preserving core domain requirements, and executable grounding refines vague descriptions into clear, unambiguous natural language rubrics with explicit pass/fail conditions. In practice, the two experts independently review draft rubrics, resolving disagreements jointly. Evaluation Metrics Design. To assess both outcome correctness and execution constraints, Skill- ForgeBench evaluates completion and process rubrics (R comp ,R proc ) under two distinct execution settings: with skill (w/skill), which tests an agentβs capacity to adhere to explicit procedural instructions, and without skill (w/o skill), which probes whether the agent can complete complex workflows without external guidance. This design allows us to distinguish whether successful task execution stems from the modelβs internal knowledge or from external procedural guidance. For a given taskTevaluated against a calibrated rubric setR=r 1 ,r 2 ,...,r k , an evaluation agent inspecting the execution trajectoryΟand final workspace stateWto evaluate each atomic checkr i β0,1. We then compute the Completion Score (S comp ) and Process Score (S proc ) using two scoring regimes: S loose (T ) = 1 k k X i=1 r i , S strict (T ) = k Y i=1 r i .(3) Loose scoring (S loose ) measures the mean pass rate across rubric items to provide fine-grained diagnostic progress, whereas strict scoring (S strict ) enforces an all-or-nothing logical AND to evaluate overall system reliability under real-world conditions. These two regimes capture complementary demands: fine-grained feedback on incremental progress and strict satisfaction of all constraints for reliable end-to-end execution. 4 Experiments In this section, we use LLM-as-a-Judge to execute rubrics and conduct a systematic empirical evaluation of how well mainstream LLM-based agents satisfy the constraints of SkillForgeBench tasks. To demonstrate the evaluation utility and diagnostic power of our benchmark, our experiments are organized into 3 core parts: (1) Main Leaderboard, measuring how well agents satisfy individual constraints and all constraints jointly, together with the impact of skill grounding and execution efficiency; (2) Reliability of Task and Rubric Construction, independently auditing the quality of generated tasks and rubrics and measuring agreement across expert judgments; and (3) Diagnostic Analysis, dissecting results across release dates, model capability tiers, and rubric categories to identify which constraints agents fail to satisfy. Experimental Setup. We evaluate eight mainstream LLMs, including both proprietary and open-weight models, using the same OpenHands agent harness. The evaluated backbone models are Claude Opus 4.8, 6 Table 2Main leaderboard of model performance and execution efficiency. Task-level mean scores (%) are reported under loose and strict scoring for Completion (S comp ), evaluated with and without skill, and Process (S proc ), evaluated with skill. The efficiency columns report the mean total number of interaction steps (# Assistant) and tool calls (# Tool-call) for runs with and without skill. Bold indicates the highest score andunderliningindicates the second-highest score in each performance column. LooseStrictEfficiency ModelS comp S proc S comp S comp S proc S comp # Assistant# Tool-call w/ skillw/ skillw/o skillw/ skillw/ skillw/o skillw/ skillw/o skillw/ skillw/o skill Opus 4.885.787.979.840.927.721.728.624.641.038.5 GLM-5.284.186.776.936.828.720.042.639.658.054.8 Sonnet 4.681.576.075.328.510.215.015.511.738.229.6 HY380.984.273.927.321.614.034.731.844.341.0 Gemini 3.6 Flash79.270.971.818.45.118.9147.044.547.544.6 Kimi K2.678.173.072.220.16.0611.235.632.644.639.1 Qwen3.5-397B71.763.966.613.32.615.3415.714.934.930.4 Haiku 4.570.063.164.312.62.386.7728.223.332.929.0 Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.6 Flash, GLM-5.2, Kimi K2.6, Qwen3.5-397B, and Hunyuan-3. Using a common harness controls for performance differences attributable to the agent framework, enabling a fairer comparison of the underlying models. 4.1 Main Leaderboard & Benchmark Utility Table 2 presents performance and trajectory efficiency across 8 mainstream models evaluated with and without agent skill. SkillForgeBench induces substantially long execution horizons: across models, with-skill runs require 15.5β47.0 total steps and 32.9β58.0 tool calls per task on average. Under the primary metric of strict-scoring completion with skill, Opus 4.8 leads the panel at 40.9%, followed by GLM-5.2 at 36.8%, with Sonnet 4.6 and HY3 tied at roughly 28%. Open-weight and proprietary models interleave throughout the ranking rather than forming disjoint capability tiers. Holistic Constraint Satisfaction Reveals Hidden Capability Gaps. While loose scoring evaluation suggests a deceptively narrow capability gap (Haiku 4.5-70.0% to Opus 4.8-85.7%), strict evaluation isolates holistic constraint satisfaction as the primary factor dividing model tiers, expanding the performance spread dramat- ically (Haiku 4.5-12.6% to Opus 4.8-40.9%). Non-frontier agents routinely succeed on isolated rubrics but suffer compound execution failures when all environmental constraints must hold simultaneously. Thus, the true gap between model tiers lies in satisfying all constraints simultaneously, not in isolation. This requires shifting agent optimization from individual rubric fixes to holistic constraint alignment. Skill Grounding Widens Rather Than Narrows the Gap in Satisfying All Constraints. Comparing performance with and without skill disentangles the contributions of an agentβs internal knowledge from external procedural guidance, testing whether skills act as compensatory scaffolding that uplifts weaker models. While external skills might intuitively seem like compensatory scaffolding, frontier models convert skill guidance into complete constraint satisfaction far more effectively than lower-tier models. As shown in Figure 2, loose scoring reveals uniform gains in satisfying individual constraints across models (β = 5.1% to 7.4%), whereas strict scoring exposes sharply divergent gains. Skill guidance yields dramatically larger improvements for top-tier agents (β = 19.2% for Opus 4.8; 16.8% for GLM 5.2) than for lower-tier agents (β = 8.0% for Qwen 3.5 397B; 5.8% for Haiku 4.5). This implies that external procedural guidance cannot serve as a standalone substitute for internal knowledge. Instead, effectively parsing and executing external guidance itself demands strong internal knowledge, meaning that advancing lower-tier agents requires improving how procedural skills are grounded into execution steps (see Appendix A). 7 Table 3Independent audit of SkillForgeBench task and rubric quality. Values are mean ratings on a 1β5 scale. Human agreement is the percentage of paired ratings that differ by at most one point. Task quality covers 100 tasks; rubric quality covers 771 completion and 840 process criteria. The overall row aggregates all 5,133 dimension-level ratings. Audit Target# Units Expert 1 Expert 2 GPT-5.6-sol Human Agr. (%) Task Quality1004.564.714.2898.0 Completion Rubrics7714.474.564.4396.1 Process Rubrics8404.444.494.1395.4 Overall1,7114.464.534.2795.9 10203040 Strict Ξ 19.2 Ξ 16.8 Ξ 13.5 Ξ 13.3 Ξ 9.5 Ξ 8.9 Ξ 8.0 Ξ 5.8 5060708090 Score (%) Loose Ξ 7.4 Ξ 7.2 Ξ 7.0 Ξ 6.2 Ξ 5.9 Ξ 5.9 Ξ 5.7 Ξ 5.1 Completion w/ skillCompletion w/o skill Figure 2Completion gains from skill. Bars compare Completion scores with and without skill under strict and loose scoring. β =S WS comp β S NS comp , shown to the right of each bar, denotes the Completion-score gain from skill access. Divergence Between Process and Completion Constraints. ComparingS comp againstS proc reveals a fundamental dis- parity in how models satisfy the two kinds of constraints. UnderS loose , top-performing agents achieveS proc that sys- tematically outpaceS comp (e.g., Opus 4.8: 87.9% vs. 85.7%), whereas non-frontier models exhibit the inverse pattern (e.g., Gemini 3.6 Flash: 70.9% vs. 79.2%; Haiku 4.5: 63.1% vs. 70.0%). This divergence indicates that these agents frequently secure deliverables while leaving execution con- straints unsatisfied, so completion-only checks overestimate how well they satisfy all constraints. UnderS strict , satisfying all execution constraints further decouples from completion rank: GLM-5.2 leadsS proc (28.7%) despite ranking second inS comp (36.8%). Ultimately, this establishes that satisfying outcome constraints and satisfying execution constraints are distinct dimensions of agent capability. Non-Monotonic Efficiency in Long-Horizon Execu- tion. Interactive agent execution exhibits a non-monotonic relationship between trajectory length and task success. For instance, Opus 4.8 reaches top strict completion (40.9%) in 28.6 steps, whereas GLM-5.2 consumes 49% more steps (42.6) for lower performance (36.8%). Similarly, Gemini 3.6 Flash uses 3.03Γthe step budget of Sonnet 4.6 (47.0 vs. 15.5) yet trails by 10.1%, resulting in a 3.7Γspread in strict completion per step. This demonstrates that extra steps do not reliably translate into satisfying more constraints. Consequently, efficiency is not merely a matter of resource cost; advancing agent capabilities requires executing each step in compliance with the constraints rather than extending trajectory length. 4.2 Reliability of Task and Rubric Construction To assess the reliability of our automated construction, two expert annotators independently review the same stratified sample, evaluating task quality in terms of realism, taskβenvironment alignment, and solvability; completion rubrics in terms of relevance, observability, and non-redundancy; and process rubrics in terms of workflow coverage, observability, and implementation independence. As an external robustness check, GPT-5.6 Sol independently applies the same evaluation protocol without replacing or adjudicating the expert judgments. Independent audits confirm construction reliability. As shown in Table 3, both experts consistently rate the generated tasks and rubrics highly, with overall mean ratings of 4.46 and 4.53 and strong agreement: 95.9% of paired ratings differ by no more than one point. GPT-5.6 Sol provides a consistent external assessment, with an overall mean of 4.27 and scores above 4.0 for all three benchmark components. These results support the reliability of the generated tasks and the completion and process rubrics that check 8 their constraints. 4.3 Analysis Beyond the aggregate leaderboard, we analyze how constraint satisfaction varies with model release date and across fine-grained rubric categories. Oct '25Jan '26Apr '26Jul '26 60 65 70 75 80 85 90 Score (%) Slope = 1.32 p/month Oct '25Jan '26Apr '26Jul '26 Slope = 1.14 p/month Oct '25Jan '26Apr '26Jul '26 Slope = 2.17 p/month Completion w/ skill (S WS comp )Completion w/o skill (S NS comp )Process w/ skill (S WS proc ) Release Date Figure 3Model performance versus release date under loose scoring. Each panel plotsS WS comp ,S NS comp , and S WS proc against official release date, with dotted lines denoting linear least-squares fits. All three scores are positively associated with release date (r >0.68), with fitted slopes of 1.32, 1.14, and 2.17 p/month, respectively. Corresponding strict scoring results are reported in Figure 7. Model performance over release time. Figure 3 reveals a consistent trend: newer models usually perform better under both evaluation metrics. In particular, the Completion w/ skill (S WS comp ) and Completion w/o skill (S NS comp ) results show medium rates of improvement over time, with slopes of 1.32 and 1.14 percentage points per month, respectively. In contrast, Process w/ skill (S WS proc ) exhibits a substantially steeper upward trend, with a slope of 2.17 percentage points per month. All the Pearsonrvalues are above 0.68, and scores under the strict scoring exhibit a similar pattern (see Figure 7). These results suggest that, for LLM agents, execution-constraint satisfaction is more strongly associated with model release date than outcome-constraint satisfaction, indicating that it more clearly differentiates earlier- and later-released models. Inspectable deliverable Workflow adherence Output structure Scope control Terminology use Iterative verification 50 60 70 80 90 100 Score (%) GeneralDetailed β± β± β± β± Completion rubric Completion w/o skill (S NS comp ) Gain w/ skill (S WS comp Β‘S NS comp ) Opus 4.8HY3Gemini 3.6 Flash Figure 4Loose rubric-category scores for models at different leaderboard positions. Opus 4.8, HY3, and Gemini 3.6 Flash are selected as representatives of different positions in the main leaderboard. For the Completion categories, WS and NS denote evaluation with and without the skill, respectively; the Process categories are evaluated only with the skill. The three models achieve relatively similar Inspectable Deliverable scores but separate more clearly in the remaining categories, particularly the three Process categories. 9 Failure modes across fine-grained rubric categories. To analyze performance gaps across model tiers, we decompose agent pass rates into six functional rubric categories (Appendix D.4), separating the coarse constraint of producing the deliverable (Inspectable Deliverable) from five finer constraints on output format, domain content, and execution. We compare three representative models spanning distinct leaderboard tiers: Opus 4.8, HY3, and Gemini 3.6 Flash. Figure 4 yields two key findings. First, producing final task deliverables does not differentiate model tiers. All three models achieve closely comparable Inspectable Deliverable pass rates, indicating that even lower-tier agents can generate plausible task outcomes. Second, model capabilities diverge primarily on the finer constraints. Gemini 3.6 Flash trails significantly in procedural step adherence and empirical verification loops, whereas HY3 maintains overall workflow structure but struggles with strict output formatting schemas and domain-specific logic. These results suggest that advancing agents requires shifting optimization from producing plausible deliverables toward satisfying fine-grained operational constraints and validation requirements. 5 Conclusion In this work, we present SkillForgeBench, an agentic benchmark that leverages agent skills to bridge broad domain diversity with authentic procedural depth. By anchoring the task lifecycle in agent skills, SkillForgeBench automatically synthesizes 842 executable environments, naturalistic instructions, and fine-grained rubrics that check constraints on both the final outcome and the execution. Our empirical evaluations yield three core findings: (1) strict, holistic constraint satisfaction defines the true capability divide between model tiers; (2) explicit skill guidance primarily amplifies frontier model performance rather than compensating for weaker baselines; and (3) non-frontier agents routinely sacrifice procedural discipline to force final deliverables. Ultimately, SkillForgeBench demonstrates the necessity of moving beyon completion-only checks, providing a scalable foundation for trajectory-level verification and trustworthy agent development. AI Use Statement AI tools were used to assist with and polish manuscript writing, and to support the generation of synthetic benchmark data, including task instructions, environments, and evaluation rubrics. While AI models were used throughout the benchmark synthesis pipeline, every benchmark task was reviewed by human experts before inclusion. The benchmark construction, validation, and evaluation procedures involving AI models are described in detail in the paper. All final content, analyses, and scientific conclusions were reviewed by the authors. References Zihao Cheng, Hongru Wang, Zeming Liu, Xinyi Wang, Xiangrong Zhu, Yuhang Guo, Wei Lin, Jeff Z. Pan, and Yunhong Wang. Terminal-World: Scaling terminal-agent environments via agent skills. In arXiv, 2026. Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, and Yuhang Zang. WildClawBench: A benchmark for real-world, long-horizon agent evaluation. In arXiv, 2026. Guanting Dong, Junting Lu, Junjie Huang, Wanjun Zhong, Longxiang Liu, Shijue Huang, Zhenyu Li, Yang Zhao, Xiaoshuai Song, Xiaoxi Li, Jiajie Jin, Yutao Zhu, Hanbin Wang, Fangyu Lei, Qinyu Luo, Mingyang Chen, Zehui Chen, Jiazhan Feng, Ji-Rong Wen, and Zhicheng Dou. Agent-World: Scaling real-world environment synthesis for evolving general agent intelligence. In arXiv, 2026. Qingnuan Han, Boli Fang, Mingzhi Hou, and Claire Liu. RideWay: Benchmarking efficient task completion for tool-using language agents. ArXiv, abs/2609.17985, 2026. Zhanbo Hua, Yifan Yao, Weihao Xie, Yongchi Zhao, Minghao Liu, Ruizhi Qiu, Zhewei Huang, Zun Wang, Yiyan Ji, Yunhai Ye, Letian Zhu, Xinping Lei, Han Li, Zhiyuan Ma, Zili Wang, Zhaoxiang Zhang, and Jiaheng Liu. CLI-Universe: Towards verifiable task synthesis engine for terminal agents. ArXiv, abs/2606.22883, 2026. 10 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. Kilo AI. PinchBench: Benchmarking system for OpenClaw coding agents. https://pinchbench.com, 2026. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452β466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026/. Yuxiang Lai, Peng Xia, Haonian Ji, Kaiwen Xiong, Kaide Zeng, Jiaqi Liu, Fang Wu, Jike Zhong, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. ClawForge: Generating executable interactive benchmarks for command-line agents. In arXiv, 2026. Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiaojun Wu, Junwei Zhou, Xiaokun Chen, Hanwen Xing, Yubo Li, Qunhong Zeng, Di Wang, Yuanli Wang, Roey Ben Chaim, Penghao Jiang, Haotian Shen, Luyang Kong, Xinyi Liu, Runhui Wang, Xuanqing Liu, Jiachen Li, Xin Lan, Yueqian Lin, Wengao Ye, Junwei He, Songlin Li, Yue Zhang, Yipeng Gao, Yijiang Li, Ze Ma, Liqiang Jing, Tianyu Wang, Kaixin Li, Yiqi Xue, Haoran Lyu, Yizhuo He, Yuchen Tian, Shutong Wu, Bowei Wang, Yixuan Gao, Bo Chen, Litong Liu, Sikai Cheng, Jiajun Bao, Shuaicheng Tong, Shuwen Xu, Terry Yue Zhuo, Tinghan Ye, Qi Qi, Miao Li, Longtai Liao, Zelin Tan, Chang Shi, Xilin Tang, Srinath Tankasala, Boqin Yuan, Yaoyao Qian, Jianhong Tu, Chenguang Wang, Yizhou Sun, Wei Wang, Aaron Taylor, Ziyue Yang, Changkun Guan, Zhikang Dong, Xinyu Zhang, Steven Dillmann, Han chung Lee, and Dawn Song. SkillsBench: Benchmarking how well agent skills work across diverse tasks, 2026a. URL https://arxiv.org/abs/2602.12670. Xirui Li, Ming Li, Ion Stoica, Cho-Jui Hsieh, and Tianyi Zhou. ClawEnvKit: Automatic environment generation for claw-like agents. In arXiv, 2026b. Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. APIGen: Automated PIpeline for generating veri- fiable and diverse function-calling datasets. In Advances in Neural Information Processing Systems, vol- ume 37, p. 54463β54482, 2024. URLhttps://proceedings.neurips.c/paper_files/paper/2024/file/ 61cce86d180b1184949e58939c4f983d-Paper-Datasets_and_Benchmarks_Track.pdf. Xiang Long, Li Du, Yilong Xu, RongJian Xu, Qiyanhui Lu, Ying Gao, Qinhua Xie, Fangcheng Liu, Ning Ding, Haoqing Wang, Ziheng Li, Changjiang Zhou, Jianyuan Guo, and Yehui Tang. LiveClawBench: Benchmarking LLM agents on complex, real-world assistant tasks. In arXiv, 2026. Zongwei Lv, Yaoming Li, Zhewen Tan, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, and Guangxiang Zhao. RealClawBench: Live OpenClaw benchmarks from real developer-agent sessions. In arXiv, 2026. Mike A. Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Kumar Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Kwesi Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, ArinbjΓΆrn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Jenia Jitsev, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=a7Qa4CcHak. Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich, Maria I. Gorinova, Amy Heineike, and Rob Willoughby. A framework for evaluating agentic skills at scale, 2026. URL https://arxiv.org/abs/2606.17819. 11 Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Annual Meeting of the Association for Computational Linguistics, p. 16022β16076, 2024. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 50208β50232. PMLR, 2024. Zhaoyang Wang, Yiming Liang, Xuchao Zhang, Qianhui Wu, Siwei Han, Anson Bastos, Rujia Wang, Chetan Bansal, Baolin Peng, Jianfeng Gao, Saravan Rajmohan, and Huaxiu Yao. SynthAgent: Adapting web agents with synthetic supervision. In Annual Meeting of the Association for Computational Linguistics, p. 15730β15752, 2026. Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang, and Chenghua Lin. Large-scale terminal agentic trajectory generation from Dockerized environments. ArXiv, abs/2602.01244, 2026. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, volume 37, p. 52040β52094, 2024. Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. AgentTrek: Agent trajectory synthesis via guiding replay with web tutorials. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=EEgYUccwsV. Sidi Yang, Chaofan Tao, Jierun Chen, Tiezheng Yu, Ruoyu Wang, Yuxin Jiang, Yiming Du, Wendong Xu, Jing Xiong, Taiqiang Wu, Lifeng Shang, Xiaohui Li, Ngai Wong, and Haoli Bai. What makes interaction trajectories effective for training terminal agents? ArXiv, abs/2606.03461, 2026. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan.Ο-bench: A benchmark forTool-Agent- User interaction in real-world domains. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=roNSXZpUDN. Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, Qi Liu, Zhifang Sui, and Tong Yang. Claw-Eval: Towards trustworthy evaluation of autonomous agents. In arXiv, 2026. Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Zhuofeng Li, Xingwei Qu, Zhengkang Guo, Yuanzhe Shen, Dingjie Song, Han Zhou, Tuney Zheng, Xian Wu, Hao Yu, Songcheng Cai, Yi Lu, Yunzhuo Hao, Minyi Lei, Liang Chen, Kai Zou, Huifeng Yin, Wendong Xu, Dongfu Jiang, Ping Nie, Jiaheng Liu, Wenhu Chen, and Kelsey R. Allen. ClawBench: Can AI agents complete everyday online tasks? In arXiv, 2026. Junyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, and Chen Ma. RubricBench: Aligning model-generated rubrics with human standards. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 31179β31200, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1439. URL https://aclanthology.org/2026.acl-long.1439/. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024. Kaijie Zhu, Yuzhou Nie, Yijiang Li, Yiming Huang, Jialian Wu, Jiang Liu, Ximeng Sun, Zhenfei Yin, Lun Wang, Zicheng Liu, Emad Barsoum, William Yang Wang, and Wenbo Guo. TermiGen: High-fidelity environment and robust trajectory synthesis for terminal agents. In arXiv, 2026. 12 A Skill Gains as a Function of Baseline Performance 606570758085 5 5.5 6 6.5 7 7.5 8 Delta ( Β’ ) (%) Completion w/o skill ( S NS comp ) Loose score (%) Pearson r = 0.42 606570758085 0 5 10 15 20 25 Delta ( Β’ ) (%) Completion w/o skill ( S NS comp ) Loose score (%) Pearson r = 0.94 0510152025 Completion w/o skill ( S NS comp ) Strict score (%) Pearson r = 0.97 Loose scoring Strict scoringStrict scoring Figure 5Gains from skill access versus baseline completion performance. We define the skill gain, β, as the difference between the completion scores obtained with and without skill access: β =S WS comp β S NS comp , where WS and NS denote the with-skill and without-skill conditions, respectively. The scatter plots relate the loose-score gain to the looseS NS comp baseline and the strict-score gain to both loose and strictS NS comp baselines. Each point represents one model, dotted lines show least-squares fits, and annotations report Pearsonβsr. Loose-score gains are comparatively uniform (5.2%β7.4%;r= 0.42), whereas strict-score gains increase strongly with baseline performance (r= 0.94 andr= 0.97). Figure 5 examines how gains β =S WS comp β S NS comp from skill access vary with baseline completion no skill scoreS NS comp . Under loose scoring, β are comparatively uniform across models, ranging from 5.2% to 7.4%, and are only moderately correlated with the no-skill baseline (r= 0.42). Under strict scoring, however, β vary substantially, from 5.8% to 19.1%, and are strongly correlated withS NS comp measured under both loose (r= 0.94) and strict (r= 0.97) scoring. This pattern supports the main finding: skill grounding provides broadly similar partial-credit improvements across models, but models with stronger baseline capabilities are considerably more likely to use the guidance to satisfy the full set of strict requirements. Thus, skill grounding appears to help models capitalize on existing operational competence rather than uniformly compensate for weak baseline capability. 13 B Model Performance Over Steps 01020304050 60 65 70 75 80 85 90 Score (%) Opus 4.8 GLM-5.2 Sonnet 4.6 HY3 Gemini 3.6 Flash Kimi K2.6 Qwen3.5-397B Haiku 4.5 More efficient 01020304050 Opus 4.8 GLM-5.2 Sonnet 4.6 HY3 Gemini 3.6 Flash Kimi K2.6 Qwen3.5-397B Haiku 4.5 More efficient 01020304050 Opus 4.8 GLM-5.2 Sonnet 4.6 HY3 Gemini 3.6 Flash Kimi K2.6 Qwen3.5-397B Haiku 4.5 More efficient 01020304050 0 10 20 30 40 Score (%) Opus 4.8 GLM-5.2 Sonnet 4.6 HY3 Gemini 3.6 Flash Kimi K2.6 Qwen3.5-397B Haiku 4.5 More efficient 01020304050 Opus 4.8 GLM-5.2 Sonnet 4.6 HY3 Gemini 3.6 Flash Kimi K2.6 Qwen3.5-397B Haiku 4.5 More efficient 01020304050 Opus 4.8 GLM-5.2 Sonnet 4.6 HY3 Gemini 3.6 Flash Kimi K2.6 Qwen3.5-397B Haiku 4.5 More efficient Loose Completion w/ skill (S WS comp )Completion w/o skill (S NS comp )Process w/ skill (S WS proc ) Strict Steps Figure 6Model performance versus trajectory length. Each panel plots model performance against mean trajectory length, measured in interaction steps. Rows correspond to loose and strict scoring, while columns report completion w/ skill (S WS comp ), completion w/o skill (S NS comp ), and process with the skill (S WS proc ). The horizontal axis is reversed, such that shorter trajectories appear farther to the right. Dashed lines indicate the cross-model mean score and mean trajectory length; consequently, the shaded upper-right region identifies models that achieve above-average performance with fewer-than-average steps. Across metrics and scoring schemes, longer trajectories do not consistently correspond to higher performance. 14 C Model Performance in Strict Scheme Over Release Time Oct '25Jan '26Apr '26Jul '26 0 10 20 30 40 Score (%) Slope = 1.82 p/month Oct '25Jan '26Apr '26Jul '26 Slope = 0.96 p/month Oct '25Jan '26Apr '26Jul '26 Slope = 2.08 p/month Completion w/ skill (S WS comp )Completion w/o skill (S NS comp )Process w/ skill (S WS proc ) Release Date Figure 7Model performance versus release date under strict scoring. Each panel plots model performance against official release date for completion with the skill (Completion WS), completion without the skill (Completion NS), and skill following with the skill (Following WS). Points represent individual models, dotted lines show linear least-squares fits, and the annotated slopes are measured in percentage points (p) per month. Release date is positively correlated with all three metrics, with Pearsonβsrvalues of 0.536, 0.502, and 0.574, respectively. As under loose scoring, Following WS exhibits the steepest fitted trend (2.08 p/month), compared with Completion WS (1.82 p/month) and Completion NS (0.96 p/month). 15 D Task Synthesis Details D.1 Synthesis Model and GPT-Series Exclusion At every stage of our data-synthesis pipeline, we use GPT-5.5 in Codex with xhigh reasoning effort as the executor agent. Consequently, the synthesized tasks and rubrics may be comparatively in-distribution for GPT-series models, potentially giving them a generator-family advantage. We therefore exclude GPT-series models from evaluation. D.2 The List of 20 Predefined Forums Table 4 Forums and their primary discussion topics. ID Forum nameDescription 1 RedditTopic-based communities called subreddits. 2 QuoraQuestions, answers, and opinion discussions. 3 Stack ExchangeExpert Q&A network, including Stack Overflow. 4 Stack OverflowProgramming questions and answers. 5 Tripadvisor ForumsTravel discussions. 6 XDA DevelopersAndroid, mobile devices, ROMs, and device modification. 7 Hacker NewsTechnology, startups, and programming discussions. 8 Digital PointSearch engine optimization, web marketing, and webmaster discussions. 9 Warrior ForumInternet marketing and online business. 10 MacRumors ForumsApple products and rumors. 11 Mumsnet TalkParenting, family life, relationships, and lifestyle discussions. 12 Linus Tech Tips ForumPC building, hardware, and technology. 13 Ubuntu ForumsUbuntu and Linux support. 14 BleepingComputer Forums Malware removal and computer support. 15 AVS ForumHome theater, televisions, and audio/video equipment. 16 Bodybuilding.com Forums Fitness and bodybuilding. 17 The Student RoomStudent life, universities, and examinations. 18 Wrong PlanetAutism, neurodiversity, and life advice. 19 ResetEraGaming, entertainment, and culture. 20 Mumsnet TalkParenting and lifestyle discussions. D.3 Skill Domain Distribution Figure 8 shows how the 842 tasks divide into domains and sub-fields, and Table 5 gives the exact shares and the fine-grained topics within each sub-field. D.4 Criteria Category During contrastive rubric construction, each criterion is assigned to a category. There are three categories in completion rubrics and three in following rubrics, specifically defined as follows. 1. Durable Artifact: Assesses whether the agent produces a persistent and inspectable deliverable, such as an executable proof of concept, test, before-and-after demonstration, or report, in the form prescribed by the skill. 2.Output Discipline: Assesses whether the agent follows the required output structure and reporting conventions, including mandated summaries, scores, counts, or verdict formats. 3.Domain Operationalization: Assesses whether the agent correctly applies the skillβs domain-specific concepts in practice, rather than merely repeating relevant terminology. 16 Technology (257) Software engineering (190) Infrastructure & DevOps (28) Security & privacy (27) Data & machine learning (12) Business (211) Strategy & operations (113) Marketing & growth (60) Finance & investment (38) Creative (136) Design & storytelling (72) Writing & documentation (64) Harness Behavior (125) Agent & skill engineering (125) Knowledge (113) Reasoning & decision-making (33) Personal & lifestyle (32) Education & teaching (25) Research & analysis (23) Total (842 tasks) Figure 8Domain and sub-field composition of the 842 SkillForgeBench tasks. The inner ring shows the five domains and the outer ring their 14 sub-fields, each sized by its number of tasks; since each task comes from a distinct skill, the same numbers hold for skills. 1.Phase Adherence: Assesses whether the agent visibly follows the skillβs named workflow phases in the prescribed order. 2.Empirical Verification Loop: Assesses whether the agent probes the issue before acting, verifies the result afterward, and uses failβdiagnoseβfix cycles when necessary. 3.Scope Discipline: Assesses whether the agent checks the skillβs applicability and task readiness, remains within the appropriate scope, and resists overreach or unsupported rationalization. D.5 Strict rubric-category scores across capability tiers 17 Table 5Domain distribution of the skills behind the 842 SkillForgeBench tasks. Each task is synthesized from a distinct skill, so skill counts equal task counts. The % column gives the share of the 842 skills in each row. Topics is the number of distinct fine-grained topics a sub-field covers (83 in total). The last column lists the three most frequent topics with their skill counts; ties are broken by taxonomy order. Domain / sub-fieldSkills% Topics Most frequent topics (skills) Technology25730.527 Software engineering190 22.610 Code implementation & refactoring (88); architecture & design decisions (31); debugging & root-cause analysis (19) Infrastructure & DevOps283.36 Cloud platforms & IaC (8); observability & alerting (7); CI/CD & release management (6) Security & privacy273.26 Security operations & incident response (10); threat modeling & security architecture (7); privacy & compliance auditing (4) Data & machine learning121.45 Data processing & pipelines (4); querying & SQL (3); machine & deep learning (3) Business21125.117 Strategy & operations113 13.46 Strategy & competitive analysis (43); product requirements & planning (22); organization, talent & leadership (18) Marketing & growth607.15 Social-media operations (25); advertising & marketing copy (16); SEO, GEO & ASO (10) Finance & investment384.56 Macro & commodity research (13); equity analysis (9); market-data APIs & tools (5) Creative13616.212 Design & storytelling728.66 UI/UX & visual design (21); fiction, scripts & stories (16); infographics & diagrams (11) Writing & documentation647.66 Technical & product documentation (38); office documents & presentations (12); translation & language editing (5) Harness Behavior12514.86 Agent & skill engineering125 14.86Skill authoring & governance (68); prompt engineering (19); tool & MCP building (12) Knowledge11313.421 Reasoning & decision-making333.96 Decision frameworks & scenario analysis (12); first-principles decomposition (7); debate & red-teaming (6) Personal & lifestyle323.86 Personal growth & self-management (10); metaphysics, divination & spirituality (6); healthcare & life sciences (5) Education & teaching253.04 Course & teaching materials (15); tutoring & problem solving (6); pedagogy & learning strategies (2) Research & analysis232.75 Academic writing & submission (11); deep research & retrieval (4); fact-checking & argument auditing (4) Total842 100.083 5 domains, 14 sub-fields Table 6Rubric-category scores under loose and strict scoring. Category-level averages are reported over the common set of 842 evaluation tasks. The completion metric (S comp ) comprises Durable Artifact, Output Discipline, and Domain Operationalization, evaluated both with and without the skill. The following metric (S follow ) comprises Phase Adherence, Empirical Verification Loop, and Scope Discipline, evaluated with the skill. Under loose scoring, each taskβcategory instance receives the weighted mean of its applicable criteria. Under strict scoring, it receives full credit only when every applicable criterion attains its maximum score, and zero otherwise. All reported values are percentages. Bold marks the highest score, and underliningmarks the second-highest score in each column. LooseStrict ModelS comp S follow S comp S follow Durable Artifact Output Discipline Domain Operationalization Phase Adherence Empirical Verification Loop Scope Discipline Durable Artifact Output Discipline Domain Operationalization Phase Adherence Empirical Verification Loop Scope Discipline w/ skillwo/ skillw/ skillwo/ skillw/ skillwo/ skillw/ skillw/ skillw/ skillw/ skillwo/ skillw/ skillwo/ skillw/ skillwo/ skillw/ skillw/ skillw/ skill Opus 4.889.287.588.981.587.182.294.187.390.172.266.069.452.657.745.575.150.169.8 GLM-5.288.385.985.576.885.680.396.390.180.769.362.064.646.655.744.786.263.248.7 HY385.382.683.075.283.276.593.383.186.963.557.058.942.350.134.474.241.562.8 Sonnet 4.686.984.882.975.584.078.785.674.882.966.359.755.843.249.838.752.028.553.4 Haiku 4.578.377.473.466.371.667.384.660.867.652.948.640.827.825.319.546.511.526.9 Kimi K2.683.781.780.472.979.875.583.869.783.458.853.851.037.839.232.049.519.954.2 Qwen3.5-397B79.979.075.468.373.669.183.256.777.053.949.340.829.327.121.044.47.0242.3 Gemini 3.6 Flash 86.684.677.869.581.075.180.871.176.063.258.046.832.140.227.843.318.842.5 18 Inspectable deliverable Workflow adherence Output structure Scope control Terminology use Iterative verification 0 20 40 60 80 100 Score (%) GeneralDetailed β± β± β± β± Completion rubric Completion w/o skill (S NS comp ) Gain w/ skill (S WS comp Β‘S NS comp ) Opus 4.8HY3Gemini 3.6 Flash Figure 9Strict rubric-category scores for models at different leaderboard positions. Opus 4.8, HY3, and Gemini 3.6 Flash are selected as representatives of different positions in the main leaderboard. For the Completion categories, WS and NS denote evaluation with and without the skill, respectively; the Process categories are evaluated only with the skill. Under strict scoring, credit is awarded only when all applicable requirements within a category are fully satisfied. Gemini 3.6 Flash remains comparable to HY3 in Inspectable Deliverable but trails both HY3 and Opus 4.8 in Output Structure, Terminology Use, and all three Process categories. Full numerical results are reported in Table 6. Table 7 Descriptive statistics for the 842-task cohort. MeasureMean Population SD Median Skill files per task11.3228.906 Distinct file types per task9.4611.174 Logical task size (MiB)13.6727.323.37 Files per task172.89434.8010 Instruction length (characters)1,333.05311.371,306.5 19 E Prompt Templates for Benchmark Construction E.1 Resource preparation E.1.1 Resource requirements planning # Requirements Planner Agent Prompt v0.3 You are the Codex Requirements Planner Agent in a data synthesis pipeline. Your job is to read the complete supplied source skill, which is composed of the `SKILL.md` and ,β its companions (`reference/`, `scripts/`, `agents/`, etc.). Finally, produce ` ,β resource_requirements.v0.3` JSON, which describes the concrete input resources that the ,β skill needs in order to perform its tasks. Hard rules: - **Output JSON only**, matching `resource_requirements.schema.json`. - Do not download external resource bytes. Do not run package managers, `git clone`, `curl`, ` ,β wget`, `requests`, or browser-save operations. - Use only resource types listed in the `## Resource type compatibility (authoritative)` table ,β later in this prompt. That table is generated from the same source the deterministic ,β validator uses; do not invent types or subtypes. Use the accompanying contract, schema, taxonomy, and complete skill context as the source of ,β truth. E.1.2 Requirements extraction contract # Requirements Extraction Contract v0.3 ## Core Task In this pipeline, we need to extract from the complete source skill the **actual input resources ,β ** that the skill would require when performing tasks that make use of it. **The skill itself implicitly describes what it operates on**. For example, a code review skill ,β needs changed code, diffs, source context, tests, and review guidelines; a document ,β summarization skill needs complete documents; an SQL generation skill needs database ,β schemas, example data, and query/task context. Your core task is: **read the complete source skill (`SKILL.md` and its companions), understand ,β what the skill is supposed to do, then deduce the input files/resources that must be ,β provided for the skill to work properly, and output these requirements in a structured ,β format**. You **must avoid** extracting resources that merely look relevant by format but are unused by ,β the skill. Do not request an irrelevant demo SQLite database, a generic DOCX template, a ,β random Markdown syntax guide, or a package lockfile unless the skill actually operates on ,β that object. What you need are the **object resources that can be directly manipulated by ,β the skill to accomplish its described task**. The output resource requirements will be passed to a _Resource Research Agent_, which will ,β search the web for real, public, stable resources. Later deterministic stages will validate, ,β download, and safely extract only resources listed by that research agent. ## Required behavior 20 - Keep the requirements compact, normally 4-10 items. - Extract intent, domain, primary operations, expected inputs/outputs, forbidden scope, and ,β domain-specific terminology from the **full source skill**, not only name or description. - Determine resource types based on skill operational semantics, not fixed format coverage. ,β Avoid format-only requirements. If a file format is requested, explain why that format is ,β naturally operated on by the skill. - `resource_family` is fully determined by `resource_type`: set each requirement's ` ,β resource_family` to exactly the Family shown for its type in that table, never choose it ,β independently. Likewise pick `acquisition` from that type's listed acquisition modes, and ( ,β for Extractable=yes types) set `archive_constraints.extract` to true. - Set `planner_assertions.read_full_skill_md`, `planner_assertions.did_not_download`, and ` ,β planner_assertions.used_schema` to `true`. ### Required Specifics Preferences - Include linkage hints when multiple resources are semantically coupled, such as source + ,β manifest + config + tests from one repository snapshot. - Include search queries that help the next agent find stable public resources. - Include enough constraints for the next agent and deterministic downloader: HTTPS, no auth, ,β expected MIME/extensions, acquisition modes, archive extraction expectations, GitHub ,β pinning policy, and linkage hints. - Prefer requirements that can be satisfied by public, stable, no-auth resources. ### Rules For Specific Tpyes - For HTML/documentation, request complete pages or page-set bundles if one page is incomplete. - For GitHub-related selected resources, require full 40-character commit pinning. - For code-review or diff/PR-based skills, remember a review spans two commit states: a base ,β tree plus a proposed change on top. When the requirement set pairs a diff/PR (the change ,β under review) with its repository snapshot, the repository requirement's `pinning_policy`/` ,β linkage_hints` must state that the repository is pinned to the change's **base** commit ( ,β pre-change state), not the merged/head state, and should carry the base/head relationship ( ,β the research agent records these as `repository.base_sha`/`head_sha` on the diff/PR ,β resource). This does not apply when the diff/PR is only background context for editing the ,β current code. ## Archive rule Archive requirements must describe expected members and extraction limits. The deterministic ,β downloader performs safe extraction. Agents never download or extract resource bytes. E.1.3 Resource taxonomy # Resource Taxonomy v0.3 The v0.3 taxonomy describes input resources that a skill may genuinely operate on. The ,β Requirements Planner Agent chooses from these types after reading the complete source skill; ,β deterministic code validates that choices stay within this taxonomy. ## Families - `document`: `pdf`, `text`, `markdown`, `docx`, `pptx` - `web`: `html` - `structured_data`: `csv`, `tsv`, `json`, `jsonl`, `yaml`, `toml`, `xml`, `parquet`, `xlsx` - `code`: `github_repository`, `source_code`, `diff_patch`, `github_issue`, `github_pull_request ,β ` 21 - `manifest`: `package_manifest`, `lockfile`, `openapi_spec` - `config`: `config` - `archive`: `archive`, `zip_archive`, `tar_archive`, `tar_gz_archive`, `package_archive` - `database`: `sqlite` - `geospatial`: `geojson`, `shapefile_archive` - `notebook`: `notebook` - `media`: `image`, `audio`, `video` ## Acquisition modes - `direct_file` β URL points directly to downloadable bytes. - `api_endpoint` β URL points to a public no-auth endpoint returning resource bytes or JSON/HTML ,β representation. - `landing_page` β URL points to a public page intended as the resource itself, usually HTML. - `github_archive` β URL points to a GitHub archive pinned to a 40-character commit SHA. - `raw_file` β URL points to a raw source file, preferably pinned when GitHub-hosted. - `archive_extract` β URL points to an archive that must be safely extracted before benchmark ,β use. Resource choices must use an acquisition mode compatible with the chosen resource type. ## Validation depth Deep-ish stdlib validation: - PDF magic/header checks - CSV/TSV parsing - JSON/JSONL parsing - XML parsing - TOML parsing when `tomllib` is available - SQLite header checks - notebook JSON shape checks - ZIP/TAR structural checks - DOCX/PPTX/XLSX container path checks - source code/text heuristics - diff marker checks - package manifest and lockfile structure where feasible Shallow validation: - YAML - Parquet - image/audio/video magic bytes and MIME/extension checks - some package archive formats ## GitHub resources Selected GitHub resources must be pinned for repeatability: - `github_repository` uses a `github.com/<owner>/<repo>/archive/<40-char-sha>.zip` or equivalent ,β pinned archive URL. - raw file resources use `raw.githubusercontent.com/<owner>/<repo>/<40-char-sha>/<path>`. - repository metadata records owner, repo, requested ref when known, and commit SHA. - repo homepages are allowed as `landing_url`, not as the selected fetch URL for ` ,β github_repository`. ## Archive resources Archive-like resource types are extractable and must declare archive handling: 22 - `github_repository` - `archive` - `zip_archive` - `tar_archive` - `tar_gz_archive` - `package_archive` - `shapefile_archive` The deterministic downloader extracts these with path-safe member handling and budget limits. ,β Requirements and manifests should describe expected top-level entries, required member ,β paths or extensions when known, maximum members, maximum extracted bytes, and whether ,β nested archives are allowed. ## Bundles Some skills need linked files rather than a single URL. The manifest schema supports bundles for ,β cases such as: - multiple raw files from the same repository commit - documentation page sets - dataset releases with separate metadata and data files Bundles must declare why the files belong together and which members are required. E.1.4 Artifact retrieval # Resource Research Agent Prompt v0.3 You are the Resource Research Agent for a data synthesis pipeline. ## Core Task A Requirements Planner Agent has analyzed a complete source _skill_, which is composed of the ` ,β SKILL.md` and its companions (`reference/`, `scripts/`, `agents/`, etc.). Finally, produced ,β `resource_requirements.v0.3` JSON that lists the input resources the skill needs to ,β operate. You receive the requirements JSON (inlined below) plus the same complete skill, ,β which has been copied into your working directory at `./skill/` (the `SKILL.md` and its ,β companions). **Before you search for anything, read the skill directly from `./skill/`** so ,β your resource choices match the skill's real operational semantics β do not skip it, and ,β never substitute a web-hosted copy of the skill for the local files. Your role is to turn these abstract requirements into concrete, downloadable URLs pointing to ,β real, public, stable resources. You do **not** download anything yourself. You search the ,β web, evaluate candidates, and output a structured manifest that a deterministic downloader ,β will later use. The downstream pipeline depends on your choices: if you pick an unstable, private, unavailable, ,β or semantically weak resource, data synthesis will fail or become misleading. Use the ,β complete original skill to avoid format-only substitutions. ### Your Job - **First, read the complete source skill from `./skill/` in your working directory** β `./skill/ ,β SKILL.md` plus every companion file listed in the source-skill section below β before doing ,β any web search. Read it from disk; it is not inlined in this prompt. 23 - Read the `resource_requirements.v0.3` JSON (inlined below) and cross-check each requirement ,β against what the skill at `./skill/` actually operates on. - Use web search to identify real, public, stable resources that satisfy each requirement. - Return only JSON matching `resource_manifest.schema.json`. - Provide one selected resource and at least one alternative for every requirement. Alternatives ,β must satisfy the same requirement `resource_type`; do not use an HTML landing page as the ,β alternative for a PDF requirement or similar type substitution. - Decide when a requirement is best represented by a linked bundle of files/pages instead of a ,β single file. ## General Rules ### Hard Rules - Do not download files or resource bytes. - Do not run `curl`, `wget`, `requests`, browser-save operations, package managers, `git clone`, ,β or commands whose purpose is fetching resource bytes. - Do not create local resource files. - Do not return search result pages as selected resource URLs. - Do not invent URLs. - Do not select resources requiring login, cookies, API keys, CAPTCHA, private credentials, or ,β temporary signed URLs. - The deterministic downloader will download only URLs listed in your manifest. It will not ,β search, crawl, or infer additional URLs. - Do not set `agent_assertions.read_full_skill_md: true` unless you actually read `./skill/SKILL. ,β md` and its companions from disk first. Reading a web-hosted copy of the skill does not ,β count and must not replace reading the local `./skill/`. ### Preferences - Prefer small, stable, public resources with clear provenance. - Prefer a lightweight project that can run on a normal personal laptop. It must not require ,β server-grade GPUs or large amounts of RAM β and, if it is software, easy to set up and run. ## Required manifest behavior - `schema_version` must be `resource_manifest.v0.3`. - `generated_by.agent` must be `codex-resource-research-agent`. - `agent_assertions.did_not_download`, `agent_assertions.search_only`, and `agent_assertions. ,β read_full_skill_md` must all be `true`. - `source_requirements_sha256` must exactly match the caller-supplied value. - `source_skill_context_sha256` must exactly match the caller-supplied value. - Every candidate must include `url`, `landing_url`, `acquisition`, `repository`, ` ,β archive_handling`, `bundle`, provenance, and validation notes. - `acquisition` must be exactly one of: `direct_file`, `api_endpoint`, `landing_page`, ` ,β github_archive`, `raw_file`, `archive_extract`. Do not invent values such as `html_page`, ` ,β landing_page_set`, or `raw_file_bundle`; represent sets/bundles only via the `bundle` field. ,β - Use `repository: null` for non-repository resources. - Use `archive_handling: null` for non-archive resources. - Use `bundle: null` for a single-file/single-page candidate. ## Resource type, family, and field compatibility - A `## Resource type compatibility (authoritative)` table is included later in this prompt. It ,β is generated from the same source the deterministic validator uses, so any value it does ,β not list **will be rejected**. Obey it exactly. - `resource_type` (on every candidate, bundle member, and resource group) MUST be one of the ,β rows in that table. Do **not** invent new types or subtypes. In particular, there is no ` 24 ,β github_issue_comments` or `github_review_comments` type β model PR review comments / issue ,β comments as part of a `github_pull_request` or `github_issue` resource (its public API JSON, ,β or as bundle members of that same type). - `resource_family` is fully determined by `resource_type`: set it to exactly the Family shown ,β for your chosen type in the table. Never choose it independently. - `expected_mime_type`, `expected_extension`, and `acquisition` MUST each be one of the values ,β listed for that type's row. The table shows `""` when an empty extension is allowed; for ,β types whose allowed extensions are filenames (e.g. `package_manifest` allows `Gemfile`, ` ,β package.json`, `.gemspec`), use one of those exact tokens rather than leaving it blank. For ,β Extractable=yes types, set `archive_handling` with `extract: true`. - If the only realistic resource for a requirement does not fit any row's allowed values (e.g. a ,β spreadsheet only published inside a `.zip`), pick the closest fitting `resource_type` ,β whose values it satisfies, or choose a different resource β do not bend a type's fields to ,β values the table does not list. ## Specific Rules ### GitHub reproducibility rules - Selected GitHub resources MUST be pinned to a full 40-character commit SHA. - `github_repository`: use a GitHub archive URL pinned to a commit, e.g. `https://github.com/< ,β owner>/<repo>/archive/<40-char-sha>.zip`; set `landing_url` to a GitHub page for the repo/ ,β ref; fill `repository.ref` and `repository.commit_sha` with that same SHA. - `source_code`, `package_manifest`, `lockfile`, `config`, `markdown`, `openapi_spec`, `notebook ,β `, and other raw GitHub files: use `https://raw.githubusercontent.com/<owner>/<repo>/<40- ,β char-sha>/<path>`, not `main`, `master`, a tag, or a GitHub `blob` page. - Do not use `/archive/refs/heads/...`, `/archive/refs/tags/...`, `/main`, `/master`, or any ,β branch/tag ref for selected resources. Selected raw GitHub URLs must include the 40- ,β character commit SHA in the URL path and in `repository.ref`/`repository.commit_sha`. - If you cannot pin a GitHub resource to a commit SHA, do not select it as a candidate resource; ,β choose a different resource or put it only in alternatives with the risk clearly stated. - If a candidate has multiple GitHub bundle members that are meant to be used together, they ,β must come from the same owner/repo and the same commit SHA. ### Code-review base pinning - A code review is inherently a **two-commit** thing: a base tree plus a proposed change applied ,β on top. Keep this in mind whenever a requirement provides a `diff_patch` or ` ,β github_pull_request` as the **change under review** (something to review, critique, or ,β apply β not merely background context). - In that case, when you also select that change's repository as a `github_repository` snapshot, ,β pin the repository archive to the change's **base commit** (the pre-change state), **NOT** ,β the PR `merge_commit_sha` or the `head` SHA. If you pin to the merge/head, the change ,β being reviewed is already present in the snapshot, which defeats the review. - For a GitHub PR, the base commit is the PR's `base.sha`. Pin the `github_repository` archive ,β URL and `repository.commit_sha` to that base SHA. - The `diff_patch` / `github_pull_request` resource then represents the proposed change (base ,β β head) layered on top of that base snapshot. - Record the change endpoints on the **diff/PR resource's** `repository` block: set ` ,β repository.base_sha` to the PR `base.sha` and `repository.head_sha` to the PR `head.sha` ( ,β that resource's own `commit_sha` may stay the merge/landing commit). This lets the ,β deterministic validator confirm the paired `github_repository` is pinned to the base. Do ,β NOT overload `bundle.linkage.commit` for this. - `base_sha` and `head_sha` are required fields on every `repository` object. For any resource ,β that is NOT a diff/PR under review (the repository snapshot itself, raw files, configs, ,β etc.), and whenever a value is unknown, set both to `null`. - Exception: if the diff/PR is included only as **background/thematic context** (e.g. a refactor ,β skill that edits the current code and merely uses the PR/issue to point at a relevant area) 25 ,β , the repository may be pinned to any coherent snapshot and base-pinning is not required. ,β State which mode applies in the candidate's `why_it_matches` / `rationale`. ### Linked bundle rules - Use `bundle: null` when one URL is sufficient. - Use `bundle.mode: "multi_file"` when the we need coordinated files, e.g. source + manifest + ,β config + tests + docs from one project snapshot. - Use `bundle.mode: "html_page_set"` when an HTML resource is split across multiple pages and ,β one page alone is incomplete for the requirement. - Bundle members must be explicit URLs. Do not rely on the downloader to crawl links. - Keep bundles small and necessary, normally 2-6 members and never more than the schema limit. - Every bundle needs a clear `rationale` explaining why the files/pages must be used together. - For `same_github_repo_commit` bundles, fill `bundle.linkage.repository` as `owner/repo` and ` ,β bundle.linkage.commit` as the 40-character SHA. - Every member needs `id`, `role`, `resource_type`, `url`, `acquisition`, `expected_mime_type`, ,β `expected_extension`, `requires_auth: false`, `repository`, `archive_handling`, `required`, ,β and `why_needed`. ### HTML completeness rules - For HTML requirements, assess whether the selected page contains the complete relevant content. ,β - Avoid search pages, login pages, thin landing pages, JavaScript-empty shells, and pages that ,β merely link to the actual content. - If content is split across pages, sections, print pages, canonical pages, or required subpages, ,β return an `html_page_set` bundle listing the needed pages. - Do not include many loosely related pages. Include only pages needed to make the resource ,β complete for the requirement. ### Resource-specific rules - `diff_patch`: prefer `.diff` or `.patch` URLs, or GitHub PR `.diff` URLs. - `github_issue` / `github_pull_request`: prefer public GitHub API JSON or stable HTML pages if ,β API JSON is not practical. - Archives: include `archive_handling` with expected entries/extensions and conservative ,β extraction limits. - Office docs (`docx`, `pptx`, `xlsx`): prefer small public templates/examples from ,β authoritative sources only when the requirement semantically calls for them. - Media: prefer small files; include size risks if unknown. - Humanities/religion/philosophy/literature/genealogy: prefer canonical texts, translations, ,β academic references, official exam/teaching materials, public-domain sources, library/ ,β archive pages, and primary sources. Do not substitute unrelated policy/statistics resources ,β merely because a keyword overlaps. ## Candidate bundle example shape ```json "url": "https://raw.githubusercontent.com/owner/repo/0123456789abcdef0123456789abcdef01234567/ ,β README.md", "landing_url": "https://github.com/owner/repo/tree/0123456789abcdef0123456789abcdef01234567", "title": "Example linked project files", "publisher": "owner/repo", "source_type": "github", "acquisition": "raw_file", "expected_mime_type": "text/plain", "expected_extension": ".md", "requires_auth": false, 26 "license_or_terms": "Repository license applies.", "repository": "host": "github.com", "owner": "owner", "repo": "repo", "ref": "0123456789 ,β abcdef0123456789abcdef01234567", "path": "README.md", "commit_sha": "0123456789 ,β abcdef0123456789abcdef01234567", "requested_ref": null, "pin_evidence_url": "https://github. ,β com/owner/repo/commit/0123456789abcdef0123456789abcdef01234567", "archive_handling": null, "bundle": null, "provenance": "discovered_via_queries": ["..."], "supporting_pages": ["..."], " ,β evidence_snippets": ["..."], "checked_at": "2026-06-10T00:00:00Z", "agent_validation": "confidence": 0.9, "why_it_matches": "...", "known_risks": [], " ,β expected_size_bytes": null ``` E.1.5 Skill intent summarization # Skill Analysis ## Objective Read the target skill's description and produce a **general, short sentence** β at most **12 ,β words** β that summarizes what the user wants to do by installing the skill for LLM agents. ,β Note that it is not about what the skill does **but about what the user wants to ,β accomplish**. It must **not** be written in the **user's first-person voice** (for example, _"Help me ..."_) ** ,β but** in an imperative sentence (start with a verb). In addition, it should be conclusive ,β enough to cover the skill's overall purpose and functions rather than emphasizing any ,β single feature. **Note that do not include anything like "follow the skill / guidelines".** Base the summary **only on the skill's description**; do not inspect the skill's implementation, ,β scripts, or bundled files, and do not invent capabilities the description does not state. ,β Write the result to `skill_analysis.json` with the fields `skill_name` and `general_request ,β `. ## Read Context - `SKILL_DIR/SKILL.md` β the target skill's description (read the description only). - `PROMPTS_DIR` β this prompt directory. ## Owned Writable Paths - `OUTPUT_DIR/skill_analysis.json` β the single output artifact. ## Forbidden Paths and Actions - Do not read or rely on anything beyond the skill's description. - Do not modify, rename, or delete the skill files or any path outside `OUTPUT_DIR`. - Do not exceed 12 words in `general_request`. - Do not name the skill. Do not quote skill-internal terms inside it. Do not say anything like " ,β follow the skill / guidelines". - Do not install dependencies, access the network, or run destructive commands. 27 E.1.6 Forum request retrieval # Request Form Read `skill_analysis.json`, `SKILL.md`, and skill's companions. The contents of `" ,β general_request"` from `skill_analysis.json` is a very general summary of a user's request. ,β Amplify the request with real human's requests. To achieve this, search on the internet mainly using the contents of `"general_request"` from ` ,β skill_analysis.json`. Note that the contents of `"general_request"` from `skill_analysis. ,β json` **is always much more important** than the skill itself here. Do not use the skill to ,β search. The skill just provides some contexts if needed. The sources must be the following 20 popular forums / discussion communities: 1. **Reddit** β massive topic-based communities called subreddits 2. **Quora** β Q&A and opinion discussions 3. **Stack Exchange** β expert Q&A network; includes Stack Overflow 4. **Stack Overflow** β programming Q&A 5. **Tripadvisor Forums** β travel discussions 6. **XDA Developers** β Android, mobile, ROMs, device modding 7. **Hacker News** β tech/startup/programming discussions 8. **Digital Point** β SEO, web marketing, webmaster forum 9. **Warrior Forum** β internet marketing and online business 10. **MacRumors Forums** β Apple products and rumors 11. **Mumsnet Talk** β parenting, family life, relationships, and lifestyle discussions. 12. **Linus Tech Tips Forum** β PC building, hardware, tech 13. **Ubuntu Forums** β Ubuntu/Linux support 14. **BleepingComputer Forums** β malware removal and computer help 15. **AVS Forum** β home theater, TVs, audio/video gear 16. **Bodybuilding.com Forums** β fitness and bodybuilding 17. **The Student Room** β student life, universities, exams 18. **Wrong Planet** β autism, neurodiversity, life advice 19. **ResetEra** β gaming, entertainment, culture 20. **Mumsnet Talk** β parenting and lifestyle discussions Specifically, spawn 20 sub-agents, each of which is responsible for one forum and returns one ,β user request candidate. Then, write one user request based on these 20 candidates. This ,β user request can be composed by selecting, merging, and polishing any of these candidates. Make sure every bit of information comes from these candidates. LLM's internal knowledge and ,β skill information are **forbidden** to be written into the user request. Skill knowledge ,β must **not** be incorporated into the user request. Also, write down the url of 20 candidates. Specify which of these 20 candidates are used to ,β construct the final user request. Finally, quickly check if the skill can be applied to help solve this user request. Again, the ,β contents of `"general_request"` from `skill_analysis.json` **is always much more important** ,β than the skill itself here. Do not include the skill into the composition of the user ,β request. ## Read Context - `SKILL_DIR/` β the target skill and its companions. - `SKILL_ANALYSIS_OUTPUT/skill_analysis.json` β Stage 0's outcome; its `"general_request"` ,β field is the general summary to amplify. ## Owned Writable Paths 28 - `OUTPUT_DIR/request_form.json` β the single output artifact. ```json "general_request": "string β copied verbatim from skill_analysis.json", "user_request": "string β the single final user request composed from the used candidates", "used_candidate_ids": [1, 3, 4, 7], "skill_applicability": "can_apply": true, "reason": "string β one sentence on whether the skill can help with user_request" ``` - `OUTPUT_DIR/source_url.json` β the list of 20 source url. ```json "candidates": [ "id": 1, "forum": "Reddit", "source_url": "https://w.reddit.com/...", "candidate_request": "string β the request candidate distilled from this forum", "used_in_final_request": true , "id": 2, "forum": "Quora", "source_url": null, "candidate_request": "string β or 'No qualifying candidate found.'", "used_in_final_request": false ] ``` ## Forbidden Paths and Actions - Do not read or rely on anything, except files from Read Context section. - Do not modify, rename, or delete the skill files or any path outside `OUTPUT_DIR`. - Do not name the skill inside the request. Do not quote skill-internal terms inside it. Do not ,β say anything like "follow the skill / guidelines". - Do not install dependencies or run destructive commands. E.2 Skill-driven task synthesis E.2.1 Artifact roles and task scenarios # Prompt template: produce a `files_to_change` change-surface analysis (hybrid) > How to use: send the whole **PROMPT** section below to an agent, after filling in three ,β placeholders: > - `SKILL` β the skill name (e.g. `37signals-rails`) > - `RUN_DIR` β absolute path to that skill's run directory (e.g. > `/Users/.../env_search_v0.3/runs/six-skill-e2e/SKILL`) > - `ENV_SEARCH_ROOT` β absolute path to the `env_search_v0.3` root (holds `scripts/` and ` ,β schemas/`) 29 > > The output must conform to the schema at: > `/Users/.../env_search_v0.3/schemas/search_env/file_change_summary/files_to_change.schema.json ,β ` > > This is a **hybrid** workflow. A deterministic script computes the *factual* layer > (provenance, caveats, duplicate/orphan/failure detection, codebase identity). You > only do the *judgment* layer (which code to change, how, tied to the skill's rules). --- ## PROMPT You're analyzing a task environment that was synthesized for a particular skill. Your job is to ,β produce a **`files_to_change` change-surface summary** as JSON: a record of **which files ,β the final task should have the agent edit, and how**, along with every data-quality problem ,β the environment carries. This is an **intermediate step** in data synthesis β not the ,β final task itself. ### How the synthesis pipeline works 1. Start from a skill, then plan a set of **resource requirements** (what files to go find). 2. Search the web β a **manifest**, where each requirement has one `selected` source plus a few ,β `alternatives`. 3. Download β a **download_report** recording what actually landed: `selected` vs some ` ,β alternative`, success/failure, per-member results for bundles. 4. The files end up under `resources/` β that directory *is* the agent's working tree. 5. The final task = this environment + a user request, engineered so the **skill is genuinely ,β needed**. β³ **Ignore the user request for now** β it gets composed later. You may glance at the request ,β candidates (`request_form.json` / `source_url.json`) as a *directional hint*, but don't ,β bake them in. Your call is driven by **what's in the environment + what the skill is built ,β to do**. ### Step 0 β Start from the deterministic prepass (already run for you) A deterministic prepass has **already been run** and written to `RUN_DIR/file_change_summary/files_to_change.prefill.json`. (To refresh it yourself: `python3 ENV_SEARCH_ROOT/scripts/search_env/file_change_summary/file_change_prepass.py --run- ,β dir RUN_DIR`.) It has **already reconciled** `manifest β download_report β resources/` for you and produced ,β the **factual layer**: - `codebase` (repo, commit, `extracted_root`, plus raw `repo_facts` β file counts, `test/` vs ` ,β spec/` counts, fixtures present, etc.) - `resource_roles[]` with `provenance` (selected vs alternative-N, download status, source) and ,β rule-based `caveats` - `context_inputs_read_only` and a top-level `provenance_and_caveats` summary - `machine_hints` β `scenario_seeds.review` (files touched by patches/PRs), `patch_vs_snapshot` ,β (SHA mismatches), and structured `duplicates` / `orphans` / `failures` / `misleading_names` - provisional `task_role`s (some may be `"undetermined"`) **Trust this output. Do NOT re-derive provenance/caveats by eyeballing the JSON.** The script is ,β exact; you are not. Carry `resource_roles`, `provenance`, `caveats`, ` ,β context_inputs_read_only`, and `provenance_and_caveats` through into your final output ** ,β verbatim** (you may append, but don't silently drop or contradict them). ### Step 1 β Read the skill's rule system - `RUN_DIR/skill_context.v0.3.json` embeds the full `SKILL.md`. **Internalize the rule system** ,β and note the rule IDs (e.g. `ctrl-crud-only`, `model-concerns`). Every change you propose ,β must cite the specific rules driving it. 30 - `RUN_DIR/resource_requirements.v0.3.json` β for each requirement read `purpose`, ` ,β detailed_description`, `expected_content`, `disallowed_content`, `priority`. This tells you ,β what each resource is *for* β which is exactly what you need to resolve the `"undetermined ,β "` roles. ### Step 2 β Read the real content to ground your conclusions (no guessing) Use `codebase.extracted_root` + `machine_hints` as your map, then open the actual files: - Repo shape: controllers/models/concerns, the test setup (Minitest or RSpec? fixtures?), ,β migrations, routes, schema, dependency manifest. `repo_facts` already hints at this β ,β confirm by reading. - Review artifacts (the `scenario_seeds.review` files; the patch / PR json / issue): what they ,β touch and what they're about. Heed `patch_vs_snapshot` β if a patch is already folded into ,β the snapshot, it's review context, not something you re-apply. > When there are many files, dispatch a read-only explorer subagent to surface relevant paths ,β and keep only the conclusions on the main thread. ### Step 3 β Resolve every `undetermined` task_role For each resource the script left `"undetermined"`, decide its role from the requirement's ` ,β purpose` + the skill's intent: is it a thing the agent **edits** (e.g. a source file in the ,β working tree), or **reads as reference/context** (e.g. a sample doc, a competitor page, a ,β spec the agent writes *against*)? A code-refactoring skill edits the repo; a document- ,β writing skill mostly reads references and **creates a new output file**. Set a concrete ` ,β task_role` and drop the `task_role_provisional` flag. ### Step 4 β Build `files_to_change` (the core judgment) For every candidate change write a concrete `how` grounded in the **actual file content** and ,β attach `skill_rules` (rule IDs) + a `confidence`. Target the spots that land squarely on ,β what the skill changes β e.g. custom member actions β CRUD resources; service objects β ,β rich models; RSpec β Minitest/fixtures; scattered logic β a concern; for a writing skill, ,β the new document to author and the references that constrain it. Since the request isn't fixed, split the surface into a few **self-consistent scenarios**, each ,β backed by a distinct cluster of resources (e.g. one feature thread stitched from issue + PR ,β + patch; one refactor thread pointed to by the requirements). List each scenario's ` ,β targets`. Every `path` is relative to `codebase.extracted_root`. **Edits belong on the repo ,β copy** β never on a standalone resource the script flagged as a duplicate; never on an ,β orphan-mismatch resource; never on a generated artifact (`db/schema.rb`, lockfiles) β those ,β are context only. ### Step 5 β Write the remaining judgment fields - `codebase.architecture_notes` (what's architecturally notable *for this skill*) and a top- ,β level `key_finding`. - `recommended_change_set`: a focused subset at the intersection of **grounded in the ,β environment Γ where the skill has real leverage Γ verifiable**, with `excluded_with_reason ,β `. - If the most self-consistent thread in the environment points in a *different direction* than ,β the skill's core strength (or than the request candidates), say so and leave the scenario ,β choice to the final request. ### Output - Write to: `RUN_DIR/file_change_summary/files_to_change.v0.3.json` (merge your judgment layer ,β into the prefill's factual layer). - **Follow the schema strictly**: `ENV_SEARCH_ROOT/schemas/search_env/file_change_summary/ ,β files_to_change.schema.json` (top-level required: `schema_version`/`skill`/`codebase`/`resource_roles`/`files_to_change`; ` ,β change_type` and `confidence` use the controlled enums; every `path` is relative to ` ,β codebase.extracted_root`). - Then validate (deliver only once it passes, exit code 0): ``` 31 python3 ENV_SEARCH_ROOT/scripts/search_env/file_change_summary/validate_files_to_change.py -- ,β file RUN_DIR/file_change_summary/files_to_change.v0.3.json --run-dir RUN_DIR ``` ### Definition of done 1. The prepass was run; its `resource_roles`/provenance/caveats/`context_inputs_read_only`/` ,β provenance_and_caveats` are carried through faithfully (every requirement still has an ,β entry). 2. Every `"undetermined"` task_role has been resolved; no `task_role_provisional` flags remain. 3. Every change_target has a real path relative to `extracted_root`, cites skill rules, and has ,β a concrete `how`. 4. There's a request-agnostic scenario grouping plus one focused recommended set. 5. The output JSON validates against the schema (validator exits 0). E.2.2 Instruction composition # Compose a User Request (Query) for a Downloaded Environment Write **one** realistic message a real engineer would send about this specific downloaded codebase β the kind of ask where doing it *well* genuinely needs a particular skill, yet the message reads like a normal practitioner's request, never like a skill manual or a solution ,β writeup. ## Who is writing A real engineer who **knew what they wanted, found a skill that fits, and read it once.** They remember, roughly, its steps/techniques and reach for those **names** in the field's own ,β vocabulary. They do **not** recall the skill's (cryptic) name, and they never paste definitions, procedures, ,β or rule numbers. They're describing a goal β not reciting a handbook, and not handing over the ,β answer. --- ## The two rules that make this work ### 1 β Speak in the skill's code words; hide that a manual exists The remembered phrase-names are the skill's **code words**. An agent that also read the skill recognizes them and does the task its way; one that didn't reads them as ordinary, slightly- ,β ambiguous language and drifts. That gap is the whole point. - **DO** use the skill's own short step/technique names in their natural-phrase form ("keep the controllers CRUD-only", "give me a verdict on whether it's ready to merge"). Use only the ones ,β this task needs. - **DON'T** explain or define them. Use the term and move on. - **DON'T** name the skill or hint one exists ("use the ... skill", "per the guide/checklist"). - **DON'T** use identifier/locator forms: rule-ID slugs, config tokens (`safe_auto`), scenario ,β keys, `task_role` labels, "Section 4 / Step 3 / Rule R6.12". Say them in plain words. - **Test:** would a practitioner who never read *this* skill find the phrase ordinary (maybe a ,β touch ambiguous) rather than gibberish or a cipher? If yes, it's a good code word. ### 2 β State the GOAL, never the SOLUTION 32 The user describes the **problem and what they want** β not how to fix it. They are *asking for* ,β the change; they do not already hold its answer. - **DO** give the symptom / goal in the user's words: what's wrong or wanted, in which area, ,β grounded in real files (e.g. "bad URLs leak a raw urllib3 error out of `requests.get` instead of a requests-native one β I want it to behave like the other invalid-URL cases"). - **DON'T** prescribe the implementation: no exact edit location, no "match the neighboring ,β branch / existing style", no exact test name / parametrization / input construct, no step-by-step of ,β the change. The `how` of the fix is the agent's job to discover. - **NEVER reference anything under "Synthesis-only inputs"** in the requirement list (the ,β upstream PR/patch). Don't name it, cite its number, link it, or hint it exists β it is the answer, and ,β it is withheld from the agent's environment. --- ## What to produce - **Grounded in this environment.** Refer to real files, paths, components, an issue number β ,β taken from the requirement list β so the task can only be done in *this* codebase. - **Cover every test-time requirement β no loss.** Touch **every** resource under **"Available resources (usability)"** whose health is `usable`/`degraded`: name the edit targets, reference ,β the context resources (issue, docs, config, tests) concretely. A few related asks in one message ,β are fine; weave them into one real person's message β not a bulleted spec. **Never** ask to edit ,β an `unusable`/orphan resource, and **never** mention a synthesis-only one. - **Genuinely skill-requiring**, signaled through the code words β expressed as the user's goal. - **A real user's voice.** First person, concrete. As long as it needs to be, never a checklist. - **One-shot ask β no reviewer in the loop.** The user states the task and expects it carried ,β out; there is no human to check in with mid-task. Don't make the request hinge on the agent sharing ,β a plan, pausing for approval, or confirming before it proceeds (no "show me your plan first", " ,β let me review before you implement", "run it by me before you change X") β ask directly for the ,β outcome. ## Inputs β read from disk (read ONLY these) - **THE SKILL** β mine it for code-word vocabulary; learn its name/tokens so you can keep them ,β out: SKILL_READING_INSTRUCTIONS - **`./requirement_list.md`** β the substance: the task, and which resources are test-time vs **synthesis-only** (withheld). Your source of truth for what to ask for. - **`./fewshot_examples.md`** β real forum requests; model their **voice and shape** only, not ,β topics. If empty, just write in a natural human voice. Do **not** open `../resources/`, and do not read other bookkeeping files in this directory. ## Output Write JSON to `./query.json` (this turn, nothing else): 33 ```json "general_request": "<one short line: the gist of what the user wants>", "user_request": "<THE FINAL QUERY β human-voiced, grounded, covers every test-time requirement, ,β speaks in code words, states the goal not the solution. This is the artifact.>", "grounded_in": ["<every repo path / component / behavior the request targets>"], "coverage": "test_time_requirements": ["<every requirement id under Available resources (usable/degraded) ,β >"], "covered": ["<the ids actually reflected in user_request β aim to equal ,β test_time_requirements>"], "omitted": ["id": "<id>", "reason": "<why β should be rare>"] , "skill_applicability": "can_apply": true, "reason": "<one sentence: why doing this well needs ,β the skill>", "skill_signal_check": "names_skill": false, "code_words_used": ["<natural-phrase skill terms ,β used>"], "tokens_avoided": ["<rule ids / locators kept out>"], "answer_leak_check": "states_goal_not_solution": true, "prescribes_no_fix_steps": true, " ,β names_synthesis_only_resource": false ``` Before writing, re-read `user_request` once and confirm: (a) every test-time requirement is ,β covered; (b) the skill's name and all tokens/locators are absent; (c) code words appear **without ,β explanation**; (d) it states the **goal**, prescribes **no** fix steps, and references **no** synthesis-only ,β resource; (e) it reads like a real practitioner, not a manual; (f) it is a single-shot ask with **no** mid- ,β task review/approval gate (no "show me your plan / let me review before you proceed"). E.3 Packaging and execution validation E.3.1 Environment classification You are classifying the Docker BUILD difficulty of one benchmark task into A / B / C / D. Output ONLY the JSON object required by the output schema β no prose. ## Base image already provides (do NOT count these as extra installs) BASE_IMAGE_INVENTORY ## Classes - **A** β base image suffices. Pure read / write / analysis, or runs code only on base runtimes. Source files present but only *read* β still A. setup.sh stays a no-op. - **B** β base + build-time package installs ONLY: `pip install ...` and/or `npm install` in an extracted repo. No new language/system toolchain. - **C** β needs a language toolchain NOT in base, installed at build time: Go, Rust, JDK+Maven/Gradle, .NET, Dart, Node 22 / pnpm, azure-cli, or a non-3.12 Python. Usually also needs offline dependency pre-warming (go mod download / cargo fetch / dotnet restore / mvn verify / pnpm install). - **D** β infeasible on Linux: imports Apple/Xcode frameworks (UIKit, SwiftUI, MapKit, Kingfisher, etc.), OR requires a persistent service at *verify* time (a live Postgres / MySQL / Redis server β `docker-compose.yml` with a DB service the task actually uses is the classic signal). A build-time `RUN` layer cannot keep a service running. ## How to decide Classify by what the task in `instruction.md` must actually **run or build**, NOT by which 34 files merely exist. Read the manifests in the env summary (`go.mod` `toolchain`, `package.json` `engines`/`packageManager`, `Cargo.toml`, `pom.xml`, `*.csproj`, `pubspec.yaml`) β they reveal the real toolchain. When class==D, fill `d_reason`. `needs_runtime_internet`: set true if the agent will run a build/make target or script that FETCHES at runtime and cannot be fully pre-warmed at build β inspect `Makefile`/`mvnw`/`package. ,β json` scripts for commands like `go get ...@latest`, `npm install`/`npx` of un-pinned deps, `mvn ... - ,β U`, or `*-install @latest`. If everything can be baked into the image at build (go mod download / ,β cargo fetch / dotnet restore / pinned pnpm install), keep it false. ## Task instruction (what the agent is asked to do) INSTRUCTION ## Environment summary (file tree + manifests) ENV_SUMMARY ## Skill (context only β NOT present in the build environment) SKILL_EXCERPT Now emit the classification JSON. E.3.2 Setup generation Write `environment/setup.sh` for this Harbor task so its Docker image builds with everything baked in for an OFFLINE runtime. Your cwd is the task directory; edit files directly. `setup.sh` runs at `docker build` time: root, cwd `/app`, network UP. `/app` already contains the task's env files (`COPY files /app/`). At agent runtime the container may be OFFLINE. ## Rules - Start with `#!/bin/bash`, then `set -euo pipefail`, then `cd /app`. - Install ONLY what is missing. The base image already has (do NOT reinstall): BASE_IMAGE_INVENTORY - Install EVERYTHING the agent needs at runtime HERE (runtime may be offline): toolchains AND their dependency caches. Pre-warm: `go mod download` / `cargo fetch` / `dotnet restore` / `mvn -B clean verify -DskipTests` / `pnpm install --frozen-lockfile` / `dart pub get`. - Be architecture-adaptive: detect `$(dpkg --print-architecture)` or `$(uname -m)` (arm64 vs ,β amd64). - Go: also `export GOTOOLCHAIN=local` and `go env -w GOTOOLCHAIN=local` so offline runs never auto-download a toolchain for the `go.mod` `toolchain` directive. - `cd /app/extracted/<repo>-*/` globs must match exactly one directory. - No long-running services (build layers don't keep processes). Skill-agnostic: NO skill files exist at build time. Make optional/best-effort deps tolerant: `... || true`. - If (and only if) the classification says `needs_runtime_internet: true`, set `allow_internet = true` in `task.toml` `[environment]`. - For heavy toolchains, raise `task.toml` `[environment]` `build_timeout_sec` and `storage_mb` if needed (Rust target ~2-3 GB, Maven ~/.m2 ~2 GB, pnpm node_modules ~1.5 GB). - The script must `exit 0` on success. ## Classification (the target class + detected toolchain/packages) CLASSIFICATION ## Task instruction INSTRUCTION 35 ## Environment summary (file tree + manifests) ENV_SUMMARY ## Gold examples (battle-tested setup.sh from shipped tasks β adapt, don't copy blindly) EXAMPLES Now write `environment/setup.sh` (overwrite the no-op), and edit `task.toml` if the rules above require it. Keep the script tight and well-commented. E.4 Contrastive rubric construction E.4.1 Rubric generation # Generate a contrastive skill rubric (schema v2) Produce **one** schema-valid contrastive skill rubric: an LLM-as-judge rubric set distilled from ,β a frontier **oracle**'s no-skill vs with-skill trajectory on the *same* task. The behavioral ** ,β DELTA** between the two arms is *what the skill adds* and becomes the criteria. A judge later applies it ,β to a **weaker** model's two arms; the with-minus-no score delta estimates skill uptake. **Ground truth is the behavioral DELTA, never the oracle trajectory.** Oracle-specific files/ ,β lines/ commands are *illustrative only*. A weaker no-skill run that scores **0 on every `skill_signal` criterion is correct by design** β internalize this; it is the #1 failure mode. This run's facts (substituted by the launcher): - set_id `SET_ID` Β· task_id `TASK_ID` Β· output file `OUTPUT_FILENAME` - oracle (the contrasted trajectories): `ORACLE_MODEL` - generator (you, for `generation_provenance.generator_model`): `GENERATOR_MODEL` - single-arm mode: `SINGLE_ARM` (when `true`, `./with.compact.json` is MISSING) DIMENSION_FOCUS ## Inputs in your working directory (read from disk) - `./no.compact.json`, `./with.compact.json` β oracle no-skill / with-skill trajectories, ,β compacted. Events are `i, t, [s], x, [o]`: `i`=stable index (cite it; align arms by it), `t`β `user|msg|read|search|exec|edit|list|other`, `s`=`fail|canc`, `x`=text/command, `o`=output. `./with.compact.json` is absent in single-arm mode β check. - `./workspace/no/`, `./workspace/with/` β each arm's FINAL deliverable files (+ `MANIFEST.txt`, ,β the full workspace tree). **As authoritative as the trajectory**: edit-bodies in the trajectory ,β are capped at 900 chars, so verify what was actually produced β completeness, structure, ,β correctness β from these files, not from trajectory narration. - `./skill/SKILL.md` (+ `references/`) β mine for `named_phases`, `mandated_outputs`, vocabulary, ,β hard-rules vs principles. - `./inputs_meta.json` β `set_id`, `task`, `skill` shell, `oracle`, `source_run_pair`, `generated_at`, `single_arm` to copy verbatim (fill `task.one_line_goal`/`domain`/ `answer_leak_risk` and `skill.name`/`named_phases`/`mandated_outputs` from SKILL.md + ,β trajectories). - `./contrastive_rubric.schema.json` β the **v2** schema you must validate against ( ,β authoritative field spec; read its one-line descriptions). 36 - `./fp-check.rubric.json` β a **v2 worked example**. Study its shape & rigor; do NOT copy its security-domain criteria unless this task genuinely is fp-check. - `./validate_rubric.py` β run it in a loop until VALID. ## Method (in order) 1. **Mine the skill.** Extract `named_phases` (ordered steps), `mandated_outputs`, vocabulary, ,β and which rules are hard vs principles. 2. **Read both arms β trajectory AND workspace.** Align by event index `i`; describe each arm's *behavior* (not files/lines): phases run, what was verified empirically, output format, over/under-reach. Then OPEN each arm's `./workspace/` deliverables to judge the actual ,β artifact (existence, completeness, structure) β the file is the truth where the trajectory was capped. 3. **Enumerate candidate behaviors and classify each HONESTLY** (the schema enforces this): - `skill_signal` β with-skill does it (or better), no-skill doesn't/weaker β `discrimination.with_skill_rank > no_skill_rank`, `arm_separation:"separated"`. **Only ,β these feed the skill delta.** - `shared_baseline` β **both** arms do it β ranks **equal**. Scored for calibration, ,β excluded from the delta. **Do not inflate a shared baseline into a skill_signal** β the cardinal sin. - `anti_pattern` β a no-skill failure mode the skill suppresses β `no_skill_rank β₯ ,β with_skill_rank`, `anti_pattern_polarity:"penalize_when_met"`, and encode it higher=better (`unmet_value:1`). - Discard no-signal behaviors. At the frontier tier expect MANY `shared_baseline`; the ,β measurable skill delta is usually a small set (durable artifact, empirical failβfix, output ,β discipline, phase adherence). The worked example honestly has ~4 skill_signal of ~9. 4. **Write each criterion** (per the schema `Criterion`). Hard constraints: - `statement`: starts `The agent`, ONE behavior, β€30 words / β€400 chars, **no oracle ,β specifics** (no backticks, file paths, source extensions, `line N`/`:N`, `Response-N` β a regex rejects ,β them). - Concreteness lives ONLY in `contrast_provenance` (`with_skill_behavior`/`no_skill_behavior` behavioral prose + `anchors[]` with `binding:"illustrative_only"`); set `hypothesized` ( ,β true in single-arm). - `judge_guidance`: `what_to_look_for` (generalizable), `positive_evidence[]`, ` ,β negative_anchor` (the no-skill/anti-pattern picture in **candidate** terms), `common_false_positives[]`. - `scoring`: always a `binary` block; add a `graded` ladder (`mode:"both"`, contiguous `0.. ,β max`, descriptors in candidate terms, one `is_negative_anchor`) for high-value `skill_signal`. - `discrimination`: integer ranks consistent with `criterion_type`; `arm_separation`. - `category_id` β a `category_taxonomy` entry you declare; `weight` (0..10); `skill_attribution_confidence` (0..1); optional `category_tag`. 5. **Pick exactly ONE `gatekeeper`** β the core correctness check (often the verdict/answer); ,β usually a `shared_baseline` (a frontier no-skill arm also gets it right), which is fine. 6. **Fill the set-level metadata β the v2 keys ONLY:** `schema_version` (` ,β contrastive_skill_rubric.v2`), `set_id`/`task`/`skill`/`oracle`/`source_run_pair` from `inputs_meta.json`, `contrast_summary ,β ` (1β2 behavioral sentences β€600 chars), `category_taxonomy` (only categories you used, with `category_weight`), `generation_provenance` (`generator_model:"GENERATOR_MODEL"`, `generator_prompt_id:"generate_rubric.md"`, `generated_at` from inputs_meta, `method`, `mandatory_abstraction_applied:true`), `set_confidence`, `criteria`. > **Do NOT emit** `judge_protocol`, `judge_output_schema`, `aggregation`, or 37 > `trajectory_truth_guard` β in v2 those are harness-owned (`judge_policy.json`). The schema > rejects them as `additionalProperties`. ## Honesty rails - **Single-arm** (`SINGLE_ARM==true`): `set_confidence.single_arm=true`, `source_run_pair.with_skill_run=null`, `method:"single_arm_inferred"`, every criterion `hypothesized=true` + `discrimination.arm_separation:"not_measured"`, `confidenceβ€0.5`. Infer ,β from no-skill + skill spec; do not fabricate measured separation. - **Empty delta** (arms behaviorally identical): `empty_delta=true`, only `shared_baseline`, `β€ ,β 0.5`. - **Harness reward is a placeholder** β `task.harness_reward_is_trustworthy=false`; judge by ,β inspecting trajectories. - **Answer leakage** β if the query telegraphs the answer or a withheld patch could leak, set `task.answer_leak_risk` and lower confidence (heavy β `β€0.6`); record in `set_confidence. ,β caveats`. ## Output contract + self-validate loop (mandatory) Prefer writing a small Python builder (construct the dict β `json.dump`) over hand-typing JSON. ,β Write `OUTPUT_FILENAME`, then run `python3 validate_rubric.py OUTPUT_FILENAME`, **fix every ,β error and re-run until `VALID ... (0 errors)`.** (If `jsonschema` is missing, `pip install jsonschema ,β >=4.0`.) Write no other file except `OUTPUT_FILENAME` + optional builder/scratch; do not modify the ,β schema, worked example, compacted inputs, or validator. Final message (read programmatically): the ,β output path, criteria counts by `criterion_type`, and the gatekeeper id. 38