Paper deep dive
Spec Kit Agents: Context-Grounded Agentic Workflows
Pardis Taghavi, Santosh Bhavani
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 3:11:05 AM
Summary
Spec Kit Agents is a multi-agent, spec-driven development (SDD) framework that addresses 'context blindness' in AI coding agents by integrating phase-level discovery and validation hooks. By grounding intermediate artifacts (SPEC, PLAN, TASKS) in repository evidence and validating them against the environment, the system improves code quality and reliability in complex software engineering tasks, achieving a 58.2% Pass@1 on SWE-bench Lite.
Entities (4)
Relation Signals (2)
Spec Kit Agents â improvesperformanceon â SWE-bench Lite
confidence 100% ¡ We further evaluate the framework on SWE-bench Lite, where augmentation hooks improve baseline by 1.7 percent, achieving 58.2 percent Pass@1.
Spec Kit Agents â utilizesmodel â MiniMax M2.5
confidence 100% ¡ The agentic workflow (PM and developer agents) is executed through Claude Code CLI, routed to an Anthropic-compatible endpoint backed by MiniMax-M2.5.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Spec-driven development (SDD) with AI coding agents provides a structured workflow, but agents often remain "context blind" in large, evolving repositories, leading to hallucinated APIs and architectural violations. We present Spec Kit Agents, a multi-agent SDD pipeline (with PM and developer roles) that adds phase-level, context-grounding hooks. Read-only probing hooks ground each stage (Specify, Plan, Tasks, Implement) in repository evidence, while validation hooks check intermediate artifacts against the environment. We evaluate 128 runs covering 32 features across five repositories. Context-grounding hooks improve judged quality by +0.15 on a 1-5 composite LLM-as-judge score (+3.0 percent of the full score; Wilcoxon signed-rank, p < 0.05) while maintaining 99.7-100 percent repository-level test compatibility. We further evaluate the framework on SWE-bench Lite, where augmentation hooks improve baseline by 1.7 percent, achieving 58.2 percent Pass@1.
Tags
Links
- Source: https://arxiv.org/abs/2604.05278v1
- Canonical: https://arxiv.org/abs/2604.05278v1
Trouble viewing inline? Open PDF directly â
Full Text
35,962 characters extracted from source content.
Expand or collapse full text
Spec Kit Agents: Context-Grounded Agentic Workflows Pardis Taghavi, Santosh Bhavani Figure 1: Overview of the Spec Kit Agents workflow. Abstract Spec-driven development (SDD) with AI coding agents provides a structured workflow, but agents often remain âcontext blindâ in large, evolving repositories, leading to hallucinated APIs and ar- chitectural violations. We present Spec Kit Agents a multi-agent SDD pipeline (with PM and developer roles) that adds phase-level, context-grounding hooks. Read-only probing hooks ground each stage (Specify, Plan, Tasks, Implement) in repository evidence, while validation hooks check intermediate artifacts against the environ- ment. We evaluate 128 runs covering 32 features across five reposi- tories. Context-grounding hooks improve judged quality by +0.15 on a 1â5 composite LLM-as-judge score. (+3.0% of the full score; Wilcoxon signed-rank,í<0.05) while maintaining 99.7â100% repository-level test compatibility. We further evaluate the frame- work on SWE-bench Lite, where augmentation hooks improve baseline by 1.7%, achieving 58.2% Pass@1. CCS Concepts ⢠Computing methodologiesâIntelligent agents; Multi-agent systems;⢠Software and its engineeringâSoftware verification and validation. Keywords LLM agents, agentic workflows, multi-agent systems, tool-augmented grounding, tool-based validation, spec-driven development , 1 Introduction Large language models (LLMs) have made it practical to automate substantial portions of software development, but end-to-end fea- ture delivery in real repositories remains brittle. Modern coding assistants are effective at local edits, yet multi-step tasks in evolv- ing codebases frequently fail for reasons including missing context about the current architecture, stale assumptions about dependen- cies, and mismatches with repository conventions. These failures tend to compound across stages such as planning, task decompo- sition, and implementation leading to wasted iterations and unre- liable outcomes. Spec-driven development (SDD) is a promising response to this brittleness. Rather than asking an agent to generate code immediately, SDD externalizes intermediate artifacts (e.g., a specification, an implementation plan, and a task checklist) that make intent explicit and provide a structured audit trail. GitHubâs Spec Kit [9] operationalizes this idea as a staged workflow (Specify âPlanâTasksâImplement), optionally gated by plan review. In principle, this âreasoning before codingâ structure should improve reliability and debuggability. In practice, however, structured workflows do not eliminate a core failure mode we refer to as context blindness: the agentâs intermediate artifacts can be internally coherent while being in- compatible with the repository as it exists. Common symptoms include referencing non-existent APIs, proposing file paths that do not exist, and violating local architectural or stylistic conventions. When these errors are discovered late during implementation or test execution the agent often backtracks, revises earlier artifacts, or introduces additional inconsistencies. We present Spec Kit Agents, arXiv:2604.05278v1 [cs.SE] 7 Apr 2026 , ,Pardis Taghavi, Santosh Bhavani an orchestrated multi-agent SDD pipeline that addresses context blindness by making grounding and validation explicit workflow operations. Spec Kit Agents augments the Spec Kit stages with a context-grounding layer: (i) discovery hooks that perform read-only probing before each stage to collect repository evidence (relevant files, conventions, dependencies, history), and (i) validation hooks that check intermediate artifacts and, after implementation, execute project checks (e.g., tests and linters) when applicable. This design keeps grounding and validation outside the core agent prompts, enabling auditable traces and selective tool access. Contributions. ⢠System. Spec Kit Agents, a multi-agent SDD pipeline (state- machine orchestrator + PM and developer roles) with a context-grounding layer that runs pre-phase discovery and post-phase validation hooks. â˘Context-grounding design. A phase scoped grounding and validation interface that operates over explicit artifacts (SPEC/ PLAN/ TASKS), enabling transparent auditing and least privilege tool access. â˘Evaluation. An empirical study over 128 experimental runs covering 32 unique feature tasks across 5 repositories, report- ing judged quality, latency, and repository-level test compat- ibility. We also report controlled comparisons of Baseline, Augmented, Full, and Full-Augmented configurations, to- gether with Discovery-only and Validation-only ablations, and evaluate generalization on SWE-bench Lite. Across 128 task instances (32 unique feature tasks across 5 open- source repositories), Spec Kit Agents yields a consistent improve- ment in judged quality (+0.15 on a 1â5 composite LLM-as-judge score) while maintaining high test pass rates (99.7â100%). These gains come with additional overhead in the full workflow family due to extra phases and context-grounding execution; accordingly, we interpret latency within each budget family rather than across families. Overall, the primary benefit is not a dramatic jump in av- erage score, but earlier detection and prevention of compounding context errors in multi-step agentic workflows. 2 Related Work 2.1 Multi-agent orchestration and agentic workflows. Recent LLM-agent research has moved from single-model prompt- ing to agentic workflows that decompose tasks into structured stages and often assign specialized roles across agents [11,30]. Common workflow primitives include closed-loop reasoning with tool use (e.g., ReAct) [36] and search over intermediate reasoning states to improve planning and execution [35]. Multi-agent frameworks such as AutoGen, CAMEL, and MetaGPT, along with newer or- chestration systems, emphasize role specialization, coordination, and interaction protocols [6,8,12,17,28,32,37]. Benchmarks like- wise show that orchestration design materially affects agent perfor- mance across tasks [5,19]. Where prior work primarily emphasizes orchestration and collaboration among agents, our work targets workflow reliability under context limitations. Spec Kit Agents adds a context-grounding layer that performs phase-level grounding and validation outside the core agent prompts. 2.2 Tool-augmented grounding for agents. A challenge in agentic systems is grounding decisions in exter- nal evidence rather than relying on parametric memory. Retrieval- augmented generation improves factuality and supports knowledge- intensive tasks by conditioning outputs on retrieved documents [3,16], while browser- and tool-augmented systems show that allowing agents to query external sources and cite evidence can improve task success [21,24]. More broadly, tool use has been studied through modular routing to external tools or experts (e.g., MRKL-style systems) [15] and through learned tool-use behaviors acquired via training or self-supervision [18,23,25,26]. In software engineering, agents extend these ideas to repository-level ground- ing through file search, code navigation, executable actions, and exploration [31,33,34,38]. Most prior approaches treat grounding as an in-trajectory behavior of the same agent that plans and gen- erates, making it sensitive to prompt design and context-window noise. In contrast, Spec Kit Agents makes grounding an explicit workflow primitive: read-only discovery hooks probe repository state before each phase, and validation hooks check intermediate artifacts against executable signals. This shifts grounding from best-effort retrieval to phase-scoped evidence collection, making it more repeatable, inspectable, and less coupled to the main agentâs generation. 2.3 Verification, context-grounding, and tool-based validation Reliability work on LLM agents includes self-critique and itera- tive refinement methods that use feedback to improve later at- tempts [7,10,14,20,27], as well as rule-based constraint approaches that steer behavior through explicit principles [4,29]. Many agent pipelines also rely on tool-based validation signals such as tests, linters, and structured checks, especially in repository-level tasks where executable feedback provides a strong correctness signal [13,34]. Benchmarks further show that verification and feedback design materially affect end-to-end reliability [19]. Our contribu- tion differs in both when and what we validate. Rather than concen- trating verification after implementation, we validate intermediate artifacts (SPEC/PLAN/TASKS) before code generation, catching hal- lucinated APIs, invalid paths, and architectural mismatches early while retaining post-implementation executable checks as a final gate. More broadly, we treat tool-based validation not as a single end stage filter, but as repeated phase-specific context-grounding hooks that reduce compounding errors across agentic workflows. 3 Method We present Spec Kit Agents, including its orchestration logic, tool interfaces, and context-grounding mechanisms. We also describe the execution and evaluation protocol used in our experiments. 3.1 System Overview and Workflow Spec Kit Agents is a multi-agent system for feature delivery in existing repositories. The system consists of (i) an orchestrator im- plemented as a state machine, (i) a product manager (PM) agent responsible for clarifying requirements and prioritization, and (i) a developer agent responsible for producing intermediate artifacts and implementing code changes. Agents communicate through a Spec Kit Agents: Context-Grounded Agentic Workflows, , centralized messaging platform, which also supports human inter- vention at defined checkpoints (e.g., plan approval). The developer agent follows the Spec Kit workflow to generate intermediate arti- facts and then implement the feature. In the Full workflow variants, the developer agent produces three intermediate artifacts before implementation:SPEC.md(requirements and acceptance criteria), PLAN.md(an implementation plan with file-level touchpoints), and TASKS.md(an executable checklist). The implementation stage then executes the plan and opens a pull request in the target repository. In Baseline variants, the agent skips all intermediate artifacts and proceeds directly to implementation. 3.2 Context-Grounded Agentic Workflows Layer We introduce a context-grounding layer that provides phase-scoped grounding and validation for the developer agent. The context- grounding hooks are invoked at workflow boundaries and operate over explicit artifacts (e.g.,SPEC.md,PLAN.md, andTASKS.md) rather than being embedded inside the developerâs main prompt. Discovery hooks (pre-phase grounding). Before each phase, a read-only prober gathers evidence about the codebase using repos- itory inspection tools (e.g., globbing, grep, andgithistory). The goal is to surface project-specific conventions, existing APIs, and relevant modules so that subsequent generation is conditioned on concrete, localized context rather than generic priors. For example, for persistence-related features, discovery can identify existing log- ging formats or storage abstractions and steer the agent away from introducing unsupported dependencies. Validation hooks (post-phase checks). After each phase, a validator checks the generated artifact for internal consistency and repository compatibility. For earlier artifacts, validation focuses on structural and referential constraints (e.g., whether file paths referenced inPLAN.mdexist, whether required libraries are present, and whether the task list is feasible and properly ordered). After implementation, validation executes repository checks (e.g., unit tests and linters) to detect regressions. This design front-loads error detection by catching hallucinated paths, missing dependencies, or infeasible plans before code generation compounds mistakes. Tool access control. The context-grounding hooks validate each phase by probing the codebase before and after reasoning steps, ensuring specifications are grounded in existing project con- ventions and plans are verified against installed dependencies. The PM agent is restricted to repository analysis and version-control inspection. The developer agent is permitted to edit files and run repository commands required to implement features. Discovery hooks are read-only, while validation hooks extend discovery per- missions with execution privileges for project checks (e.g.,pytest, ruff, and JavaScript test runners) when applicable. 3.3 Models, Tools, and Execution Environment Spec Kit Agents separates generation from evaluation. The agentic workflow (PM and developer agents) is executed through Claude Code CLI, routed to an Anthropic-compatible endpoint backed by MiniMax-M2.5. Using a single execution interface ensures consis- tent tool invocation, logging, and run control across all experiments. Quality is evaluated independently using Claude Opus 4.6 as an LLM-as-judge. Outputs are scored on a 1â5 scale along four di- mensions: completeness, correctness, style, and maintainability, and the composite score is their mean. This separation reduces self- evaluation bias by isolating scoring from the agentâs prompts and tool access. We additionally conduct a small blinded human review on a subset of outputs using the same rubric. We log prompts, tool calls, intermediate artifacts, and execution traces for each run. Rate- limited runs are excluded from latency analyses but retained for quality reporting when a pull request artifact is available; comple- tion rates include such runs. 3.4 Experimental Protocol and Configurations Configurations. We evaluate four primary configurations: (i) Base- line, which skips intermediate artifacts and proceeds directly to implementation; (i) Augmented, which follows the same direct- to-implementation flow and adds discovery and validation hooks; (i) Full, which executes the full Spec Kit workflow; and (iv) Full- Augmented, which adds discovery and validation hooks to Full. To isolate context-grounding effects, we also evaluate Discovery- only (pre-phase hooks only) and Validation-only (post-phase hooks only) ablations. Budgets and timeouts. Each phase is subject to bounded timeouts to control end-to-end runtime. Human-facing checkpoints for plan-review are auto approved. End-to-end, Base- line and Augmented runs use a 40-minute budget, while Full and Full-Augmented runs use a 90-minute budget. Runs exceeding these limits are terminated and marked as failures. Success criteria. A run is considered successful if it produces a pull request in the target repository, includes at least one file modification, and completes without critical execution errors (e.g., authentication failures or tool-permission violations). Quality is assessed post hoc using the judge model; in analysis, composite scores below 3.0 are treated as requiring manual review. 4 Experiments 4.1 Evaluation Setup We evaluate Spec Kit Agents on 32 feature tasks across five repos- itories: FastAPI, Airflow, Dexter, Plausible Analytics, and Strapi. Each task is run under four configurations: Baseline, Augmented, Full, and Full-Augmented. The task set spans multiple change types, including API additions, configuration changes, new modules, refac- tors, and test updates; Appendix A lists a subset of tasks and cate- gories. FastAPI and Airflow are Python repositories evaluated with pytest -q; Dexter and Strapi are TypeScript repositories; Plau- sible Analytics is primarily Elixir with supporting JavaScript. For each task, the agent receives a natural-language feature request and executes the assigned workflow end-to-end, producing a pull request when successful. Our primary outcome is judged quality, measured by an independent LLM-as-judge (Claude Opus 4.6) using a 1â5 composite score. We also report wall-clock completion time, test-suite compatibility based on post-change repository test execu- tion, and failure category for unsuccessful runs. Generalization to SWE-bench Lite is evaluated separately. For statistical comparisons, we treat each feature task as a paired subject across conditions and use the Wilcoxon signed-rank test for paired analyses of judged quality and wall-clock completion time. , ,Pardis Taghavi, Santosh Bhavani Table 1: Quality scores. Shaded cells indicate the best result within each workflow family. ConditionOver. F-API Airf. Dext. Plau. Strap. Baseline3.463.213.753.653.303.25 Augmented3.503.583.563.313.453.55 Full3.513.103.353.903.483.61 Full-Augmented3.663.523.444.003.643.69 Table 2: Blinded human preference on paired pull-request comparisons. Comp.Tasks Votes Full Tie Full-Aug. Full vs. Full-Aug.66019338 4.2 Quality Results Table 1 reports judged quality, with the overall score computed as a feature-count-weighted average across repositories. In the 40-minute workflow family (Baseline, Augmented), the developer agent skips intermediate artifacts (SPEC.md/PLAN.md/TASKS.md) and proceeds directly to implementation; in the 90-minute family (Full, Full-Augmented), those artifacts are produced before coding. Within the 90-minute workflow family, Full-Augmented achieves the strongest overall quality, improving from 3.51 to 3.66 (+0.15) rel- ative to Full. On the paired subset of completed tasks, this difference is statistically significant (Wilcoxon signed-rank,í<0.05). Gains appear across repositories, with especially strong improvements on FastAPI and Plausible. To complement the LLM-based evaluation, we also conduct a blinded human preference study on paired tasks completed successfully under both Full and Full-Augmented. Evalua- tors compare anonymized pull requests shown in random order and may select either version or a tie. Table 2 summarizes the resulting pairwise judgments. Repository-level test-suite compatibility re- mains high across configurations, indicating that the quality gains do not come at the expense of breaking existing project behavior. 4.3 Ablation Results We ablate context-grounding components by enabling only pre- phase discovery or only post-phase validation. Table 3 reports the re- sulting quality and runtime relative to the Full baseline (3.51). Both partial variants improve over Full, with Validation-only outperform- ing Discovery-only. The combined design achieves the strongest result, suggesting that the two components are complementary. 4.4 Latency Results We report wall-clock completion time on completed runs only. Base- line and Augmented use a 40-minute budget, whereas Full and Full- Augmented use a 90-minute budget, latency is compared only within each budget family. Context-grounding hooks add only modest overhead in the 40-minute family, but a larger cost in the 90-minute family due to the longer workflow and repeated hook execution. We therefore view latency as a qualityâruntime trade-off. Table 3: Ablation of phase-level context-grounding compo- nents relative to the Full baseline (3.51). ConditionQual.Î%TimeDescription Discovery-only3.53+0.57%25.5 minPre-phase grounding Validation-only3.57+1.71%31.2 minPost-phase checks Full-Augmented 3.66 +4.27%37.2 minBoth hooks enabled Table 4: Within-family latency comparisons (completed runs only). ComparisonA (min) B (min)Î (min)í pairs Baseline vs. Augmented14.415.5+1.115 Full vs. Full-Augmented24.037.2+13.216 Table 5: Comparative analysis on SWE-bench Lite. Spec Kit Agents shows competitive performance using MiniMax-M2.5. FrameworkPrimary LLMPass@1 Aider [2]GPT-4o & Claude 3 Opus 26.33 Moatless Tools [22]Claude 3.5 Sonnet38.00 OpenHands [31]CodeAct v2.141.67 DARS Agent [1]Claude 3.5 Sonnet + DeepSeek R1 47.00 SWE-Agent [34]Claude 4 Sonnet56.67 Spec Kit Agents (Ours, Baseline)MiniMax-M2.556.5 Spec Kit Agents (Ours, Augmented)MiniMax-M2.558.2 4.5 SWE-bench Lite Results To assess generalization beyond our custom repository tasks, we evaluate Spec Kit Agents on SWE-bench Lite, a standard bench- mark of 300 real-world software engineering issues. Table 5 com- pares our framework against prior SOTA. Spec Kit Agents achieves a 56.5% pass rate in the baseline configuration and 58.2% with context-grounding hooks enabled. All experiments in this paper use MiniMax-M2.5 as the base model, however, the proposed or- chestration framework is model-agnostic and readily generalizes to other API-accessible models.Additional implementation and failure- mode details are provided in Appendices B and C. 5 Conclusion We presented Spec Kit Agents, a multi-agent, spec-driven de- velopment workflow that augments Spec Kit with phase-scoped discovery and validation context-grounding hooks. Across 128 runs covering 32 features, the context-grounded full workflow achieves the strongest overall quality, indicating that explicit repository- grounded orchestration improves reliability. The gains are consis- tent and stem from stronger alignment between the specification, the discovered repository context, and the final implementation. This improvement comes with additional runtime overhead, making the approach most appropriate for higher-risk or high complex- ity tasks. Overall, the results support explicit context-grounded orchestration as a practical design principle for more dependable autonomous software engineering. Spec Kit Agents: Context-Grounded Agentic Workflows, , References [1]Vaibhav Aggarwal, Ojasv Kamal, Abhinav Japesh, Zhijing Jin, and Bernhard SchĂślkopf. 2025. DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). [2]Aider. 2024. How aider scored SOTA 26.3% on SWE Bench Lite. https://aider. chat/2024/05/22/swe-bench-lite.html. [3]Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations. [4]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 (2022). [5]Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information pro- cessing systems 37 (2024), 74325â74362. [6]Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al.2023. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations. [7]Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023). [8] Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, et al. 2025. Multi-agent collaboration via evolving orchestration. arXiv preprint arXiv:2505.19591 (2025). [9] GitHub. 2026. Spec-Driven Development with Spec Kit. https://github.com/ github/spec-kit/blob/main/spec-driven.md Accessed March 12, 2026. [10] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738 (2023). [11]Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024). [12]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. 2023. MetaGPT: Meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations. [13] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023). [14]Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, and Jing Bai. 2025. ReVeal: Self-Evolving Code Agents via Reliable Self-Verification. arXiv preprint arXiv:2506.11442 (2025). [15]Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, et al.2022. MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint arXiv:2205.00445 (2022). [16] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich KĂźttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33 (2020), 9459â9474. [17]Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in neural information processing systems 36 (2023), 51991â52008. [18]Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 conference on empirical methods in natural language processing. 3102â3116. [19]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al.2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688 (2023). [20]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems 36 (2023), 46534â46594. [21]Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 (2021). [22] Albert Ărwall. 2024. Moatless Tools. https://github.com/aorwall/moatless-tools. [23]Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2024. Go- rilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems 37 (2024), 126544â126565. [24]Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023. 5687â5711. [25]Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 (2023). [26]Timo Schick, Jane Dwivedi-Yu, Roberto DessĂŹ, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems 36 (2023), 68539â68551. [27] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems 36 (2023), 8634â8652. [28]Anubhav Shrimal, Stanley Kanagaraj, Kriti Biswas, Swarnalatha Raghuraman, Anish Nediyanchath, Yi Zhang, and Promod Yenigalla. 2024. MARCO: Multi-agent real-time chat orchestration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 1381â1392. [29]Haoyu Wang, Christopher M Poskitt, and Jun Sun. 2025. Agentspec: Cus- tomizable runtime enforcement for safe and reliable llm agents. arXiv preprint arXiv:2503.18666 (2025). [30]Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al.2024. A Survey on Large Language Model Based Autonomous Agents. Frontiers of Computer Science 18, 6 (2024), 186345. [31]Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al.2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024). [32]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al.2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First conference on language modeling. [33] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 (2024). [34]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528â50652. [35]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems 36 (2023), 11809â11822. [36] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations. [37] Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. 2025. AgentOrchestra: Orchestrating Multi-Agent Intelligence with the Tool-Environment-Agent (TEA) Protocol. arXiv preprint arXiv:2506.12508 (2025). [38]Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- tocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592â1604. A Representative Task Set Table 6 provides a representative subset of the task set used in the custom repository evaluation. These examples illustrate the diversity of repositories and change types considered in the study. B Reproducibility Details B.1 Repository-Level Analysis on SWE-bench Lite The gain from augmentation is not uniform across SWE-bench repository families. We observe stronger improvements when fail- ures are tightly coupled to unit-tested, test-adjacent code paths (e.g., pytest- or linter-facing fixes), where discovery and validation hooks can directly align the implementation with executable checks. By contrast, augmentation is less reliable on repositories such as , ,Pardis Taghavi, Santosh Bhavani Table 6: Illustrative subset of tasks used in the custom repository evaluation. Repository Task ID CategoryDescription Dexter dex-01 config_changeAdd âjson flag for JSON output Dexter dex-02 new_moduleSession persistence with the âsession flag Dexter dex-03 api_endpointTelegram bot integration Dexter dex-04 new_moduleStreaming response mode Dexter dex-05 refactorPortfolio analysis module FastAPI fapi-01 new_moduleSSE streaming support FastAPI fapi-02 refactorValidation error improvements FastAPI fapi-03 new_modulePlugin system FastAPI fapi-04 api_endpointOpenAPI schema enhancements FastAPI fapi-05 new_moduleTyped middleware Airflow af-01 new_moduleError message improvements Airflow af-02 testDAG testing utilities Airflow af-03 new_moduleCustom metrics support Airflow af-04 config_changeType annotations Airflow af-05 new_moduleMemory monitoring Plausible pla-01 new_moduleFunnel visualization with conversion tracking Plausible pla-03 new_moduleAdvanced filter builder (AND/OR conditions) Plausible pla-05 api_endpointGraphQL API for analytics data Strapi str-01 new_moduleContent version history with restore Strapi str-03 new_moduleRedis query result caching Strapi str-07 new_moduleAlgolia search plugin Table 7: Model and tool versions used in the experiments. RoleSystemVersion Notes GeneratorMiniMax-M2.5N/APrimary LLM used for Spec Kit Agents execution Execution wrapperClaude Code CLI2.1.50Invoked via claude -p Judge evaluatorClaude Opus 4.620250501LLM-as-judge model ID django and matplotlib, where many failures originate in deeper application/library logic (ORM state transitions or visualization- state interactions) that are only weakly exposed by local unit tests. In SymPy-like cases, mathematically subtle edge conditions can also be under-specified by the available tests, so context-grounding hooks may anchor on incomplete signals. Overall, augmentation helps most when tests directly exercise the underlying defect, and helps less when fixes require integration context, database state, or net-new functionality beyond the tested path. B.2 Model and Tool Versions Table 7 summarizes the primary models and tools used in the ex- periments. We separate generation, execution, and evaluation roles to make the pipeline explicit. B.3 Prompting and Artifacts The context-grounding layer uses structured prompts for pre-phase discovery and post-phase validation. In Full and Full-Augmented, hooks run atspecify,plan,tasks, andimplement, and prompts are parameterized by the current workflow state and intermediate artifacts (SPEC.md,PLAN.md,TASKS.md). In Baseline and Augmented, no intermediate artifacts are generated; execution proceeds directly to implementation (with implementation-stage hooks when en- abled). Prompt templates are part of the experimental pipeline and are parameterized by workflow stage and intermediate artifacts (e.g., SPEC.md, PLAN.md, TASKS.md). B.4 Configuration Files The following configuration files were used to support reproducibil- ity: ⢠config.yaml: system-level configuration (timeouts, tool per- missions, and workflow settings), ⢠experiments/features.yaml: task definitions for the re- ported feature set, ⢠experiment_runner.py : execution and orchestration logic, and ⢠quality_evaluator.py: LLM-as-judge scoring implemen- tation. B.5 Execution Environment Experiments were conducted on a MacBook Pro (Apple Silicon) with 32 GB RAM running macOS. The execution pipeline relies on remote model inference; typical network latency to the serving endpoint was approximately 200â500 ms. As described in the main text, transient rate limits were handled via exponential backoff, and rate-limit events were logged in the run metadata. C Failure Taxonomy To make unsuccessful runs more interpretable, we categorize fail- ures by their primary cause. These categories align with the success criteria used in the main text. â˘Budget timeout: the run exceeds the end-to-end time bud- get and is terminated. â˘Human-checkpoint timeout: an approval or clarification step is not resolved within the configured timeout. â˘Artifact validation failure: an intermediate artifact fails phase-level validation (e.g., invalid paths, missing dependen- cies, or infeasible task ordering). â˘Execution or environment failure: the run encounters an authentication issue, tool-permission violation, or another execution-layer failure. Spec Kit Agents: Context-Grounded Agentic Workflows, , â˘Repository-check failure: implementation completes, but post-change tests or linters fail. ⢠Incomplete implementation: no pull request is produced, or no meaningful file modification is made. â˘Rate-limited or interrupted run: progress is interrupted by API throttling or another transient execution failure be- fore completion.