Paper deep dive
A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents
Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, David Lo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/12/2026, 3:20:20 AM
Summary
The paper introduces SWE-RPG, a repository-level benchmark designed to evaluate coding agents on requirement clarification, implementation planning, and code generation. Unlike previous benchmarks that only assess final patch success, SWE-RPG provides validated ground-truth references for implicit requirements and implementation plans to diagnose agent trajectories. The benchmark comprises 163 tasks from 31 Python and Java repositories. Evaluation of three coding agents (Claude Code, Codex, OpenCode) with six LLM backends reveals an average resolved rate of 31.5%, with implicit requirement recovery identified as the primary bottleneck.
Entities (9)
Relation Signals (7)
SWE-RPG → evaluates → Requirement Clarification
confidence 95% · SWE-RPG evaluates agents’ ability to recover implicit requirements using gold clarification references
SWE-RPG → evaluates → Implementation Planning
confidence 95% · Each task provides implementation-sufficient gold plan steps, which we use to assess the quality of implementation plans generated by coding agents.
OpenCode → usedinevaluation → SWE-RPG
confidence 95% · We evaluate 3 coding agents, including... OpenCode... on SWE-RPG
Codex → usedinevaluation → SWE-RPG
confidence 95% · We evaluate 3 coding agents, including... Codex... on SWE-RPG
Claude Code → usedinevaluation → SWE-RPG
confidence 95% · We evaluate 3 coding agents, including Claude Code... on SWE-RPG
SWE-RPG → containstasksfrom → Python and Java Repositories
confidence 92% · SWE-RPG comprises 163 tasks from 31 Python and Java repositories
Implicit Requirement Recovery → causesfailurein → Coding Agents
confidence 90% · Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.09072v1
- Canonical: https://arxiv.org/abs/2608.09072v1
Trouble viewing inline? Open PDF directly →
Full Text
42,756 characters extracted from source content.
Expand or collapse full text
SWE-RPG: A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents Xin Zhou 1 , Chun Yong Chong 2 , Kisub Kim 3 , Yun Peng 4 , Rui Shu 5 , Zihan Wu 6 , Xu Han 7 , Guowen Yuan 8 , Zeyang Zhuang 4 , Jounghoon Kim 7 , Jeongjin Ju 3 , Seongmin Ju 3 , Taein Yoon 3 , David Lo 1 1 Singapore Management University, Singapore 2 Monash University Malaysia, Malaysia 3 Daegu Gyeongbuk Institute of Science and Technology (DGIST), Republic of Korea 4 The Chinese University of Hong Kong, Hong Kong SAR, China 5 North Carolina State University, USA 6 City University of Hong Kong, Hong Kong SAR, China 7 The Hong Kong University of Science and Technology, Hong Kong SAR, China 8 The University of Hong Kong, Hong Kong SAR, China Abstract Large language model-powered coding agents are increas- ingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository- level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level bench- mark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) R equirement Clarifica- tion and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 LLM backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Re- sults show that the evaluated popular coding agents still strug- gle to implement user requests in existing repositories, achiev- ing an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit require- ment recovery as the main bottleneck, accounting for 24.5%– 46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench. 1 Introduction Large language model-powered coding agents are reshaping software development by automating repository-level main- tenance and evolution, from bug fixing (Yang et al. 2024) to feature implementation (Li et al. 2025). Reliably trans- lating users’ raw requirements into correct changes in exist- ing codebases is therefore central to their practical deploy- Figure 1: Comparison of SWE-RPG with popular bench- marks. Unlike them, SWE-RPG jointly evaluates implicit requirement clarification, planning, and code generation. ment. To assess this capability, repository-level issue resolu- tion benchmarks have progressed from SWE-bench (Jimenez et al. 2024) to recent efforts such as RACE-bench (Liu et al. 2026) and Dialogue SWE-Bench (King and Flanigan 2026). These benchmarks provide valuable end-to-end in- sights by evaluating generated patches with executable tests. However, producing a correct patch requires a long chain of decisions spanning requirement clarification, implemen- tation planning, and code generation. A final patch verdict cannot identify the actual bottleneck or reveal clear directions for improving coding agents. Limitations of Existing Benchmarks. We argue that faithfully diagnosing coding agents requires observing the complete issue-resolution 1 process. Accordingly, we pose the following question: When a coding agent fails, does the pri- mary bottleneck lie in 1requirement clarification,2im- plementation planning, or3code generation? However, prior works cannot answer this question in three respects, as summarized in Table 1: (1) Requirement Clarification. 1 In repository-level issue-resolution task, we call the original input request either the issue description or the user requirement. arXiv:2608.09072v1 [cs.SE] 10 Aug 2026 BenchmarkTime(1) Req. Clarification(2) Planning(3) Evaluation (yymm) Implicit Req. Categorized Ref. Step Ref. Reprod. Impl. Exec. Patch Stage Attr. SWE-bench (Jimenez et al. 2024)2310 SWE-bench Verified (OpenAI 2024)2408 SWE-bench Multimodal (Yang et al. 2025)2410 Multi-SWE-bench (Zan et al. 2025)2504 SWE-PolyBench (Rashid et al. 2025)2504 SWE-rebench (Badertdinov et al. 2025)2505 SWE-bench-Live (Zhang et al. 2025)2505 SWE-bench Pro (Deng et al. 2026b)2509 SWE-Compass (Xu et al. 2026)2511 SWE-Evo (Thai et al. 2025)2512 EvoClaw (Deng et al. 2026a)2603 RACE-bench (Liu et al. 2026)2603 SWE-Explore (Zhang et al. 2026)2606 Dialogue SWE-Bench (King and Flanigan 2026) 2606 SWE-RPG (Ours)– Table 1: Comparison of SWE-RPG with previous benchmarks. SWE-RPG combines implicit requirements, implementation-sufficient planning, and stage-aware evalua- tion. (1) Req. Clarification: Implicit Req. recovers implicit requirements; Categorized Ref. provides gold references un- der a practitioner-informed taxonomy. (2) Planning: Step Ref. provides actionable gold steps; Reprod. Impl. verifies the plans can generate equivalent code. (3) Evaluation: Exec. Patch evaluates patches with executable tests; Stage Attr. as- signs main failure reasons to specific stages. //denote no/partial/direct support. In practice, initial requirement descriptions are often incom- plete or ambiguous, requiring developers to recover implicit constraints from task and repository context (Knauss et al. 2015; Franch et al. 2023). Recent benchmarks expand evalu- ation to long-horizon and continuous-evolution tasks (Deng et al. 2026b; Thai et al. 2025; Deng et al. 2026a), assess high-level issue understanding (Liu et al. 2026), or study clarification in simulated dialogue (King and Flanigan 2026). However, none provides validated references for systemati- cally evaluating agents’ recovery of implementation-critical requirements left implicit in original requests. (2) Planning. Implementation plans translate intended behavior into ac- tionable repository changes (Bairi et al. 2024) and should be sufficient to generate functionally equivalent implemen- tations. Most benchmarks provide no implementation-plan references. RACE-bench comes closest with structured task steps, but does not test their plan-to-code reproducibility. (3) Evaluation. Existing benchmarks primarily determine whether submitted patches pass executable tests, revealing whether agents fail but not where. Some provide module- or failure-level analyses, but none consistently attributes un- successful trials to aligned requirement, planning, or coding stages. Our Solution. We propose SWE-RPG, a new stage- aware benchmark for evaluating coding agents’ capabili- ties in requirement clarification, planning, and coding dur- ing repository-level issue resolution. SWE-RPG addresses the aforementioned limitations through three main fea- tures: (1) Requirement Clarification. SWE-RPG evalu- ates agents’ ability to recover implicit requirements using gold clarification references across six practitioner-informed dimensions (e.g., functional intent and technical context). (2) Planning. Each task provides implementation-sufficient gold plan steps, which we use to assess the quality of imple- mentation plans generated by coding agents. (3) Evaluation. SWE-RPG evaluates final patches with executable tests, an- alyzes trajectories to assess clarification and planning, and attributes failures to requirement, planning, or coding stages. Evaluation. We evaluate 3 coding-agent frameworks paired with 6 LLM backends. Agents achieve an average resolved rate of only 31.5%, with 46.7% of agent runs fail- ing primarily during requirement clarification or implemen- tation planning. Claude Code, Codex, and OpenCode also expose distinct stage-aware failure profiles across require- ment, planning, implementation, and artifact stages. These findings highlight distinct capability bottlenecks and show how stage-aware evaluation can suggest candidate areas for improvement. In summary, our contributions are as follows: • We propose SWE-RPG, a stage-aware benchmark com- prising 163 tasks from 31 Python/Java repositories. To our knowledge, it is the first repository-level benchmark to jointly evaluate agents’ capabilities in requirement clar- ification, implementation planning, and code generation. • We evaluate three popular coding-agent frameworks paired with six recent LLMs in a3×6 matrix on SWE- RPG. Agents achieve an average resolved rate of only 31.5%, while 46.7% of all agent runs fail during require- ment clarification or implementation planning. Our stage- aware analyses reveal framework-specific bottlenecks and suggest candidate areas for improving coding agents. 2 Related Work Repository-level Issue-resolution Benchmarks. SWE- bench established an executable benchmark for repository- level issue resolution: agents modify code repositories to complete requirements, and their patches are evaluated with tests (Jimenez et al. 2024). Later benchmarks improve relia- bility and freshness (OpenAI 2024; Badertdinov et al. 2025; Zhang et al. 2025) or broaden programming-language and task coverage (Yang et al. 2025; Zan et al. 2025; Rashid et al. 2025; Deng et al. 2026b; Xu et al. 2026; Thai et al. 2025; Deng et al. 2026a). However, as Table 1 shows, most benchmarks primarily evaluate final patches and lack val- idated requirement and planning references for diagnosing unsuccessful trajectories. Close Works. RACE-bench (Liu et al. 2026) and Dialogue SWE-Bench (King and Flanigan 2026) are the two closest works. RACE-bench. RACE-bench is a concurrent work that shares our goal of exposing intermediate reasoning (Liu et al. 2026). However, the following two features distinguish SWE- RPG from RACE-bench. First, RACE-bench does not pro- vide references for implicit requirements, a central challenge in requirement clarification; it summarizes only the high- level intent already stated in the original requirement. In contrast, SWE-RPG provides evidence-supported references for implementation-critical information not fully specified in the initial request. Second, RACE-bench provides plan steps and validates their semantic necessity, but does not test whether code generated from them can reproduce a function- ally equivalent code. In contrast, SWE-RPG directly checks plan-to-code reproducibility through code generation. B GTs Construction for Planning C1: Functional Intent (2 examples) C2: Business Semantics (12 examples) C3: Technical Context (12 examples) C4: Interface/Protocol Specification (13 examples) C5: Code Structure and Conventions (10 examples) C6: Data-Structure Semantics (8 examples) Identify req. clarification categories and provide typical examples Examples (QAs) Code Repos Original Req. (Issue) Pull Request ① Practitioner-Guided Clarification Taxonomy & Examples 10 SE Engineers Interviews Clarified Req. Points (QA format) C1: QA Pair1 , ... C2: QA Pair1, ... C3: QA Pair1, ... Inputs Clarified Req. (Reconstructed Initial GT) 1. Analyze PR and Repo Synthesis Agent 2. Identify Missing Req. 3. Clarify Missing Req. ① Synthesis Code Repos Original Req. (Issue) Pull Request Inputs Initial Planning Step 1: ... Step 2: ... ... Plan Steps (Reconstructed Initial GT) ② Synthesis ② Validation Plan (Step 1) Code Generation Agent Apply Step 1’s Plan Generated Patch (Step 1) Gold Patch (Step 1) Semantic Equivelent? NO Patch Gap Analysis & Resolution Agent Plan (Step 1) Plan (Step 2) Verfied GTs for Plannings (Can Replicate GT Patches ; Manually Reviewed ) Generate Patch YES Manual Review ③ Validation Clarified Req. Points (QA) 1. Code Repo 2. GT Patch 3. GT Plans Evidence Check Manual Review Req. (C1) Req. (C2) .... Verfied GTs for Req. Clarification (Evidence Checked; Manually Reviewed ) A GTs Construction for Requirment Clarification (Reconstrcucted Initial GT) 1. Analyze PR and Repo Synthesis Agent 2. Understand Raw Req. 3. Synthesize Plans Validation Agent Initial Planning Step-by-Step Validation Revise Plan Validation Agent .... Figure 2: Ground-truths Construction for Requirement Clarification and Planning. Using the original requirement, repository, merged PR, and tests, we synthesize Clarification GT and an actionable implementation plan. Clarification GT is validated for evidential support, and cross-stage consistency. Planning GT is validated step by step by checking whether each step can guide an implementation functionally equivalent to its gold subpatch. Finally, two authors independently review every resulting reference for evidential support and cross-stage consistency. Dialogue SWE-Bench. Dialogue SWE-Bench reformulates SWE-bench Verified (OpenAI 2024) tasks as interactive ses- sions. For each task, it constructs a shortened initial query by deliberately omitting critical details, then studies whether agents can recover them through simulated dialogue. The resulting clarification setting is therefore synthetic and do not provide references mined from naturally implicit require- ments. In contrast, SWE-RPG retains the original request and mines implementation-critical implicit requirements from real development evidence. We also provide reproducible planning references, which Dialogue SWE-Bench lacks. 3 SWE-RPG Benchmark SWE-RPG is a repository-level issue-resolution benchmark with executable patch oracles and validated intermediate GTs. In this section, we first formulate the benchmark task and its GT-aligned diagnostic protocol, then describe task and annotation construction and summarize the resulting dataset. Figure 2 summarizes how we reconstruct and validate the ground-truth references for Req. Clarification and Planning. 3.1 Task Formulation and Example Input and Output. The input to each task consists of (i) a repository checked out at a specified base commit and (i) a user requirement in natural language. Given these inputs, the coding agent autonomously modifies the repository and outputs a code patch. Example Instance. Figure 3 illustrates a SWE-RPG in- stance, in which TSV import incorrectly removes boundary whitespace when trimStrings is disabled. The instance provides stage-specific ground-truth artifacts for diagnosis: Clarification GT makes implicit requirements explicit, in- cluding the TSV-only scope and unchanged CSV behav- ior; Plan GT provides the implementation steps. The Gold Patch, Functional Tests, and Docker environment support Example: OpenRefine-6609. Raw requirement. TSV import incorrectly removes boundary whitespace when trimStrings is disabled. Clarification GT. Functional Intent (C1) Q: Regarding the fixing scope, should the fix change whitespace handling for all separator-based imports? A: No. It must affect only the TSV path; CSV behavior must remain unchanged . . . Plan GT. Step 1: Locate the TSV branch in SeparatorBasedImporter.parseOneFile. Step 2: Disable Univocity’s automatic removal of leading and trailing whitespace in TsvParserSettings . . . Patch GT. TsvParserSettings settings = ...; + settings.setIgnoreLeadingWhitespaces(false); + settings.setIgnoreTrailingWhitespaces(false); ... Tests. readDoesNotTrimLeadingTrailingWhitespace... Environment: in a reproducible docker container Figure 3: An illustrative SWE-RPG instance. executable, reproducible patch evaluation. Together, these artifacts support stage-aware analysis and evaluation. 3.2 Benchmark Construction Repository and Task Selection. SWE-RPG collects merged issue–PR pairs from mature, actively maintained Python and Java repositories. Each pair must include re- producible tests, involve a recent and traceable change, and not overlap with existing issue-resolution benchmarks. This screening yields 2000+ issue–PR instances with task-relevant tests from 100+ repositories. We further retain only bug- fixing and new-feature tasks whose tests directly exercise the target behavior, resulting in 1000+ candidate instances. Repo and Task Collection Task FilteringEnvironment BuildTest ValidationIntermediate GT Validation 2,000+ Eligible PR-Issue Pairs 1,000+ Relevant tasks 400+ With Runnable Environments 200+ With Stable Tests 163 With Verified Intermediate GTs Table 2: Benchmark Candidate reduction during benchmark construction. SWE-RPG Dataset Composition Tasks163 Repositories31 Java / Python tasks85 / 78 Java / Python repos.17 / 14 Bug fixes113 (69.3%) Feature implementations50 (30.7%) Task StatisticsPlan GT StatisticAvg. / Max. StatisticAvg. / Max. Requirement Words301.10 / 2,195 Plan steps2.06 / 10 Codebase LOC272.51K / 1.59M File locations2.47 / 10 Gold-patch LOC56.74 / 639 Rule constraints11.80 / 50 Test cases2,248.56 / 15,119 Clarification GT StatisticAvg. / Max. StatisticAvg. / Max. # of Implicit Clarification Points4.53 / 8 C1 Functional intent1.21 / 4 C4 Interface/protocol0.99 / 2 C2 Business semantics0.60 / 3 C5 Structure/conventions0.47 / 1 C3 Technical context0.72 / 2 C6 Data-structure semantics0.54 / 2 Table 3: Statistics of SWE-RPG and its intermediate GTs. Building Environments. We reuse the automated agen- tic build pipeline of SWE-bench-Live (Zhang et al. 2025) to construct an isolated Docker image for each candidate at its base commit. We extend this pipeline with expert-guided recovery: when an automated build fails, the first author in- spects its logs, identifies missing dependencies or configura- tion, and feeds these corrections into the next build attempt. We validate each image by comparing test outcomes before and after applying the gold patch. We repeat tests to remove flaky instances and manually verify that fail-to-pass tests di- rectly target the intended behavior. This process yields 400+ runnable images and 200+ qualified instances. Overview of Ground-Truth Construction. We construct ground truth differently for the three stages. Original require- ments often omit implementation-critical details (Bo et al. 2024; Knauss et al. 2015; Franch et al. 2023), while existing benchmarks generally provide no explicit implementation- plan references (Table 1). We therefore construct the Require- ment Clarification and Planning references through synthesis and validation. During synthesis, we derive the references from the base repository, issue and PR discussion, devel- oper patch, and selected tests. During validation, we check them against the development evidence and implemented behavior. We instantiate the agentic components of this con- struction pipeline with GPT-5.4 (OpenAI 2026c), the latest available model at the time of benchmark construction, to maximize the quality of the synthesized references. For Code Generation, no synthesis is needed: the merged PR provides the developer patch, and its task-relevant tests provide the executable correctness oracle. We detail the synthesis and validation procedure for each stage below. Ground-Truth Construction for Req. Clarification. Practitioner-Guided Clarification Taxonomy. Require- ment clarification is difficult to define because it targets implementation-critical requirements that remain implicit in the original request. Which implicit decisions matter to de- velopers, and which aspects they repeatedly clarify in real software development, are unknown to some extent. We therefore conducted seminar-style interviews with ten ex- perienced software engineers from Fortune Global 500 tech- nology companies. We asked what information developers need before implementation, what missing information they recover during clarification, and which omissions most of- ten cause incorrect implementations. Their responses served two purposes: deriving a practitioner-grounded taxonomy and collecting representative QA examples for GT synthe- sis. The resulting taxonomy uses standardized QA pairs and comprises six categories: C1: Functional Intent, C2: Busi- ness Semantics, C3: Technical Context, C4: Interface and Protocol Specifications, C5: Code Structure and Naming Conventions, and C6: Detailed Data-Structure Semantics. We curated 57 seed question-answer pairs. Synthesis and Validation. As illustrated in Figure 2, the taxonomy guides a synthesis-and-validation pipeline. Using the corresponding seed QA examples as few-shot demon- strations, synthesis agents examine the issue, repository, and merged PR to identify implementation-critical information that is underspecified or implicit in the issue and formulate candidate questions. Next, a validation agent checks each QA pair against the repository, developer patch, and validated plan for evidential support, cross-stage consistency, and ab- sence of implementation detail leakage. Finally, two authors independently review every resulting clarification reference for evidential support and cross-stage consistency. Ground-Truth Construction for Planning. Planning- ground-truth construction has two phases: synthesis and val- idation. During synthesis, a single synthesis agent examines the original issue, base repository, and merged PR, recov- ers the implementation intent, explains how the developer patch realizes it, and organizes these findings into an or- dered sequence of modular steps. Each step specifies its goal, relevant files, intended changes, and repository constraints. These fields make the step actionable and auditable: they tell a coding agent what to achieve, where and how to modify the repository, and which constraints to preserve, while enabling step-level alignment with the gold patch. During validation, an alignment agent associates each plan step with the cor- responding subset of the complete developer patch. We call this subset the step’s gold subpatch. We validate the plan sequentially, one step at a time. For each step, a coding agent follows the step to generate a patch, and an evaluator judges whether it is semantically equivalent to the corresponding gold subpatch. If the check fails, a gap-analysis and resolu- tion agent revises the step plan. Steps that pass are retained. This process establishes functional reproducibility: an ac- 10 −1 10 0 Average Cost per Task (USD, Log Scale) 15 20 25 30 35 40 45 50 Resolve Rate (%) Average Cost: $1.59 Average Resolve Rate: 31.5% OC-Claude5 OC-DSV4 OC-GLM5.2 OC-GPT5.6 OC-M3 OC-KimiK3 C-Claude5 C-DSV4 C-GLM5.2 C-GPT5.6 C-M3 C-KimiK3 Codex-Claude5 Codex-DSV4 Codex-GLM5.2 Codex-GPT5.6 Codex-M3 Codex-KimiK3 OCCCCodex (a) Cost versus resolve rate 246810121416 Average Time per Task (min) 15 20 25 30 35 40 45 50 Resolve Rate (%) Average Time: 8.5 min Average Resolve Rate: 31.5% OC-Claude5 OC-DSV4 OC-GLM5.2 OC-GPT5.6 OC-M3 OC-KimiK3 C-Claude5 C-DSV4 C-GLM5.2 C-GPT5.6 C-M3 C-KimiK3 Codex-Claude5 Codex-DSV4 Codex-GLM5.2 Codex-GPT5.6 Codex-M3 Codex-KimiK3 OCCCCodex (b) Time versus resolve rate Figure 4: Cost and execution time versus resolve rate. Claude5 DSV4GLM5.2GPT5.6M3 KimiK3 LLM OC C Codex Agent 30.1%36.2%36.8%25.2%20.2%49.7% 31.3%39.3%25.8%22.7%28.8%49.1% 26.4%40.5%23.3%22.1%17.8%41.1% 0 20 40 60 80 100 Resolve Rate (%) Figure 5: Resolve rates across agent–LLM combinations. cepted plan must provide sufficient guidance for a coding agent to produce a functionally equivalent implementation. Finally, two authors independently review every resulting plan for evidential support and cross-stage consistency. Ground-Truth Construction for Code Generation. Un- like Req. Clarification and Planning, Code Generation re- quires no annotation reconstruction. Each merged PR directly provides the developer implementation patch, while its task- relevant tests serve as the executable correctness oracle. Quality Assurance. As described above, we enforce qual- ity control through the following stage-specific validation and expert-review measures. At the task level, we retain tasks with buildable environments and stable tests. For Plan- ning, every step must support an implementation functionally equivalent to its gold subpatch. For Requirement Clarifica- tion, every point (QA pair) must be supported by repository evidence, consistent with the validated plan and final imple- mentation. Lastly, human experts review every retained task and its GTs across all three stages. 3.3 Benchmark Statistics and Characteristics Table 2 demonstrates the rigor and selectivity of our quality- oriented construction: successive checks of task relevance, environment reproducibility, test stability, and GT validity re- duce more than 2,000 real-world PR–issue candidates to 163 quality-controlled instances. Table 3 further reflects substan- tial repository-scale complexity: codebases contain 272.51K lines of code (LOC) on average, with the largest reaching 1.59M LOC; and each task contains 2,248.56 functional test cases on average. For each task, Clarification GT recovers 4.53 implicit and implementation-critical requirement points on average, while Plan GT provides an average of 2.06 ac- tionable implementation steps with 11.80 implementation constraints. Our benchmark covers across 7 engineering do- mains, including static analysis, compilers, and schema sys- tems, algorithms, mathematics, and scientific computing. 4 Experimental Setup Evaluated Agents and Models. We evaluate three pop- ular coding-agent frameworks (Claude Code, Codex, and OpenCode) paired with six recent LLMs: Claude-Sonnet- 5 (Anthropic 2026), DeepSeek-V4-Pro (DeepSeek-AI 2026), GLM-5.2 (Z.AI 2026), GPT-5.6-Terra (OpenAI 2026b), MiniMax-M3 (MiniMax 2026), and MoonshotAI-Kimi- K3 (Moonshot AI 2026). We run each agent-LLM config- uration twice on every task and report run-level aggregates over the two runs. Computing Infrastructure. All experiments were con- ducted on Ubuntu 26.04 LTS using an Intel Xeon 6982P-C CPU (48 cores and 96 threads) with 182 GiB of RAM. We accessed all LLMs through their official API providers. Evaluation Metric. We use Resolve Rate as the primary evaluation metric, following the SWE-bench execution pro- tocol (Jimenez et al. 2024). A task is considered resolved only if its patch applies successfully, all fail-to-pass tests pass, and no pass-to-pass test regresses. Failure Attribution to Stages. For each unresolved run, we use GPT-5.6-Sol (OpenAI 2026a) as an LLM judge. Its dedicated prompt jointly examines the task, intermediate- stage GTs, and the agent trajectory. To make long trajectories tractable without discarding stage-relevant evidence, we re- move only verbose tool-call payloads and outputs unrelated to requirement understanding, planning, implementation, or verification, while retaining relevant repository observations, decisions, edits, and validation actions. The judge assigns OC KimiK3 C KimiK3 Codex KimiK3 Codex DSV4 C DSV4 OC GLM5.2 OC DSV4 C Claude5 OC Claude5 C M3 Codex Claude5 C GLM5.2 OC GPT5.6 Codex GLM5.2 C GPT5.6 Codex GPT5.6 OC M3 Codex M3 0 20 40 60 80 100 49.7% 49.1% 41.1% 40.5% 39.3% 36.8% 36.2% 31.3% 30.1% 28.8% 26.4% 25.8% 25.2% 23.3% 22.7% 22.1% 20.2% 17.8% 25.8% 24.5% 35.6% 26.4% 40.5% 39.3% 30.7%40.5% 39.3% 40.5% 37.4%38.7% 42.9% 39.9% 29.4% 46.0% 35.6% 33.7% 6.7% 5.5% 12.3% 9.2% 12.9% 6.1% 8.0% 9.8%14.7% 10.4% 14.7% 11.0% 17.8% 6.1% 10.4% 17.8% 11.0% 8.6% 16.0% 18.4% 9.8% 23.9% 7.4% 17.2% 25.2% 18.4% 15.3% 17.8% 19.6% 22.7% 14.1% 24.5% 37.4% 14.1% 33.1% 36.2% 6.1% ResolvedRequirement FailurePlanning FailureCode Generation FailureOthers (Including Timeout, Patch Format, etc.) Percentage of Agent Runs (%) Figure 6: Failure diagnoses across agent–LLM configurations. Green denotes resolved runs; the remaining segments show the proportions attributed to requirement, planning, code-generation, and other failures (e.g., timeouts). each failure to the earliest deviating stage: requirement un- derstanding, planning, implementation, or others. On a strat- ified sample of 50 unresolved runs, its labels exactly match human consensus in 46 cases (92%). Pre-code Stage Evaluation. Our Clarification and Plan GTs decompose each task into labeled information points. Clarification spans six categories (C1–C6), whereas Plan- ning covers the goal, target location, implementation ap- proach, constraints, and validation strategy. For each run, the LLM judge (GPT-5.6-Sol) determines whether the agent’s trajectory covers each GT information point. Because trajec- tories can be long, we remove verbose tool-call payloads and outputs that carry no evidence about requirement understand- ing or planning, while retaining relevant repository observa- tions, decisions, edits, and validation actions. We compute category- and dimension-level coverage rates to evaluate the Clarification and Planning stages from full-trajectory evi- dence rather than from the final patch outcome alone. To accommodate multiple valid plans, the prompt evaluates se- mantic implementation responsibilities rather than exact plan matching: it accepts alternative files, symbols, architectures, and step orderings when they realize the same responsibility and satisfy repository constraints, without requiring textual or structural similarity to the reference plan. On a manu- ally annotated sample of 50 information-point assessments, the judge’s binary coverage decisions agree with the human consensus in 96% of cases. 5 Results and Analysis 5.1 Patch Correctness Overall Patch Correctness Results. On average, coding agents achieve a resolved rate of only 31.5%, indicating the difficulty of the SWE-RPG tasks. At the configuration level, OpenCode with MoonshotAI-Kimi-K3 achieves the high- est resolved rate (49.7%). Thus, even the best configuration leaves roughly half of the tasks unresolved, indicating sub- stantial room for improvement in repository-level issue res- olution. Effectiveness of Agents, LLMs, and Pairings. Figure 5 compares performance at the agent, LLM, and agent– LLM pairing levels. Among agents, OpenCode achieves the highest average resolved rate (33.0%), followed closely by Claude Code (32.8%) and Codex (28.5%). Among LLMs, MoonshotAI-Kimi-K3 performs best, averaging 46.6% across the three agents, followed by DeepSeek- V4-Pro at 38.7%; the remaining backends average 22.3– 29.2%. At the pairing level, OpenCode–MoonshotAI-Kimi- K3 is strongest (49.7%), closely followed by Claude Code– MoonshotAI-Kimi-K3 (49.1%), whereas Codex–MiniMax- M3 is weakest (17.8%). The resulting spread shows substan- tial descriptive variation across backend–scaffold pairings. Cost Analysis. Figure 4(a) relates resolve rate to cost, with dashed lines marking the average cost ($1.59 per task) and av- erage resolve rate (31.5%). OpenCode–MoonshotAI-Kimi- K3 achieves the highest resolve rate at below-average cost, whereas Claude Code with the same backend attains similar performance at above-average cost. DeepSeek-V4-Pro pair- ings likewise perform above average but vary substantially in cost, showing that cost-effectiveness depends on both the backend and agent scaffold. Time Analysis. Figure 4(b) shows a partial performance– time trade-off. Many above-average configurations, partic- ularly those using MoonshotAI-Kimi-K3, exceed the aver- age execution time of 8.5 minutes. However, Claude Code– DeepSeek-V4-Pro performs above average in substantially less time, while several slower configurations remain be- low average. Thus, longer execution does not consistently produce better patches, and pairings exhibit distinct time– performance trade-offs. 5.2 Failure Diagnosis Patch correctness reveals whether a run succeeds, but not why it fails. We therefore align agent trajectories with vali- C1 Intent & Scope C2 Business C3 Technical C4 Interface C5 Structure C6 Data Semantics OC C Codex 54.8%37.7%49.5%33.6%29.3%26.1% 86.6%74.2%74.2%64.7%54.2%63.9% 75.1%56.4%63.8%47.3%42.0%41.9% Requirement Clarification GoalLocationApproach Constraints & Validation 41.2%43.8%31.1%24.8% 74.8%79.7%64.5%41.6% 60.8%65.6%49.0%37.1% Planning 0 20 40 60 80 100 Quality Score (%) Capability Analysis by Agent Figure 7: Clarification-category and planning-dimension coverage by agent, averaged across six LLMs. Boxes mark the lowest scores. dated Clarification and Plan GTs to diagnose stage-specific failures. Figure 6 decomposes results into resolved cases, and unresolved cases are attributed to main failure stages. Requirement failure is the largest component for most configurations (24.5–46.0%), followed by code- generation (7.4–37.4%) and planning (5.5–17.8%) fail- ures. This ordering identifies implicit requirement recov- ery as the most common bottleneck, while planning errors are also non-negligible. Code-generation failure rates show substantial descriptive variation across LLM–scaffold pair- ings, particularly for several MiniMax-M3 pairings. Similar resolve rates mask different bottleneck profiles. Codex– and Claude Code–DeepSeek-V4-Pro achieve similar resolve rates (40.5% versus 39.3%) but markedly different profiles: their requirement-failure rates are 26.4% and 40.5%, whereas their implementation/verification-failure rates are 23.9% and 7.4%. Similar end-to-end performance can there- fore accompany distinct stage-attribution profiles. SWE-RPG suggests candidate areas for agent improve- ment. Grounded in validated intermediate GTs, SWE-RPG provides evidence of where failures arise and identifies can- didate modification areas beyond what previous outcome- only evaluation can support. These diagnoses suggest stage- specific directions: requirement failures motivate investigat- ing stronger implicit-constraint recovery, while planning fail- ures motivate investigating better implementation plans. 5.3 Req. Clarification and Planning of Agents We compare each agent’s trajectory with the validated Clar- ification and Plan GT information points to measure how completely it captures the expected pre-code information. Figure 7 reports coverage rates by clarification category and planning dimension, averaged across 6 LLMs for each agent. Lower clarification coverage is concentrated in inter- faces, structure, and data semantics. All three agents cover intent and scope most reliably, but weaken on the implementation-facing categories. Claude Code’s lowest clarification score is code structure (54.2%); Codex falls to 42.0% on structure and 41.9% on data semantics; These re- sults suggest candidate requirement-clarification targets, in- cluding interface contracts, repository conventions, and data invariants beyond the high-level intent. Lower planning coverage is concentrated in approach, constraints, and validation. For every agent, planning coverage decreases from target location to implementation approach and then to constraints and validation: 79.7% to 64.5% to 41.6% for Claude Code, 65.6% to 49.0% to 37.1% for Codex, and 43.8% to 31.1% to 24.8% for OpenCode. This consistent cascade shows that agents often identify where to edit without fully specifying how to implement and verify the change. These lower-coverage dimensions suggest evaluating plans that more explicitly capture the implementation mecha- nism, boundary conditions, compatibility requirements, and validation obligations. 6 Conclusion SWE-RPG combines 163 real PR-derived tasks, executable patch oracles, and validated Clarification and Plan GTs. Be- yond measuring final patch correctness, it supports full- trajectory, GT-aligned retrospective diagnosis of require- ment, planning, implementation, and artifact gaps. These diagnostic profiles make otherwise identical unresolved out- comes more interpretable and suggest stage-specific hypothe- ses for future improvement. The Code and Data Supplement contains the code needed to run the benchmark and reproduce the reported analyses. Limitations SWE-RPG contains 163 Python and Java tasks from 31 repositories, so its findings may not generalize to other languages or ecosystems. Because each agent runs au- tonomously end to end, the benchmark cannot assess the effect of human feedback or interactive refinement. Its in- termediate GTs and retrospective alignment labels depend on LLM-assisted construction and judging, although all GTs are manually validated. Future work should expand task and model coverage and strengthen annotation audits, including inter-annotator agreement. Generative AI tools assisted with manuscript drafting and language editing; the authors re- viewed and verified all content, claims, references, and final wording. References Anthropic. 2026. Introducing Claude Sonnet 5. https: //w.anthropic.com/news/claude-sonnet-5. Model an- nouncement; accessed July 29, 2026. Badertdinov, I.; Golubev, A.; Nekrashevich, M.; Shevtsov, A.; Karasik, S.; Andriushchenko, A.; Trofimova, M.; Litv- intseva, D.; and Yangel, B. 2025. SWE-rebench: An Au- tomated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. In Advances in Neural Information Processing Systems, volume 38. Bairi, R.; Sonwane, A.; Kanade, A.; D. C., V.; Iyer, A.; Parthasarathy, S.; Rajamani, S.; Ashok, B.; and Shet, S. 2024. CodePlan: Repository-Level Coding Using LLMs and Plan- ning. Proceedings of the ACM on Software Engineering, 1(FSE): 675–698. Bo, L.; Ji, W.; Sun, X.; Zhang, T.; Wu, X.; and Wei, Y. 2024. ChatBR: Automated Assessment and Improvement of Bug Report Quality Using ChatGPT. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 1472–1483. ACM. DeepSeek-AI. 2026. DeepSeek-V4-Pro API Documentation. https://api-docs.deepseek.com/. Model documentation; ac- cessed July 29, 2026. Deng, G.; Chen, Z.; Yu, Z.; Fan, H.; Liu, Y.; Yang, Y.; Parikh, D.; Kannan, R.; Cong, L.; Wang, M.; Zhang, Q.; Prasanna, V. K.; Tang, X.; and Wang, X. 2026a. EvoClaw: Evaluat- ing AI Agents on Continuous Software Evolution. In Pro- ceedings of the 43rd International Conference on Machine Learning. Deng, X.; Da, J.; Pan, E.; He, Y. Y.; Ide, C.; Garg, K.; Lauffer, N.; Park, A.; Pasari, N.; Rane, C.; Sampath, K.; Krishnan, M.; Kundurthy, S.; Hendryx, S. M.; Wang, Z.; Zhang, C. B. C.; Jacobson, N.; Liu, B.; and Kenstler, B. 2026b. SWE- Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? In Proceedings of the 43rd International Conference on Machine Learning. Franch, X.; Palomares, C.; Quer, C.; Chatzipetrou, P.; and Gorschek, T. 2023. The State-of-Practice in Requirements Specification: An Extended Interview Study at 12 Compa- nies. Requirements Engineering, 28(3): 377–409. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. R. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations. King, B.; and Flanigan, J. 2026. Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents. arXiv preprint arXiv:2606.13995. Knauss, E.; Damian, D.; Cleland-Huang, J.; and Helms, R. 2015. Patterns of Continuous Requirements Clarification. Requirements Engineering, 20(4): 383–403. Li, W.; Zhang, X.; Guo, Z.; Mao, S.; Luo, W.; Peng, G.; Huang, Y.; Wang, H.; and Li, S. 2025. FEA-Bench: A Bench- mark for Evaluating Repository-Level Code Generation for Feature Implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 17160–17176. Liu, S.; Zhao, Z.; Hu, X.; Liu, K.; Yang, X.; and Xia, X. 2026. RACE-bench: A Benchmark for Evaluating Repository- Level Code Agents with Intermediate Reasoning on Feature Addition Task. arXiv preprint arXiv:2603.26337. MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Con- text, Native Multimodality. https://w.minimax.io/blog/ minimax-m3. Model announcement; accessed July 29, 2026. Moonshot AI. 2026. Kimi K3: Our Most Capable Model. https://forum.moonshot.ai/t/kimi-k3-is-here-our- most-capable-model/480. Official model announcement; ac- cessed July 29, 2026. OpenAI. 2024. Introducing SWE-bench Verified. https: //openai.com/index/introducing-swe-bench-verified/. Tech- nical blog post; accessed July 27, 2026. OpenAI. 2026a. GPT-5.6 Sol Model. https://developers. openai.com/api/docs/models/gpt-5.6-sol. Model documen- tation; accessed July 29, 2026. OpenAI. 2026b. GPT-5.6 Terra Model. https://developers. openai.com/api/docs/models/gpt-5.6-terra. Model docu- mentation; accessed July 29, 2026. OpenAI. 2026c. Introducing GPT-5.4. https://openai.com/ index/introducing-gpt-5-4/. Technical blog post; accessed July 29, 2026. Rashid, M. S.; Bock, C.; Zhuang, Y.; Buchholz, A.; Esler, T.; Valentin, S.; Franceschi, L.; Wistuba, M.; Sivaprasad, P. T.; Kim, W. J.; Deoras, A.; Zappella, G.; and Callot, L. 2025. SWE-PolyBench: A Multi-Language Benchmark for Repos- itory Level Evaluation of Coding Agents. arXiv:2504.08703. Thai, M. V. T.; Le, T.; Manh, D. N.; Nhat, H. P.; and Bui, N. D. Q. 2025. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. arXiv preprint arXiv:2512.18470. Xu, J.; Deng, K.; Li, W.; Yu, S.; Tang, H.; Huang, H.; Lai, Z.; Zhan, Z.; Wu, Y.; Zhang, C.; et al. 2026. SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models. In Proceedings of the 43rd Inter- national Conference on Machine Learning. Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; and Press, O. 2024. SWE-agent: Agent- computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, vol- ume 37. Yang, J.; Jimenez, C. E.; Zhang, A. L.; Lieret, K.; Yang, J.; Wu, X.; Press, O.; Muennighoff, N.; Synnaeve, G.; Narasimhan, K. R.; Yang, D.; Wang, S. I.; and Press, O. 2025. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? In The Thirteenth International Conference on Learning Representations. Z.AI. 2026. GLM-5.2: Built for Long-Horizon Tasks. https: //z.ai/blog/glm-5.2. Model announcement; accessed July 29, 2026. Zan, D.; Huang, Z.; Liu, W.; Chen, H.; Zhang, L.; Xin, S.; Chen, L.; Liu, Q.; Zhong, X.; Li, A.; et al. 2025. Multi-SWE- bench: A Multilingual Benchmark for Issue Resolving. In Advances in Neural Information Processing Systems, vol- ume 38. Zhang, L.; He, S.; Zhang, C.; Kang, Y.; Li, B.; Xie, C.; Wang, J.; Wang, M.; Huang, Y.; Fu, S.; Nallipogu, E.; Lin, Q.; Dang, Y.; Rajmohan, S.; and Zhang, D. 2025. SWE-bench Goes Live! In Advances in Neural Information Processing Systems, volume 38. Zhang, S.; Wang, Y.; Liang, J.; Shi, Y.; Zeng, W.; Wang, M.; He, S.; Xu, N.; Ye, S.; Cai, K.; and Gu, X. 2026. SWE- Explore: Benchmarking How Coding Agents Explore Repos- itories. arXiv preprint arXiv:2606.07297.