Paper deep dive
SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/5/2026, 2:09:35 AM
Summary
The paper introduces SWE-Gate, a repository-level benchmark for software engineering agents that evaluates both functional correctness and compliance with review-derived acceptance constraints. Unlike existing benchmarks that only test if patches pass functional tests, SWE-Gate synthesizes instances from real pull request reviews to separate issue resolution from constraint adherence. The benchmark comprises 303 instances across 75 Python repositories, demonstrating that many agents pass functional tests but fail to satisfy review constraints, indicating that functional-only evaluation overestimates agent capabilities.
Entities (8)
Relation Signals (5)
Review Constraints → derivedfrom → Pull Request Reviews
confidence 95% · SWE-Gate derives review constraints from real pull request review comments
SWE-Gate → evaluates → Review Constraints
confidence 95% · SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness.
SWE-Gate → usesdatafrom → Python
confidence 95% · We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories
SWE-Gate → compareswith → SWE-BENCH
confidence 90% · SWE-Gate is introduced to bridge the gap left by existing benchmarks like SWE-Bench which primarily measure functional tests.
SWE-Gate → createdby → Sun Yat-sen University
confidence 90% · Authors Xin He, Yanlin Wang, Mingwei Liu are affiliated with School of Software Engineering, Sun Yat-sen University.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.04167v1
- Canonical: https://arxiv.org/abs/2609.04167v1
Trouble viewing inline? Open PDF directly →
Full Text
58,691 characters extracted from source content.
Expand or collapse full text
SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents Xin He 1 , Yanlin Wang 1∗ , Mingwei Liu 1 , Jiachi Chen 2 , Hongyu Zhang 3 , Guanbin Li 4 1 School of Software Engineering, Sun Yat-sen University 2 College of Computer Science and Technology, Zhejiang University 3 School of Big Data and Software Engineering, Chongqing University 4 School of Computer Science and Engineering, Sun Yat-sen University Abstract Repository-level software engineering benchmarks have sig- nificantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived ac- ceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software devel- opment. We introduce SWE-Gate, a repository-level bench- mark for software engineering agents that explicitly evalu- ates review constraint compliance alongside functional cor- rectness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level re- pair instances around these constraints. Each instance pro- vides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separa- tion between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository- level repair instances spanning 75 open-source Python repos- itories across diverse software domains. Experiments with four LLM backends spanning different capability levels un- der a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the func- tional tests, 221 fail to satisfy the provided review con- straints. These findings show that functional-only evaluation overestimates agents’ ability to satisfy the full requirements of repository-level repair tasks. The replication package in- cluding code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate. Introduction Recent advances in large language models (LLMs) have transformed automated software engineering, evolving from early function-level code generation benchmarks [1, 2] to repository-level agents capable of understanding large code- bases [3–5], localizing defects, and repairing real software issues using code, version history, tests, and natural-language reports [6–9]. Repository-level repair has consequently be- come an important measure of practical programming abil- ity. SWE-Bench [6], the de facto benchmark, constructs tasks from real issue–pull request pairs and evaluates patches with executable functional tests, driving systems such as SWE- Agent [7] and OpenHands [8]. ∗ Yanlin Wang is the corresponding author. Despite their success, existing repository-level bench- marks largely adopt a single evaluation criterion: a repair is considered successful if the generated patch passes the func- tional test suite which is typically provided with the PR and mainly assesses whether the reported issue has been func- tionally resolved. However, in practice, a patch that resolves the reported issue and passes functional tests is not neces- sarily accepted [10, 11]. During code review, repository maintainers frequently require contributors to satisfy additional requirements beyond functional correctness. We call such requirements review-derived acceptance con- straints (gates): additional, objectively testable require- ments that restrict which functionally correct repairs are acceptable. For brevity, we refer to them as review con- straints hereafter. These requirements are often reflected in review comments that request changes before a patch can be accepted. Examples of such requirements include preserving backward compatibility, maintaining established exception semantics, or following repository-specific implementation conventions [12–14]. For example, a Pydantic pull request sought to add con- ditional serialization for extra fields. Its initial implemen- tation provided the requested behavior by introducing a new core-schema representation, but a reviewer noted that changing this public interface would be breaking and rec- ommended adding an extras_ser_exclude_if pa- rameter instead. 1 The contributor subsequently adopted this backward-compatible design. The two requirements can be tested separately. A func- tional test checks whether qualifying extra fields are omitted from serialized output, whereas a compatibility test checks whether the existing public schema remains unchanged and usable by code written against the previous format. A patch may therefore pass the functional test while failing compat- ibility, showing that implementing the requested behavior does not necessarily satisfy the review constraint raised dur- ing review. Existing benchmarks answer whether a model can generate a functionally correct repair, but not whether that repair also complies with review-derived review constraints. These are separable evaluation dimensions: issue tests establish func- 1 https://github.com/pydantic/pydantic/pull/12657#discussion_ r2671466868 arXiv:2609.04167v1 [cs.SE] 3 Sep 2026 tional success, whereas constraint tests determine whether a functional repair also meets additional acceptance require- ments. Evaluating only the first may therefore overestimate agents’ ability to satisfy complete repair requirements. To bridge this gap, we introduce SWE-Gate, the first repository-level benchmark to evaluate review constraint compliance alongside functional correctness using separate executable tests. Rather than manually inventing rules or at- taching them to unrelated tasks, SWE-Gate derives natural- language constraints from maintainer reviews and synthe- sizes realistic repairs around their engineering intent. Each instance includes a non-compliant patch that passes func- tional tests but violates the constraint and a gold patch that passes both, demonstrating that the dimensions are separable, executable, and jointly satisfiable. Using SWE-Gate, we evaluate representative LLMs on 303 repository-level repair instances spanning 75 reposito- ries across six software domains. Our experiments reveal that many repairs that successfully pass functional tests neverthe- less violate review constraints, demonstrating that functional success alone does not establish that an agent-generated re- pair satisfies the full set of repository requirements or is acceptable for integration. Our contributions are summarized as follows: • We introduce SWE-Gate, the first repository-level soft- ware engineering benchmark that incorporates executable review constraints into realistic software repair tasks. • We propose a dual-dimension evaluation protocol that separately measures functional correctness and review constraint compliance with executable tests. • We develop a semi-automated construction framework and use it to create 303 quality-assured instances across 75 repositories and six domains, then characterize constraint following across four LLM backends evaluated under a common Mini-SWE-Agent scaffold. Related Work Early benchmarks such as HumanEval and MBPP evaluate function-level programs with unit tests [15, 16], whereas Re- poBench, RepoCoder, and CrossCodeEval introduce repos- itory or cross-file context, with subsequent work further studying repository-level code generation[3, 17–19]; mul- tilingual code evaluation and real-fault corpora are further represented by xCodeEval, Defects4J, and BugsInPy [20– 22]. Pre-trained code models have substantially advanced code representation and understanding [23, 24], while sub- sequent work has incorporated richer contextual informa- tion into code intelligence[25–27]. Recent studies further examine context utilization, task-relevant retrieval, and the quality of repository-level agent trajectories, including work on context use and retrieval [28–32] and on selecting high- quality trajectories for more effective agent supervision [33], advancing the study of LLMs for repository-level software engineering tasks. SWE-bench formulates repository-level repair as resolv- ing real GitHub issues with executable tests [6], motivat- ing agents such as SWE-agent, OpenHands, and Agent- less [7–9]. Multi-agent approaches such as MAGIS further explore collaborative issue resolution, while recent repair- agent research also investigates how agents explore and se- lect repair strategies [34, 35], reflecting the broader adop- tion of LLM-based agents in software engineering. Later benchmarks extend languages, enterprise projects, reposi- tory evolution, freshness, contamination control, multimodal issues, and security [36–47], but still primarily evaluate tested functional behavior. Related empirical studies have also highlighted that benchmark design and contextual as- sumptions can substantially affect the realism of LLM-based code-generation evaluation [48]. Scalable task construction has been explored by SWE-smith, which synthesizes test- breaking tasks within repositories [49], and SWE-Mirror, which transfers the semantic essence of real issues across repositories [50]. Recent work further investigates automated construction of software engineering datasets, including au- tomated SWE data construction and LLM-assisted rebuilding of code-intelligence benchmarks [51, 52]. SWE-Gate is in- spired by SWE-Mirror’s cross-repository transfer paradigm but instead transfers review-derived review constraints to- gether with their bug patterns, applicability conditions, and validation requirements, producing separate functional and constraint tests as well as non-compliant and gold repairs. Passing available tests does not necessarily establish patch correctness because weak suites can admit overfitting repairs [10, 53, 54], a problem also observed in SWE-bench eval- uations [11]. Moreover, code reviews address maintainabil- ity, consistency, compatibility, and other engineering con- cerns beyond functional defects [12–14, 55]. These concerns also require engineering knowledge beyond the target code, with project and testing knowledge shown to improve LLM- based test generation [56]. More broadly, surveys of code LLMs and LLM-based software engineering document the limitations of narrow evaluation criteria for complex devel- opment tasks [57, 58]. Constraint-aware code benchmarks have begun evaluating additional requirements [59], while repository-level work attaches design constraints but relies on LLM-based verification [60]; such judges can exhibit subjec- tivity and evaluation biases [61, 62]. In contrast, SWE-Gate constructs every instance around a review-derived constraint and separately executes functional and constraint tests, en- abling deterministic evaluation of both dimensions. SWE-Gate Benchmark SWE-Gate adopts a constraint-first construction strategy be- cause a review-derived constraint is tied to a particular func- tional and code context rather than being a standalone rule that can be appended to an arbitrary issue. Moreover, directly reusing the source issue–pull request pair preserves a public one-to-one correspondence between the task and its known repair. SWE-Gate instead jointly abstracts the bug pattern, engineering intent, and applicability conditions from real re- view discussions and instantiates them in distinct, compatible repository contexts. This strategy preserves the authenticity of the original review requirement, expands a single con- straint into multiple contextually valid tasks across reposito- ries and domains, and mitigates direct solution memorization without claiming to eliminate all training-data contamina- tion. Figure 1 illustrates the resulting construction pipeline. Phase I ConstraintSeed Extraction Phase I Instance Synthesis Phase I Validation Raw Data Collection Seed Repos Patch-Test Validation Repository State F C Original R Pass -- MutantR+M Fail -- Non-comp. R+M+N Pass Fail Gold R+M+G Pass Pass Quality Assurance Remove badinstances patterns ManualReview Released SWE-GateInstances 303 instances across 75 repositories Issue Description Implement Patch Review Comments Raw Data Phase I Instance Synthesis Constraint Construct gold.patch constraint_test.py non-compliant.patch Bug Inject mutant.patch function_test.py R+MFail -- RPass -- State F C Textual Items Generation Issue description Constraint description Generate Tasks Candidate Localization Route Repository Failure Filtering Merged PRs Review Suggestion Mining Extract explicit requirements from code review comments. Constraint Candidate Validation Verify candidate constraints with code evidence to ensure validity. Constraint Normalization Remove repo-specific wording and preserve reviewintent. ×Workflow ×Documentation ×Test-only ×Formatting ×Naming ... Constraint Seed Constraint Abstract review constraint description Functional Context -Original issue -Bug root cause -Related patch -Bug pattern Transfer Metadata -Scenario features -Search keywords -Retrieval cues Source Data -Linked issues -PR data -... R+M+GPass Pass R+M+NPass Fail State F C Constraint Seed Construction LLM-Based Figure 1: Overview of the SWE-Gate construction pipeline. SWE-Gate reconstructs issue–pull request artifacts, uses LLM- based processing to extract atomic review suggestions and select verifiable constraint seeds, transfers the seeds into compatible repository contexts, and validates the resulting instances through executable tests and quality assurance. Instance Schema Instance IdRepo/Base Image Issue: Unsupported data types should raise an explicit error instead of being silently accepted... Constraint: Checks that the raised error message contains the unsupported dtypename @@ -2392,8 +2392,7 @@ def _dtype_to_default_stata_fmt( return "%12.0g" elif dtype in (np.int8, np.int16): return "%8.0g" -else: # pragma : no cover -raise NotImplementedError(f"Data type dtype not supported.") Mutant Patch: def test_unsupported_dtype_should_raise(): df = DataFrame( "complex_col": [np.complex128(1 + 2j)] ) with pytest.raises(NotImplementedError): df.to_stata("/tmp/test.dta", version=117) Functional Test: def test_unsupported_dtype_error_contains_type(): df = DataFrame( "complex_col": [np.complex128(1 + 2j)] ) with pytest.raises(NotImplementedError, match=“complex128”): df.to_stata("/tmp/test.dta", version=117) Constraint Test: @@ -2392,7 +2392,8 @@ def _dtype_to_default_stata_fmt( return "%12.0g" elif dtype in (np.int8, np.int16): return "%8.0g" + else: # pragma : no cover + raise NotImplementedError(f"Data type dtype not supported.") Gold Patch: @@ -2392,7 +2392,8 @@ def _dtype_to_default_stata_fmt( return "%12.0g" elif dtype in (np.int8, np.int16): return "%8.0g" + else: # pragma : no cover + raise NotImplementedError(f"Data type not supported.") Non-Compliant Patch: Figure 2: SWE-Gate benchmark instance schema. Benchmark Instance Each SWE-Gate instance is a repository-level repair task that combines natural-language descriptions, patches, and executable tests to evaluate issue resolution and constraint following (Figure 2). • Issue Description. A natural-language specification of the observed incorrect behavior and expected functional- ity, without revealing the root cause or repair strategy. • Mutant Patch. A patch that injects a reproducible target defect into the original repository while preserving the surrounding engineering context. • Functional Test. An executable test that exposes the de- fect and validates its repair. • Constraint Description. A natural-language engineering requirement that the repair must satisfy beyond functional correctness. • Constraint Test. An executable test that validates com- pliance with the review constraint. • Non-compliant Patch. A reference repair that passes the functional test but violates the constraint. • Gold Patch. A reference repair that passes both the func- tional and constraint tests. Among these artifacts, the non-compliant patch and the gold patch play a central role in the design of SWE-Gate in- stances. The non-compliant patch demonstrates that a patch can fix the issue while violating the review constraint; there- fore, the review constraint is not merely a restatement of the functional requirement. The gold patch demonstrates that the review constraint is satisfiable and can hold simultaneously with the functional repair. Together, they show that constraint compliance is not implied by functional correctness, can be validated separately, and is jointly satisfiable with a func- tional repair. Constraint-First Instance Construction Repository Selection SWE-Gate distinguishes between two types of repository roles: seed repositories and instance repositories. Seed repositories provide engineering knowl- edge from real code review discussions, while instance repos- itories provide the code contexts needed to construct exe- cutable benchmark instances. Seed repositories are mature open-source Python projects. These projects typically have active development communi- ties, rich pull request review histories, and high community adoption. We select such repositories because their review comments are more likely to reflect engineering requirements raised by maintainers during real software development. Because constraints are context-dependent, candidate in- stance repositories are selected from related functionality or domains; for example, data-processing constraints prioritize that ecosystem and CLI constraints prioritize command-line projects. This domain mapping only defines a search space rather than presuming transferability: synthesis must still ver- ify every target location against the seed’s semantic and vali- dation requirements. Representative mappings appear in the supplementary material . Constraint Extraction Before the two LLM-assisted ex- traction stages, SWE-Gate collects merged pull requests from seed repositories through the GitHub API. For each pull re- quest, we retrieve the pull request itself, review comments, linked issues, changed files, and the final unified diff. Inline review comments retain their file paths, line locations, and diff hunks. Together, these artifacts form the raw data used as input to the first stage. Inspired by the atomic-suggestion stage of DesignHunter in SWE-Shield [60], the first-stage LLM extracts only sug- gestions explicitly stated in review comments. It decomposes comments containing multiple requests into atomic records of the reviewer-identified problem, requested change, ratio- nale, category, source identifiers, and confidence, without inferring unstated rules. Rule-based filtering then removes workflow, documentation-only, test-only, formatting, nam- ing, typographical, vague, and otherwise unverifiable re- quests. Linking each retained record to its source comment and original and revised diff context preserves review and implementation traceability. Second, deterministic checks require a linked issue, sub- stantive suggestion, and sufficient diff context. An LLM then evaluates each survivor with its issue, review, and code evi- dence. Candidates are retained only when the issue is user- visible, the diff supports the root cause or repair direction, the review adds rather than restates a requirement, and the governed control flow, API, error behavior, state, or repre- sentation can be validated independently. The LLM must identify a plausible functional but non-compliant repair, jus- tify non-implication and transferability, and propose separate executable oracles. Candidates lacking code support, sepa- rability, or distinct oracles are rejected. Each retained candidate becomes a structured constraint seed that preserves the normalized engineering intent and behavioral scope while removing incidental repository word- ing. It records the constraint, category, rationale, comments, and code evidence; the issue, root cause, relevant change, bug pattern, and an example non-compliant repair; scenario features and retrieval cues for compatible contexts; and non- redundancy, transferability, and separate oracle specifica- tions. Complete issue, review, code, and diff provenance Repository StateNotationFC Original RepositoryRPass – Mutant BugR+ MFail– Non-compliant Repair R+ M+ N Pass Fail Gold RepairR+ M+ G Pass Pass Table 1: Validation matrix for a SWE-Gate instance. Follow- ing the fail-to-pass repair paradigm [6], the original reposi- tory passes F and the mutant one fails it. The non-compliant repair passes F but fails C, whereas the gold repair passes both, establishing constraint separability and satisfiability. remains linked by a stable identifier. At this stage, oracle proposals establish suitability for instantiation; executable tests are realized only after transfer. Instance Synthesis Building on SWE-Mirror’s demon- strated cross-repository transfer paradigm [50], SWE-Gate transfers a constraint seed into a compatible target repository, jointly instantiating its functional bug pattern and review- derived constraint as an executable repair task with separate oracles. The agent first explores the repository’s source files, tests, existing abstractions, and implementation conventions to lo- cate a compatible synthesis anchor. An anchor is a concrete implementation context in which the agent can introduce a user-visible functional failure while preserving an inde- pendently testable review constraint. Candidate anchors are rejected when the constraint is not naturally motivated by the target context, every plausible functional repair would neces- sarily satisfy the constraint, or the functional and constraint dimensions cannot be validated separately. Rather than generating all mutually dependent ar- tifacts at once, the synthesis agent follows a pro- gressive two-phase generate–execute–refine workflow. First, it creates mutant.patch and a functional-only function_test.patch, verifying that the test passes on the original repository and fails on the mutant. It revises the anchor, injected bug, or test until this relationship holds. Second, it creates constraint_test.patch, a natural functional but non-compliant non-compliant.patch, and a compliant gold.patch. The former must pass the functional test and fail the constraint test; the latter must pass both. Execution feedback refines the test and repairs. If a failure shows that the functional task cannot support a sepa- rable violation, the agent revisits the first phase, regenerates dependent artifacts, and reruns the full matrix. This progres- sive construction strategy first establishes a valid functional task and then adds the stricter constraint-compliance relation- ships, reducing the difficulty of producing all interdependent patches and tests correctly in a single generation step. Let F and C denote functional and constraint tests, and R, M, N, and G the original repository, mutant patch, non- compliant repair, and gold repair. Valid instances satisfy Ta- ble 1. Finally, the agent writes issue.md in a realistic user or contributor voice, describing only the visible failure without exposing injected locations, hidden tests, reference repairs, or validation internals. Any included reproduction program must run against the constructed buggy state, al- though this requirement is prompt-guided rather than inde- pendently rule-enforced. The separate constraint.md states the review-derived requirement, remains aligned with its test, and avoids revealing an expected implementation. Quality Assurance Executable validity alone does not exclude artificial, seman- tically inconsistent, or otherwise unsuitable LLM-generated instances. We manually inspected pilot candidates, consol- idated recurring semantic rejection reasons into a failure- pattern taxonomy, and encoded this human-derived taxon- omy as criteria for scalable LLM review. Each candidate first undergoes basic structural checks to ensure that the required artifacts, execution commands, and provenance fields are complete. It is then validated in a con- tainerized environment using the complete validation matrix: the original repository must pass the functional test, the in- jected bug must fail it, the non-compliant repair must pass the functional test but fail the constraint test, and the gold repair must pass both tests. Patch-application failures, execu- tion timeouts, and other infrastructure errors are not treated as expected test failures. This Docker-based validation auto- matically rejects candidates whose functional and constraint behaviors cannot be independently established. Only candidates that pass the Docker validation matrix proceed to LLM-based semantic review. The reviewer re- ceives the constraint seed, source provenance, generated ar- tifacts, and execution results, and applies the rejection cri- teria derived from manual inspection. Common semantic failure patterns include constraints that merely restate the functional issue, constraint descriptions that prescribe a par- ticular API or implementation strategy, constraint tests that enforce behavior not stated in the natural-language descrip- tion, benchmark-specific helpers or abstractions introduced solely to make the constraint testable, constraints that are not naturally motivated by the synthesized scenario, and descrip- tions that expose hidden evaluation artifacts. The complete taxonomy and corresponding rejection rationales are pro- vided in the supplementary material . Finally, all instances retained after LLM-based screening are manually inspected. This final review verifies that the is- sue resembles a realistic user-reported defect, the constraint remains faithful to its review-derived seed and arises natu- rally from the target repository, the issue and constraint are non-redundant, the non-compliant repair represents a natu- ral functional solution, and the gold patch provides a gen- eral repository-consistent repair rather than hard-coding the generated tests. Stage-wise candidate counts and rejection reasons are reported in the supplementary material. Dataset Characteristics The current version of SWE-Gate contains 303 repository- level repair instances covering 75 open-source Python repos- itories. SWE-Gate covers multiple software domains, in- cluding data analysis, web frameworks, testing frameworks, command-line tools, configuration management, symbolic computation, and data validation. This repository diversity enables the benchmark to evaluate review constraints across different code contexts, rather than only measuring local con- ventions in a single project or a single project family. Because a constraint may involve several engineering con- cerns, SWE-Gate uses a multi-label taxonomy. The most fre- quent categories are Error Semantics (152 instances, 50.2%) and Schema / Metadata / Typing (143, 47.2%). The taxonomy also covers ordering and argument preservation, encoding and escaping, scope generalization, compatibility, sentinel distinctions, performance, idempotence, and resource life- cycle requirements. Table 5 reports the complete category counts together with category-level model performance. This distribution shows that code review evaluates interface be- havior, exception semantics, metadata consistency, and other engineering properties beyond whether functional tests pass. Evaluation Experimental Setup We evaluate all 303 instances. For each, an agent receives the target repository and issue description and generates a repair. Its patch is applied to a clean repository containing the injected bug and evaluated by both functional and constraint suites. Any generated changes to benchmark test files are discarded before execution, preventing evaluation leakage and ensuring that only the repair patch is assessed. Under Constraint-Provided (+C), the agent also receives the natural-language constraint; under Constraint-Omitted (−C), it receives only the same repository and issue. In- stances, framework, interaction budget, and hidden func- tional and constraint tests are identical, so the only difference is whether constraint guidance is visible during repair. The +C condition measures intended SWE-Gate performance, while−C is a controlled input ablation. Evaluation Models. We evaluate GPT-5.5, GPT-5.4-mini, DeepSeek-V4-Flash, and GPT-4o-mini, spanning providers and capability levels that support repository-level engineer- ing. All use Mini-SWE-Agent with at most 100 interaction steps. The standard SWE-Bench execution pipeline validates each patch in an isolated container. Evaluation Metrics. We report three complementary met- rics. Let N be the total number of instances, N F the number of generated patches that pass the functional tests, and N F∩C the number that pass both the functional and constraint tests. • Functional Success Rate (FSR). The proportion of all instances for which the generated patch resolves the re- ported functional issue: FSR = N F N . • Constraint Following Rate (CFR). The proportion of functionally successful repairs that also satisfy the review constraint: CFR = N F∩C N F . • Joint Success Rate (JSR). The proportion of all instances for which the generated patch satisfies both the functional requirement and the review constraint: JSR = N F∩C N . ModelF.Pass J.Pass FSR CFR JSR GPT-5.522716074.9 70.5 52.8 GPT-5.4-mini18712061.7 64.2 39.6 DeepSeek-V4-Flash 20213066.7 64.4 42.9 GPT-4o-mini28139.2 46.4 4.3 Table 2: Overall performance in the Constraint-Provided set- ting. Rates are reported as percentages. F. Pass denotes the number of generated patches that pass the functional test suite. J. Pass (Joint Pass) denotes the number of patches that pass both the functional and constraint test suites. JSR is the primary metric of SWE-Gate because it cap- tures complete success under the benchmark’s dual evalu- ation protocol. For the input ablation, we also report the percentage-point difference between the two conditions. For any metric M ∈FSR, CFR, JSR, we define ∆M = M +C − M −C . A positive ∆M indicates a higher observed rate when the review constraint is provided to the agent. RQ1: How Do Coding Agents Perform under SWE-Gate’s Dual Evaluation? Table 2 reports functional and joint performance under the intended +C setting. The models differ substantially in functional repair perfor- mance. GPT-5.5 achieves the highest FSR at 74.9%, followed by DeepSeek-V4-Flash at 66.7% and GPT-5.4-mini at 61.7%. GPT-4o-mini resolves only 9.2% of the instances, showing that SWE-Gate remains difficult for a substantially weaker model. Functional success, however, does not imply joint success. GPT-5.5 produces 227 functionally successful repairs, but only 160 also pass constraint validation. The correspond- ing CFR is 70.5%, meaning that 29.5% of its functionally successful repairs violate the accompanying constraint. The same gap appears for every evaluated model. Failures Hidden by Functional-Only Evaluation. We call a patch that passes functional validation but fails con- straint validation a hidden failure. Such a patch would be accepted under an evaluation protocol that observes only issue-related functional tests. We define the Hidden Failure Rate (HFR) as HFR = N F − N F∩C N F = 1− CFR. Across the four models, 644 generated patches pass the functional tests, but only 423 pass both test suites. SWE- Gate therefore identifies 221 hidden failures, corresponding to 34.3% of all functional successes. The HFR ranges from 29.5% for GPT-5.5 to 53.6% for GPT-4o-mini. These results demonstrate that functional-only evaluation leaves a substan- tial fraction of constraint-violating repairs undetected. More precisely, they show that functional success does not entail review constraint compliance under SWE-Gate’s executable evaluation protocol. ModelF. PassHidden FailuresHFR GPT-5.52276729.5 GPT-5.4-mini1876735.8 DeepSeek-V4-Flash2027235.6 GPT-4o-mini281553.6 Total64422134.3 Table 3: Functionally successful repairs that fail review con- straint validation in the Constraint-Provided setting. Rates are percentages. Answer to RQ1. Current coding agents resolve a substan- tial fraction of SWE-Gate’s functional issues, but their joint success rates are consistently lower than their functional suc- cess rates. Across all evaluated model–instance pairs, 221 of 644 functional successes fail constraint validation. SWE- Gate therefore provides an additional, independently exe- cutable evaluation signal that is not captured by functional tests alone. RQ2: How Does Explicit Constraint Guidance Affect Repair Outcomes? Table 4 compares the controlled input conditions. Providing the constraint increases JSR for every model. The largest gain is observed for GPT-5.5, whose JSR rises from 41.3% to 52.8%, an improvement of 11.5 percentage points. DeepSeek-V4-Flash and GPT-5.4-mini improve by 4.9 and 3.3 points, respectively, while GPT-4o-mini improves by 1.0 point. Across all four models, the number of joint successes increases from 360 to 423. Providing the constraint description also substantially im- proves CFR for all four models. GPT-5.5 achieves the high- est CFR improvement, increasing from 54.6% under the Constraint-Omitted condition to 70.5% under the Constraint- Provided condition. The CFR of GPT-5.4-mini increases from 50.7% to 64.2%, while that of DeepSeek-V4-Flash increases from 54.2% to 64.4%. GPT-4o-mini exhibits the largest relative change, with its CFR increasing from 20.8% to 46.4%. Overall, providing the constraint improves CFR by 10.2–25.6 percentage points across the evaluated mod- els. These results show that, among functionally successful repairs, explicit constraint guidance substantially increases the likelihood of satisfying the corresponding engineering requirement. At the same time, FSR does not improve under the Constraint-Provided condition. It decreases slightly for GPT- 5.5 and by 3.3–9.9 percentage points for the other models. One possible explanation is that satisfying an additional re- quirement increases the complexity of the repair and may steer an agent away from a simpler functionally adequate patch. Because each condition contains one generation per model and instance, these results should be interpreted as an observed trade-off under the controlled input ablation rather than as a general causal claim about model behavior. Answer to RQ2. Explicit constraint descriptions improve the observed joint success rate for all evaluated models and FSRCFRJSR Model−C +C∆ −C +C∆ −C +C∆ GPT-5.575.6 74.9 -0.7 54.6 70.5 +15.9 41.3 52.8 +11.5 GPT-5.4-mini71.6 61.7 -9.9 50.7 64.2 +13.5 36.3 39.6 +3.3 DeepSeek-V4-Flash 70.0 66.7 -3.3 54.2 64.4 +10.2 38.0 42.9 +4.9 GPT-4o-mini15.8 9.2 -6.6 20.8 46.4 +25.6 3.3 4.3 +1.0 Table 4: Constraint input ablation. +C provides the natural-language review constraint;−C omits it while retaining the same hidden functional and constraint tests. ∆ is +C minus−C in percentage points. Constraint CategoryN GPT-5.5GPT-5.4-mini DeepSeek-V4-Flash GPT-4o-mini FSR CFR JSR FSR CFR JSR FSR CFR JSR FSR CFR JSR Error semantics152 75.0 72.8 54.6 67.1 72.5 48.7 70.4 66.4 46.7 9.2 78.6 7.2 Schema / metadata / typing143 72.7 68.3 49.7 62.2 60.7 37.8 64.3 64.1 41.3 11.9 52.9 6.3 Ordering / argument preservation86 60.5 76.9 46.5 51.2 75.0 38.4 62.8 79.6 50.0 7.0 66.7 4.7 Encoding / escaping / quoting74 81.1 68.3 55.4 60.8 51.1 31.1 79.7 55.9 44.6 5.4 0.0 0.0 Scope generalization62 87.1 63.0 54.8 66.1 46.3 30.6 69.4 53.5 37.1 12.9 0.0 0.0 Compatibility / deprecation55 74.5 75.6 56.4 56.4 71.0 40.0 63.6 77.1 49.1 12.7 28.6 3.6 Missing vs. empty / sentinel distinction 51 74.5 81.6 60.8 60.8 74.2 45.1 56.9 75.9 43.1 17.6 66.7 11.8 Performance / structure41 75.6 71.0 53.7 68.3 78.6 53.7 75.6 74.2 56.1 9.8 75.0 7.3 Idempotence / duplicate processing30 60.0 72.2 43.3 46.7 71.4 33.3 46.7 71.4 33.3 0.0 – 0.0 Lifecycle cleanup / resource19 84.2 62.5 52.6 68.4 53.8 36.8 73.7 57.1 42.1 10.5 50.0 5.3 Table 5: Performance by review constraint category in the Constraint-Provided condition. FSR and JSR are calculated over all instances in each category, whereas CFR is calculated over functionally successful repairs. “–” indicates that CFR is undefined because the model has no functional success in that category. Categories are multi-label and therefore not mutually exclusive. substantially increase the fraction of functional repairs that also satisfy constraint validation. However, this improvement is accompanied by lower functional success for three models and a small decrease for GPT-5.5. Constraint information therefore improves compliance and overall joint success in these runs, but does not uniformly improve functional repair. RQ3: Which Review Constraint Categories Remain Challenging? Table 5 analyzes overlapping categories under +C; rows are not mutually exclusive and their counts should not be summed. For the three stronger models, Scope Generaliza- tion has CFRs of 63.0%, 46.3%, and 53.5%, while Lifecycle Cleanup/Resource reaches 62.5%, 53.8%, and 57.1%. These constraints demand coverage beyond the immediate failure or preservation across a resource lifecycle, making them dif- ficult to satisfy with a narrowly localized patch. Encoding/Escaping/Quoting is also difficult for GPT-5.4- mini and DeepSeek-V4-Flash, which satisfy only 51.1% and 55.9% of constraints after functional success, while Schema/Metadata/Typing yields 60.7–68.3% across stronger models. By contrast, Missing-vs.-Empty/Sentinel Distinc- tion reaches 74.2–81.6%, and Ordering/Argument Preser- vation 75.0–79.6%. The larger functional–joint gaps in the former categories therefore reflect conditional difficulty in satisfying the review constraint, not only variation in func- tional repair rates. Model profiles also differ: DeepSeek-V4-Flash leads Or- dering/Argument Preservation at 79.6% CFR, GPT-5.4-mini leads Performance/Structure at 78.6%, and GPT-5.5 reaches 81.6% on Missing-vs.-Empty/Sentinel Distinction. GPT-4o- mini has too few functional successes in most categories for stable conclusions, so its high conditional values should not be interpreted as superior constraint following. Since cate- gories overlap and several are small, all comparisons are de- scriptive, but they show that aggregate scores obscure which engineering requirements agents fail. Answer to RQ3. Constraint-following difficulty is not uniform across engineering requirements. Among func- tionally successful repairs, Scope Generalization, Life- cycle Cleanup/Resource, Encoding/Escaping/Quoting, and Schema/Metadata/Typing yield some of the lowest CFR val- ues for the three stronger models. In contrast, Missing-vs.- Empty/Sentinel Distinction and Ordering/Argument Preser- vation are satisfied more frequently. These results show that aggregate scores hide meaningful differences in the review constraints that agents fail to follow. Conclusion We introduced SWE-Gate, a repository-level benchmark that derives review constraints from real pull request re- views, constructs repair tasks around them, and evaluates constraint compliance separately from functional correct- ness. Across 303 instances from 75 Python repositories, 221 of 644 functionally successful repairs fail to satisfy the pro- vided constraints, showing that functional-only evaluation overestimates agents’ ability to satisfy complete repair re- quirements. Future work should extend SWE-Gate beyond Python and develop reliable evaluation methods for review requirements that cannot yet be expressed as executable tests. Incorporating broader maintainer feedback into instance con- struction could further improve the realism of transferred tasks and reduce potential bias introduced by LLM-assisted synthesis and screening. Acknowledgments This work is supported by the National Natural Science Foun- dation of China (Grant No. 92582202, No. 62302534) References [1] Ensheng Shi, Fengji Zhang, Yanlin Wang, Bei Chen, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. Sotana: The open-source software development assistant, 2023. [2] Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianx- iang Wang, and Zibin Zheng. Agents in software engi- neering: Survey, landscape, and vision, 2024. [3] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali, ed- itors, Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, pages 2471–2484, Singapore, December 2023. Association for Computational Linguistics. [4] Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao, and Zibin Zheng. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitiga- tion. Proceedings of the ACM on Software Engineering, 2(ISSTA):481–503, 2025. [5] Dewu Zheng, Yanlin Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. Top gen- eral performance = top domain performance? domain- codebench: A multi-domain code generation bench- mark, 2025. [6] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models re- solve real-world github issues? In The Twelfth Interna- tional Conference on Learning Representations, 2024. [7] John Yang, Carlos Jimenez, Alexander Wettig, Kil- ian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 50528–50652. Curran Associates, Inc., 2024. [8] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Jun- yang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Rep- resentations, 2025. [9] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents, 2024. [10] Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In Pro- ceedings of the 2015 International Symposium on Soft- ware Testing and Analysis, ISSTA 2015, page 24–36, New York, NY, USA, 2015. Association for Computing Machinery. [11] Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. UT- Boost: Rigorous evaluation of coding agents on SWE- bench. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 3762–3774, Vienna, Austria, July 2025. Association for Computational Linguistics. [12] Alberto Bacchelli and Christian Bird. Expectations, outcomes, and challenges of modern code review. In 2013 35th International Conference on Software Engi- neering (ICSE), pages 712–721, 2013. [13] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. Modern code review: a case study at google. In Proceedings of the 40th International Conference on Software Engineer- ing: Software Engineering in Practice, ICSE-SEIP ’18, page 181–190, New York, NY, USA, 2018. Association for Computing Machinery. [14] Peter C. Rigby and Christian Bird. Convergent contem- porary software peer review practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Soft- ware Engineering, ESEC/FSE 2013, page 202–212, New York, NY, USA, 2013. Association for Computing Machinery. [15] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brock- man, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. [16] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. [17] Tianyang Liu, Canwen Xu, and Julian McAuley. Re- pobench: Benchmarking repository-level code auto- completion systems, 2023. [18] Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code com- pletion. In Thirty-seventh Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track, 2023. [19] Yifan Li, Ensheng Shi, Dewu Zheng, Kefeng Duan, Jiachi Chen, and Yanlin Wang. Repomincoder: Im- proving repository-level code generation based on in- formation loss screening. In Proceedings of the 15th Asia-Pacific Symposium on Internetware, pages 229– 238, 2024. [20] Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual mul- titask benchmark for code understanding, generation, translation and retrieval, 2023. [21] René Just, Darioush Jalali, and Michael D. Ernst. De- fects4j: a database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, page 437–440, New York, NY, USA, 2014. Association for Computing Machinery. [22] Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Fer- dian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Con- ference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, page 1556–1560, New York, NY, USA, 2020. Association for Computing Ma- chinery. [23] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 7212–7225, 2022. [24] Jiayi Lin, Yanlin Wang, Yibiao Yang, Lei Zhang, and Yutao Xie. Towards better code understanding in decoder-only models with contrastive learning. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 40, pages 32006–32014, 2026. [25] Hanyang Guo, Xiangping Chen, Yuan Huang, Yanlin Wang, Xi Ding, Zibin Zheng, Xiaocong Zhou, and Hong-Ning Dai. Snippet comment generation based on code context expansion. ACM Transactions on Soft- ware Engineering and Methodology, 33(1):1–30, 2023. [26] Yanlin Wang, Yanxian Huang, Daya Guo, Hongyu Zhang, and Zibin Zheng. Sparsecoder: Identifier-aware sparse transformer for file-level code summarization. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 614–625. IEEE, 2024. [27] Yanlin Wang, Ensheng Shi, Lun Du, Xiaodi Yang, Yuxuan Hu, Shi Han, Hongyu Zhang, and Dongmei Zhang. Cocosum: Contextual code summarization with multi-relational graph neural network. arXiv preprint arXiv:2107.01933, 2021. [28] Yanlin Wang, Kefeng Duan, Dewu Zheng, Ensheng Shi, Fengji Zhang, Yanli Wang, Jiachi Chen, Xilin Liu, Yuchi Ma, Hongyu Zhang, et al. Towards an un- derstanding of context utilization in code intelligence. ACM Computing Surveys, 58(11):1–43, 2026. [29] Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. Rlcoder: Reinforcement learning for repository-level code com- pletion. In 2025 IEEE/ACM 47th International Con- ference on Software Engineering (ICSE), pages 1140– 1152. IEEE, 2025. [30] Tianyue Jiang, Yanli Wang, Yanlin Wang, Daya Guo, Ensheng Shi, Yuchi Ma, Jiachi Chen, and Zibin Zheng. Aligncoder: Aligning retrieval with target intent for repository-level code completion, 2026. [31] Yanlin Wang, Suiquan Wang, Yanli Wang, Bowen Zhang, Daya Guo, Jiachi Chen, and Zibin Zheng. Re- poreasoner: Evaluating repository-level code reasoning ability of long-context language models. Proceedings of the ACM on Software Engineering, 3(FSE):2790– 2812, 2026. [32] Wenchao Gu, Juntao Chen, Yanlin Wang, Tianyue Jiang, Xingzhe Li, Mingwei Liu, Xilin Liu, Yuchi Ma, and Zibin Zheng. What to retrieve for effective retrieval- augmented code generation? an empirical study and beyond. arXiv preprint arXiv:2503.20589, 2025. [33] Dewu Zheng, Ruizhe Ye, Yanlin Wang, Yang Ye, Hongyu Zhang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jianxing Yu, and Zibin Zheng. Swe-prime: Fewer tra- jectories, better performance, 2026. [34] Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. Magis: Llm- based multi-agent framework for github issue resolu- tion. Advances in Neural Information Processing Sys- tems, 37:51963–51993, 2024. [35] Tianyue Jiang, Yanlin Wang, Xin He, Daya Guo, Ji- achi Chen, Ming Wen, Ensheng Shi, Xilin Liu, Yuchi Ma, and Guanbin Li. Phoenixrepair: Rethinking repair strategy exploration in software agents. arXiv preprint arXiv:2607.18859, 2026. [36] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, RUI LONG, Ming Ding, and liang xiang. Multi-swe- bench: A multilingual benchmark for issue resolving. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Ko- niusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc., 2025. [37] Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, Dezhi Ran, Muhan Zeng, Bo Shen, Pan Bian, Guangtai Liang, Bei Guan, Pengjie Huang, Tao Xie, Yongji Wang, and Qianxiang Wang. Swe-bench-java: A github issue resolving benchmark for java, 2024. [38] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?, 2025. [39] Thomas Joshi, Shayan Chowdhury, and Fatih Uysal. Swe-bench-cl: Continual learning for coding agents, 2025. [40] Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. Swe-bench goes live! In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghas- semi, and N. Chen, editors, Advances in Neural Infor- mation Processing Systems, volume 38. Curran Asso- ciates, Inc., 2025. [41] Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Pro- cessing Systems, volume 38. Curran Associates, Inc., 2025. [42] John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muen- nighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024. [43] Lianghong Guo, Wei Tao, Runhan Jiang, Yanlin Wang, Jiachi Chen, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, and Zibin Zheng. Omnigirl: A mul- tilingual and multimodal benchmark for github issue resolution. Proceedings of the ACM on Software Engi- neering, 2(ISSTA):24–46, 2025. [44] Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. Humanevo: An evolution-aware benchmark for more realistic evaluation of repository-level code generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1372–1384. IEEE, 2025. [45] Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, et al. Repotransbench: A real-world multilingual benchmark for repository-level code translation. arXiv preprint arXiv:2412.17744, 2024. [46] Yanlin Wang, Bowen Zhang, Yanli Wang, Daya Guo, Terry Yue Zhuo, Jiachi Chen, Mingwei Liu, Xingong Zhang, and Zibin Zheng. Arkrepobench: A repository- level code completion benchmark for harmonyos de- velopment. In Findings of the Association for Compu- tational Linguistics: ACL 2026, pages 19409–19429, 2026. [47] Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen, and Zibin Zheng. Realsec-bench: A benchmark for evaluating secure code generation in real-world repositories. In Findings of the Association for Computational Linguis- tics: ACL 2026, pages 35866–35883, 2026. [48] Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. Towards more realistic evaluation of llm-based code generation: an experimental study and beyond. arXiv preprint arXiv:2406.06918, 2024. [49] John Yang, Kilian Lieret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe- smith: Scaling data for software engineering agents. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc., 2025. [50] Junhao Wang, Daoguang Zan, Shulin Xin, Siyao Liu, Yurong Wu, and Kai Shen. Swe-mirror: Scaling issue- resolving datasets by mirroring issues across reposito- ries, 2025. [51] Lianghong Guo, Yanlin Wang, Caihua Li, Wei Tao, Pengyu Yang, Jiachi Chen, Haoyu Song, Duyu Tang, and Zibin Zheng. Swe data construction, automati- cally! Proceedings of the ACM on Software Engineer- ing, 3:525 – 546, 2025. * . [52] Kang Yang, Xinjun Mao, Shangwen Wang, Yanlin Wang, Tanghaoran Zhang, Bo Lin, Yihao Qin, Zhang Zhang, Yao Lu, and Kamal Al-Sabahi. Large language models are qualified benchmark builders: Rebuilding pre-training datasets for advancing code intelligence tasks. In 2025 IEEE/ACM 33rd International Confer- ence on Program Comprehension (ICPC), pages 298– 309, 2025. [53] Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. Is the cure worse than the disease? over- fitting in automated program repair. In Proceedings of the 2015 10th Joint Meeting on Foundations of Soft- ware Engineering, ESEC/FSE 2015, page 532–543, New York, NY, USA, 2015. Association for Computing Machinery. [54] Jinqiu Yang, Alexey Zhikhartsev, Yuefei Liu, and Lin Tan. Better test cases for better automated program repair. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, page 831–841, New York, NY, USA, 2017. As- sociation for Computing Machinery. [55] Moritz Beller, Alberto Bacchelli, Andy Zaidman, and Elmar Juergens. Modern code reviews in open-source projects: which problems do they fix? In Proceedings of the 11th Working Conference on Mining Software Repositories, MSR 2014, page 202–211, New York, NY, USA, 2014. Association for Computing Machin- ery. [56] Anji Li, Mingwei Liu, Zhenxi Chen, Zheng Pei, Zike Li, Dekun Dai, Yanlin Wang, and Zibin Zheng. Knowl- edge matters: Injecting project and testing knowledge into llm-based unit test generation. arXiv preprint arXiv:2511.14224, 2025. [57] Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. A survey of large language models for code: Evolution, benchmarking, and future trends, 2024. [58] Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Ji- achi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. Towards an understanding of large language models in software engineering tasks, 2024. [59] Guoliang Duan, Mingwei Liu, Yanlin Wang, Chong Wang, Xin Peng, and Zibin Zheng. A hierarchical and evolvable benchmark for fine-grained code instruction following with multi-turn feedback, 2025. [60] Kai Yu, Zhenhao Zhou, Junhao Zeng, Ying Wang, Xueying Du, Zhiqiang Yuan, Junwei Liu, Ziyu Zhou, Yujia Wang, Chong Wang, and Xin Peng. Does pass rate tell the whole story? evaluating design con- straint compliance in llm-based issue resolution. ArXiv, abs/2604.05955, 2026. [61] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, ed- itors, Advances in Neural Information Processing Sys- tems, volume 36, pages 46595–46623. Curran Asso- ciates, Inc., 2023. [62] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 9440–9450, Bangkok, Thailand, August 2024. Associ- ation for Computational Linguistics.