Paper deep dive
SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, Weiyi Shang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/1/2026, 1:32:47 AM
Summary
The paper introduces SWE-NFI, a benchmark designed to evaluate coding agents on non-functional improvements (NFIs) such as documentation, error handling, and code style, beyond mere functional correctness. The benchmark consists of 188 tasks derived from real-world Python pull requests and utilizes 92 executable rules for evaluation. The study finds that while agents achieve high functional correctness, they significantly lag behind human developers in performing NFIs, particularly in structural and logic pattern improvements.
Entities (7)
Relation Signals (5)
SWE-NFI → derivedfrom → GitHub
confidence 95% · Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects... constructed from real merged GitHub pull requests
SWE-NFI → evaluates → Coding Agents
confidence 95% · SWE-NFI is a benchmark for evaluating coding agents on NFIs beyond functional correctness.
SWE-NFI → focuseson → Non-Functional Improvements
confidence 95% · SWE-NFI is designed to evaluate whether coding agents can make developer-oriented non-functional improvements beyond functional correctness.
SWE-NFI → uses → 92 Executable Rules
confidence 92% · We operationalize developer-oriented NFIs into 92 executable rules... We evaluate state-of-the-art commercial and open-source coding agents.
Coding Agents → outperformedby → Human Developers
confidence 90% · all evaluated agents generally fall short of human developers in overall NFI capability.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.
Tags
Links
- Source: https://arxiv.org/abs/2607.27409v1
- Canonical: https://arxiv.org/abs/2607.27409v1
Trouble viewing inline? Open PDF directly →
Full Text
79,131 characters extracted from source content.
Expand or collapse full text
SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements Pengyu Xue pengyu13@yorku.ca York University Toronto, Ontario, Canada He Yang Yuan yuanh@yorku.ca York University Toronto, Ontario, Canada Xin Wang xwang496@connect.hkust-gz.edu.cn The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China Junkai Chen junkaichen@smu.edu.sg Singapore Management University Singapore, Singapore Haonan Zhang haonan.zhang@uwaterloo.ca University of Waterloo Waterloo, Ontario, Canada Boyuan Chen chenfsd@gmail.com Independent Researcher Canada Zishuo Ding zishuoding@hkust-gz.edu.cn The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China Zhenhao Li lzhenhao@yorku.ca York University Toronto, Ontario, Canada Weiyi Shang wshang@uwaterloo.ca University of Waterloo Waterloo, Ontario, Canada Abstract Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior- preserving non-functional improvements (NFIs) remains underex- plored. In real-world software development, developers continu- ously improve software quality without changing observable behav- ior, yet existing benchmarks primarily evaluate functional correct- ness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluat- ing coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull re- quests in open-source Python projects. We operationalize developer- oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0% func- tional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents’ NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correct- ness. 1 Introduction Coding agents powered by large language models have become increasingly integrated into modern software engineering work- flows. These agents are capable of assisting developers in a wide range of functionality-oriented software engineering tasks, such as resolving GitHub issues, fixing bugs, implementing feature re- quests [7,9,11,31,34,72,74,85], and supporting emerging vibe coding workflows [10, 55]. In practice, real-world software development and maintenance involve far more than implementing new functionality or fixing bugs. Developers continuously make non-functional improvements (NFIs) to existing code, such as refining documentation [29], im- proving error handling [47,61] and security [45], modernizing library usage [66], enhancing coding style [12,82], constructing system observability [19,35–38,40,67,87,88], and simplifying program structure [52]. Although these changes typically preserve the original functionality, they play a critical role in improving soft- ware maintainability, readability, and long-term evolution [6,46]. Consequently, NFIs constitute an indispensable part of software maintenance. Despite the importance of NFIs, existing benchmarks for cod- ing agents predominantly evaluate functionality-oriented tasks. Representative benchmarks typically evaluate the agents by exe- cuting unit tests or comparing the generated code against expected behavior [5,13,14,31,42]. Such evaluations have substantially advanced the development of coding agents [28,41], but provide limited insights into their ability to perform NFIs. More importantly, functional correctness and NFIs capture complementary aspects of software quality. A functionally correct solution may still exhibit poor documentation, inadequate error handling, outdated library usage, or unnecessarily complex program structure. Since these issues affect maintainability and long-term evolution, evaluating coding agents solely by functional correctness gives an incomplete picture of practical usefulness. To bridge this gap, we present SWE-NFI, a benchmark for sys- tematically evaluating coding agents on NFIs beyond functional correctness. Following the five high-level aspects of non-functional improvements discussed in SWE-IF [86], we derive 92 executable rules from prior studies, real-world issues, and coding guidelines. Based on these rules, we construct 188 benchmark tasks from real merged GitHub pull requests, including 145 single-file tasks and 43 multi-file tasks. Each task is accompanied by a comprehensive ArXiv, 2026, OnlinePengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang evaluation suite consisting of manually implemented functional test cases and rule-based NFI evaluation. Since the benchmark is constructed entirely from real merged pull requests, it captures the non-functional improvements that developers routinely perform during software development and maintenance. We systematically evaluate state-of-the-art commercial and open- source coding agents using SWE-NFI. Our results reveal a substan- tial gap between current coding agents and human developers. Although the best-performing agent achieves a 70.0% functional correctness rate, all evaluated agents mostly fall short of the corre- sponding human reference across multiple NFI aspects, especially for logic patterns, where the average improvement score ranges from 0.0 to 1.3 across all evaluated agents, compared with 1.5 for the human reference. We further find that multi-file tasks remain considerably more challenging than single-file tasks, while repeated executions exhibit relatively stable NFI performance. In addition, we investigate the consistency of coding agents across repeated executions of the same task, as well as the practical costs of making non-functional improvements in terms of execution time and to- ken consumption. These findings suggest that, while coding agents have made substantial progress in generating functionally correct code, enabling them to perform high-quality non-functional im- provements remains an important open challenge. Our main contributions are summarized as follows: • We operationalize NFIs into an executable evaluation frame- work. Specifically, following the five high-level NFI aspects proposed in prior work, we systematically derive 92 executable rules from multiple complementary sources, including prior studies, real-world open-source development practices, and coding guidelines. •We construct SWE-NFI, a benchmark from real merged GitHub pull requests. The benchmark contains 188 NFI tasks, includ- ing 145 single-file and 43 multi-file. Each task is accompanied by a comprehensive evaluation suite consisting of manually implemented functional test cases and rule-based NFI evalua- tion. •We conduct a comprehensive empirical study of state-of-the- art commercial and open-source coding agents on SWE-NFI. Our study provides a systematic evaluation of coding agents on NFIs beyond functional correctness and discusses insights into their effectiveness, robustness, and practical deployment costs. 2 Related Work 2.1 Functional Code Generation Benchmarks In the early days of large language models, code generation bench- marks, such as HumanEval [14] and MBPP [5], generally consist of method-level, self-contained programming problems with several test cases for functional evaluation. Motivated by these works, a series of follow-up benchmarks has emerged, improving upon them in various respects, such as task granularity (e.g., ClassEval [21], RepoEval [83]), language coverage (e.g., MultiPL-E [8], MBXP [4]), domain diversity (e.g., BigCodeBench [89], DS-1000 [32]), and test enhancement (e.g., EvalPlus [42]). In particular, SWE-bench [31] advances the evaluation of code generation to a new stage: given an issue description, the model is required to edit multiple files to resolve it, with the resulting patch verified against comprehensive test suites. This work opens a new direction that more closely mirrors real-world development workflows, and has become the prevailing standard for evaluating the coding capabilities of LLMs and agents. It also inspires a number of works that extend and refine the original SWE-bench (e.g., feature addition [34], multilingual [81], and multimodal [75] extensions, among many others [17,57,79,84]). Very recently, we have also observed that the generation of entire repositories or complete software systems [18,27,76] is nascent but developing rapidly. For example, ProgramBench [76] has attracted considerable attention from the community, which asks LLM agents to reproduce program executables and evaluate their function with agentic black-box fuzzing. Compared with these works focusing on functional evaluation, SWE-NFI proposes five non-functional aspects for code generation evaluation, with rules mined from the open-source community, constituting a complementary and distinctive contribution. 2.2 Code Generation Evaluation Beyond Correctness In addition to functional correctness, a growing number of stud- ies examine other aspects of LLM-generated code. Among these, code efficiency has received considerable attention [20,26,30,39, 43,44,54,56,70]. For example, EffiBench [30] collects efficiency- critical problems from LeetCode and compares the running time of LLM-generated solutions against human-written canonical ones. COFFE [54] uses computationally intensive test inputs and the num- ber of CPU instructions for a more discriminative and precise evalu- ation of code efficiency. The security of code produced by LLMs and agents is another dimension beyond functionality [10,51,53,60]. Pearce et al. [51] pioneer this direction by auditing GitHub Copi- lot completions and find that approximately 40% of them may be vulnerable. SecureVibeBench [10] reconstructs scenarios in which developers introduce security issues and uses them to evaluate agents. Beyond efficiency and security, more specialized aspects of code have been studied, such as portability [23,66,71], compli- ance [73], and fairness [22]. In contrast to the aforementioned research, which typically fo- cuses on evaluating classical code properties such as efficiency and security, SWE-NFI is grounded in real-world coding behaviors and assesses developer-oriented dimensions mined from large-scale open-source development. Moreover, to achieve verifiable and de- terministic evaluation, we codify these abstract, descriptive criteria into reproducible, deterministic rules, rather than relying on less re- liable measurements such as manual assessment or LLM-as-a-judge, which significantly distinguishes SWE-NFI from similar work [86]. 2.3 LLM-Based Agents in Software Engineering LLM-based agents have shown great capabilities in various software engineering tasks, including program repair, vulnerability discov- ery, and software refactoring [28,41]. Bouzenia et al. [7] present the first autonomous LLM agent for program repair, namely RepairA- gent. Built around tool use, this work designs a well-crafted tool set and employs dynamic templates together with a state machine to guide tool invocation for the LLM. AutoCodeRover [85] utilizes SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional ImprovementsArXiv, 2026, Online Stage1:NFI Rules Stage 2: Task ConstructionStage 3: Evaluation Suite Academic Literature GitHub Issues Manual Rule Analysis NFI Qualification NFI Scoring Pull Request Collection PR Cleaning Keyword Matching NFI Rule-base Filtering Data CleaningHuman Validation Construct Task Description Cross Validation NFI Task Dataset (188 tasks) Functional Correctness Evaluation NFI Rule Evaluation Result Analysis RQ1: Performance Comparison RQ2: Scope Impact RQ3: Consistency Analysis RQ4: Practical Cost Coding Documents 92 NFI Rules Threshold Deterministic Figure 1: Overview of SWE-NFI. the reasoning ability of LLMs to retrieve code, localize relevant ele- ments, and generate patches in an iterative manner. SWE-agent [74] proposes the idea of an Agent-Computer Interface, which trans- forms common computer operations into agent-friendly instruc- tions to enable effective iteration between action and feedback. Building on SWE-agent, Abramovich et al. [2] design interactive agent tools and a context summarization strategy tailored for vul- nerability discovery tasks. RefAgent [50] introduces a multi-agent framework for automatic code refactoring, where several special- ized agents are responsible for roles such as planning and testing during the refactoring process. Beyond academia, efforts from industry and the open-source community, exemplified by Claude Code [15], Codex [48], and OpenHands [68], have also joined in releasing and iterating on software engineering agents. As these agents become more capable, the need for a multi-dimensional evaluation beyond functionality alone becomes increasingly pressing, which is precisely what SWE- NFI targets. 3 SWE-NFI Benchmark Framework 3.1 Benchmark Overview SWE-NFI is designed to evaluate whether coding agents can make developer-oriented non-functional improvements beyond func- tional correctness. As shown in Figure 1, SWE-NFI consists of three main stages: (1): NFI Rules: We operationalize five aspects of NFIs into 92 executable rules; (2) Task Construction: We construct 188 benchmark tasks from real-world merged GitHub pull requests; (3) Evaluation Suite: We implement an evaluation suite that com- bines functional correctness tests with NFI rule-based evaluation. We illustrate each component in detail below. 3.2 NFI Rules Non-Functional Improvement Aspects. Non-functional improve- ments cover a broad range of software quality concerns, such as maintainability, performance, security, reliability, and usability. In this work, we focus on developer-oriented, behavior-preserving improvements that are routinely performed during software main- tenance. Specifically, following prior study [86], we study five rep- resentative aspects: Documentation, Error Handling, Library Con- straints, Coding Style, and Logic Patterns. Although these aspects provide a high-level characterization of non-functional improve- ments, they do not define executable evaluation criteria. Therefore, we further operationalize them into concrete rules that can be au- tomatically evaluated. In this study, we focus on Python because it provides mature static-analysis tools and well-established cod- ing standards, enabling reproducible benchmark construction and deterministic evaluation. Rule Derivation. We derive rules from three complementary sources following a similar process to prior studies [69,77,80]: prior studies, open-source software artifacts, and Python coding guidelines. First, we collect studies related to software maintainability and non-functional improvements from major software engineering venues (e.g., ICSE, FSE, ASE) and AI venues (e.g., AAAI, NeurIPS, ACL), published between 2020 and 2025. We retrieve candidate pa- pers using aspect-specific keywords associated with the five NFI aspects, (eg., "static analysis", "linter", "code quality", "refactoring", "code smells", etc.) resulting Coding Style with 95 papers, Logic Patterns with 101 papers, Documentation with 78 papers, Error Handling with 97 papers, and Library Constraints with 39 papers. Following an in-depth manual screening of the literature, we nar- rowed the selection to 35 highly relevant papers that provided the theoretical foundation for our rule derivation. Second, we collect real-world developer discussions from issues in widely used open-source Python projects. We start from the Top-100 starred Python projects in GitHub-Ranking and remove repositories that correspond to tutorials, demos, educational mate- rials, instructions, or curated lists. We then use the GitHub Issue Search API to run aspect-specific keyword queries over the remain- ing repositories, restricting the search to issues created between 2020 and 2025 and removing duplicates by URL. To focus on dis- cussions with clear developer engagement, we keep only issues with at least 10 reactions and comments in total. This procedure ArXiv, 2026, OnlinePengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang Table 1: Summary of the 92 executable rules for Non- Functional Improvements evaluation. Aspect DeterministicThreshold-based Total RequiredProhibitedQuantitative Documentation (Doc)124319 Error Handling (EH)46212 Library Constraints (LC) 119222 Coding Style (CS)710825 Logic Patterns (LP)08614 Total24472192 yields 326 issues across five aspects: Coding Style (54), Logic Pat- terns (21), Documentation (104), Error Handling (31), and Library Constraints (116). These issues, together with the literature and coding guidelines, inform the candidate NFI rules. Third, we incorporate coding guidelines that define recommended Python development practices. We systematically categorize and synthesize these guidelines into rules corresponding to our five eval- uation aspects. Our rule set integrates standards from PEP 8 [65], PEP 3134 [78], PEP 585 [33], PEP 257 [24], and PEP 484 [58]. We also incorporate best practices from reputable technical sources, including the Tufts University coding standards [64], the pyupgrade documentation [62], and industry-standard metrics [59]. Each de- terministic and threshold-based rule is derived from these official documentations and specialized technical resources to ensure that our benchmark accurately reflects established professional coding standards. Manual Analysis and Refinement. We then manually analyze the collected artifacts to identify concrete code improvement rules that can be mapped to the five NFI aspects. During the analysis, we focus on rules that satisfy two requirements. First, they should represent practices that frequently appear in prior studies or real- world software improvements. Second, they should be measurable through static analysis without requiring evaluation-time human annotation or LLM-based judgment. The first author initially extracted candidate rules from the col- lected data. The extracted rules were then reviewed and refined by a second author. Disagreements were resolved through discussion until a consensus was reached. Rules Summary. Through this process, we derive 92 executable rules across the five NFI aspects. These rules transform high-level NFI aspects into concrete and measurable evaluation targets. To provide a clear overview of our evaluation mechanism, Table 1 reports the number of rules in each aspect, and Table 2 summarizes representative rules for each NFI aspect, categorized by their type of measurement and requirement. Due to space constraints, we list a portion of example rules in the table; the full list of all 92 rules, including specific threshold parameters and static analysis configurations, is provided in our replication package [1]. NFI Rules Overview. Through this process, we derive 92 exe- cutable rules across the five NFI aspects. We further organize these rules into two categories according to their measurement charac- teristics: Deterministic Rules and Threshold-based Rules: •Deterministic rules correspond to rules that can be directly verified using static analysis without additional calibration. These rules are further divided into Required Rules and Prohib- ited Rules. Required rules describe coding practices that should be present in the generated code (e.g., function names should follow snake_case), whereas prohibited rules describe undesir- able patterns that should be avoided (e.g., wildcard imports should not be used). Each deterministic rule is converted into a binary indicator (i.e., 1 if the code complies with the rule and 0 otherwise). •Threshold-based Rules quantify code quality using numeri- cal thresholds. Unlike deterministic rules, these rules require comparing a measured value against a predefined threshold. Examples include maximum function length, maximum nest- ing depth, cyclomatic complexity, and the number of parame- ters per function. These thresholds are configurable and can be adjusted for different projects or coding standards. In our benchmark, we adopt threshold values based on widely ac- cepted Python coding guidelines and static analysis recom- mendations to reflect common software engineering practices. Non-Functional Improvements Quantification. For each NFI aspect푝, let푃 푝 (푐)denote the number of rules satisfied by code version푐under aspect푝, and let푇 푝 denote the total number of rules for aspect푝. We first define the normalized rule score of code version 푐 as: 푆 푝 (푐)= 푃 푝 (푐) 푇 푝 .(1) Therefore, 푆 푝 (푐) ∈ [0, 1]. Given two code versions푐 before and푐 after , we define the normal- ized NFI improvement under aspect 푝 as: 퐼 푝 (푐 before ,푐 after )= 푆 푝 (푐 after )− 푆 푝 (푐 before ) = 푃 푝 (푐 after )− 푃 푝 (푐 before ) 푇 푝 . (2) A positive value indicates an improvement under NFI aspect푝, zero indicates no measurable change, and a negative value indicates a regression. For benchmark instance푖, let푐 푖 before denote the original code and let푐 푖 after (푥)denote the completed code produced by source푥. For dataset instances,푥=ℎdenotes the human reference implementa- tion; during agent evaluation,푥= 푎denotes the output generated by agent푎. We compute the NFI improvement of source푥under aspect 푝 as: 퐼 푖,푝 (푥)= 퐼 푝 푐 푖 before , 푐 푖 after (푥) .(3) We report all NFI improvement values after scaling퐼 푖,푝 (푥)to a percentage scale. Here,퐼 푖,푝 (ℎ)denotes the human reference im- provement, and퐼 푖,푝 (푎)denotes the improvement achieved by agent 푎. 3.3 Benchmark Dataset Construction We construct the dataset of SWE-NFI through a multi-stage pipeline that progressively transforms raw pull request data into benchmark tasks. Starting from real-world merged pull requests, we first collect candidate records from production-grade Python repositories and identify developer-driven non-functional improvements using the SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional ImprovementsArXiv, 2026, Online Table 2: Rules catalogue for the five NFI aspects. AspectMeasurementRequirementExample / ThresholdExplanation Documentation DeterministicRequiredModules, classes, and methodsDocstring presence enforcement DeterministicProhibitedNo empty docstrings, no TODOsContent quality verification Threshold-basedQuantitativeDocstring/Type hint coverage≥ 80%Documentation coverage targets Error Handling DeterministicRequiredMandatory ‘raise X from e’Exception chaining enforcement DeterministicProhibitedNo bare ‘except:’, no silent dropsSafety and exception robustness Threshold-basedQuantitativeTry-block LOC ratio≤ 0.5, handlers≤ 3Exception scope management Library Constraints DeterministicRequiredUse pathlib, collections.abcModern API migration DeterministicProhibitedNo tf.Session, np.bool, .ix[]Legacy API prevention Threshold-basedQuantitativeDeprecated API count≤ 2 per fileAPI usage frequency monitoring Coding Style DeterministicRequired snake_case, PascalCase, Line break after operators, UPPER_CASENaming conventions enforcement DeterministicProhibitedNo trailing whitespace, no mixed tabsFormatting standards maintenance Threshold-basedQuantitativeLine length≤ 100, magic numbers≤ 5Complexity & style density control Logic Patterns DeterministicRequiredAssertions for critical logicState validation requirement DeterministicProhibitedNo mutable default args, no ‘while True’Anti-pattern avoidance Threshold-basedQuantitativeCyclomatic complexity≤ 10, depth≤ 3Complexity and logic density aspect-specific rule checkers introduced in Section 3.2. We then apply automatic filtering and human validation to remove noisy, ambiguous, and unsuitable samples. Next, we manually construct task descriptions that preserve the original development intent while preventing benchmark information leakage. Overall, SWE- NFI contains 188 benchmark tasks, including 145 single-file tasks and 43 multi-file tasks, constructed from real-world merged pull requests. Step 1: Real-world Pull Request Collection. We construct SWE- NFI from real-world merged pull requests in production-grade Python repositories. We first collect repositories using the GitHub Search API and filter them by programming language, repository status, and popularity. We exclude forks, archived repositories, tu- torials, demos, educational projects, and curated lists, yielding 100 production-grade Python repositories, each with at least 45,000 stars (median 66,254). We then collect merged pull requests from 2020 to 2025. To focus on developer-driven non-functional improvements, we remove bot-generated pull requests, dependency updates, re- lease pull requests, version bumps, and revert commits. We further apply aspect-specific keyword matching to increase the likelihood of collecting modifications related to the five NFI aspects. The basic unit of data collection is a PR-file pair, where each record corre- sponds to one merged pull request and one modified Python file. After deduplication, we obtain 226,955 raw records. Step 2: NFI Rule-Based Filtering. We next identify records with measurable non-functional improvements by computing the NFI improvement score using Eq.(3). We retain only records satisfying 퐼 푖,푝 (ℎ)>0. This process yields 6,775 positive-improvement records, 211,914 records without measurable changes, 7,566 regressions, and 700 scorer errors. Only the positive-improvement records are retained for subsequent benchmark construction. Step 3: Data Cleaning and Human Validation. We further clean and validate the remaining records to construct high-quality benchmark candidates. We first apply automatic filters, includ- ing repository blocklists, diff size and file length constraints, per- repository caps, and SHA-256 deduplication, resulting in 1,215 can- didate records for manual review. Two authors independently review each candidate to verify that it is focused and represents a genuine non-functional improvement. We further exclude instances requiring substantial external context, large-scale migrations, public API changes, or semantic changes. Disagreements are resolved through discussion until consensus is reached (휅=0.71). This process yields 484 validated instances for subsequent benchmark construction. Step 4: Task Construction. For each validated instance, we man- ually construct a natural-language task description that converts the original pull request into a developer-facing request. To pre- serve benchmark integrity, the description omits the final aspect, executable rules, code patches, and the reference implementation while preserving the original development intent. One author drafts each task description based on the pull request context and code changes, and a second author verifies and refines it for clarity, self-containment, and fidelity. Disagreements are resolved through discussion until consensus is reached. The benchmark contains both single-file and multi-file tasks. Single-file tasks modify one file, whereas multi-file tasks require coordinated modifications across related files sharing a common improvement objective. After a final selection that discards non-self-contained or untestable instances, we obtain the final 188 tasks. 3.4 Evaluation Suite The evaluation suite of SWE-NFI consists of two complementary components: functional correctness and NFI rule-score evaluation. We first verify whether the generated code preserves the intended functionality. NFI evaluation is performed only on valid, function- ally correct outputs. The entire process is fully automated and does not rely on LLM-as-a-judge or human annotation. Functional Correctness Evaluation. For each benchmark task, we manually implement unit test cases to verify functional correct- ness. One author drafts the tests, and a second author verifies and refines them to ensure comprehensive coverage of the intended functionality. Disagreements are resolved through discussion until consensus is reached. We validate the evaluation suite by executing all test cases on the human reference implementation, where every benchmark instance passes successfully. ArXiv, 2026, OnlinePengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang NFI Rule-Score Evaluation. We implement rule-based check- ers using Ruff, Radon, AST-based analyses, and custom scripts to evaluate the 92 executable rules introduced in Section 3.2. Ruff detects coding-style violations and library usage issues, Radon mea- sures structural properties (e.g., cyclomatic complexity and function length), AST analyses identify Python-specific patterns, and custom scripts implement rules unsupported by existing tools. Benchmark Summary: SWE-NFI operationalizes five NFI aspects into 92 executable rules and contains 188 real-world tasks, including 145 single-file and 43 multi-file tasks. Each task is accompanied by a comprehensive evaluation suite combining functional correctness tests and rule-based NFI evaluation. 4 Experimental Setup 4.1 Studied Agents We evaluate eight coding agent configurations, including two com- mercial coding agents and two open-source coding agents paired with three different foundation models. Commercial Coding Agents. We study OpenAI Codex [48] and Claude Code Agent [15], two commercial coding agents designed for software engineering tasks. In our experiments, Codex is con- figured with gpt-5.3-codex, while Claude Code Agent uses claude- sonnet-4-6. Open-source Coding Agents. We follow prior work [10] and study two open-source coding agents, Aider [3] and mini-SWE- agent [63], the officially recommended successor to SWE-agent [74]. To examine the influence of different foundation models, we equip both agents with the same three backbone models: gpt-4.1 [49], gemini-2.5-flash [25], and deepseek-chat [16]. 4.2 Experimental Protocol To reduce the impact of stochasticity, we set the temperature to zero whenever the corresponding agent supports temperature con- trol. For each agent configuration and each benchmark task, we repeat the experiment five times and report the average results across all runs. All agents are given the same task input, consisting of the natural-language task description and the corresponding code_before. Agents are expected to generate the complete revised code_after. The human reference implementation, i.e., the human- written code_after extracted from real pull requests, is never ex- posed to the agents and is only used to compute the human refer- ence improvement 퐼 푖,푝 (ℎ). 4.3 Results Analysis Protocol Output Validity. Before evaluating the code generated by agents, we first verify whether the generated output is valid. An output is considered valid if it is non-empty and syntactically parseable Python. We report output validity first because an agent that fails to produce valid code cannot be meaningfully evaluated by our evaluation suites. We report valid-output rate, which measures the fraction of agent runs where the generated output is both non-empty and syntacti- cally valid Python. Agent runs with empty output or syntactically invalid Python output are excluded from subsequent functional and NFI rule-score analysis. Functional Correctness. We then execute the manually imple- mented unit tests introduced in Section 3.4. A valid agent run is considered functionally correct if the generated code passes all corresponding test cases. Since SWE-NFI focuses on non-functional improvements beyond functional correctness, NFI rule-score analy- sis is performed only on functionally correct agent runs. NFI Improvement. For each valid and functionally correct agent run, we compute the NFI improvement score under the target as- pect푝using Eq.(3). In this evaluation setting, the source푥in Eq.(3) is instantiated as the evaluated agent푎, and푐 푖 after (푎)denotes the evaluated output generated by that agent. The human reference im- provement퐼 푖,푝 (ℎ)is pre-computed from the dataset by instantiating 푥=ℎ, where푐 푖 after (ℎ)denotes the human reference implementation. Both quantities are computed relative to the same code_before, and the human reference implementation is never exposed to agents. Aggregate Analysis. We aggregate NFI results over valid and func- tionally correct agent runs. Unless otherwise specified, reported NFI values are average improvements computed from퐼 푖,푝 (푎)in Eq.(3)and scaled to a percentage. For aspect-level analysis, we compute averages separately for each NFI aspect. For subset-level analysis, such as single-file versus multi-file tasks, we compute averages over the corresponding evaluated outputs. If an agent has no valid and functionally correct outputs for a given subset, its NFI improvement for that subset is removed from calculation rather than treated as zero. 5 Results In this section, we systematically evaluate coding agents on SWE- NFI and analyze their ability to perform tasks on non-functional improvements. Specifically, we investigate the following research questions: •RQ1: How well do coding agents perform on non-functional improvements compared with human reference improvements? •RQ2: How does task scope affect the ability of coding agents to make non-functional improvements? •RQ3: How consistent are coding agents in producing non- functional improvements across repeated runs? •RQ4: What is the practical cost of making non-functional improvements in terms of execution time and token consump- tion? RQ1: How well do coding agents perform on non-functional improvements compared with human reference improvements? Motivation. Modern coding agents have demonstrated strong ca- pabilities in achieving functional correctness. However, it remains unclear whether these agents can also produce measurable non- functional improvements. In professional software development, these aspects are critical for long-term maintainability and usability. RQ1 evaluates whether agent-generated revisions produce valid, SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional ImprovementsArXiv, 2026, Online Table 3: Performance comparison of coding agents on non-functional improvements and functional correctness (RQ1). AgentValid (%)Func. Pass (%)DocEHLCCSLP Codex99.364.86.8 − 2.1 + 1.6 + 2.0 + 1.3 − Claude Code Agent93.770.07.4 − 2.1 + 1.4 + 1.0 − 0.7 − Aider + GPT-4.196.466.3 8.1 + 2.3 + −1.1 − 0.1 − 0.9 − Aider + DeepSeek73.967.06.0 − 2.5 + 0.5 − 1.2 − 0.7 − Aider + Gemini 2.595.859.17.0 − 2.1 + −0.5 − 0.6 − 0.7 − Mini-SWE + GPT-4.198.44.80.00.00.00.00.0 Mini-SWE + DeepSeek83.811.52.7 − 1.6 − 2.1 + 1.0 − 0.6 − Mini-SWE + Gemini 2.591.15.30.00.00.5 − 0.9 − 0.0 Human Reference100.0100.08.02.01.11.51.5 Table 4: Average NFI improvement by task scope (RQ2). Single-file (S) vs. multi-file (M) tasks. Agent Improvement (Single-file %) Improvement (Multi-file %)S− M (Difference %) DocEHLCCSLPAvg.DocEHLCCSLPAvg.DocEHLCCSLPAvg. Codex6.82.31.22.01.52.86.81.13.31.90.42.70.0+1.2−2.1+0.1+1.1+0.1 Claude Code Agent6.92.41.11.10.72.4 9.30.72.50.70.52.8−2.4+1.7−1.4+0.4+0.1−0.3 Aider + GPT-4.17.92.6 −1.10.40.92.1 9.21.1 −1.3 −1.00.61.7−1.3+1.4+0.2+1.4+0.3+0.4 Aider + DeepSeek6.12.70.31.40.72.25.61.51.70.70.42.0+0.4+1.3−1.4+0.7+0.3+0.3 Aider + Gemini 2.56.42.3 −0.50.60.91.99.91.20.10.4 −0.12.3−3.5+1.1 −0.5+0.2+1.0−0.3 Mini-SWE + GPT-4.10.00.00.00.00.00.0 – Mini-SWE + DeepSeek2.42.01.91.10.91.74.00.82.70.90.01.7−1.6+1.2−0.8+0.2+0.90.0 Mini-SWE + Gemini 2.50.00.00.50.90.00.3– functionally correct, and measurable NFI. We compare these aspect- level improvements with human-reference improvements from real pull requests. Approach. For RQ1, we evaluate each agent on the 188 tasks in SWE-NFI. We first report valid output rate and functional pass rate to separate executable outputs from functionally correct revisions. For NFI performance, each aspect column in Table 3 reports the average NFI improvement defined in Eq.(3). For an agent푎, the reported value is the average of퐼 푖,푝 (푎)over evaluated outputs under aspect푝. For the Human row, the reported value is the average of 퐼 푖,푝 (ℎ), where code_after denotes the human reference implementa- tion. Results. Table 3 shows the results of this RQ. For each aspect, a “+” or “-” following the number indicates whether an agent’s average improvement is above or below the corresponding human reference improvement for that aspect. Overall, current coding agents remain substantially behind human developers in making non-functional improvements. Although the best-performing agent achieves a functional correctness rate of 70.0%, most agent configurations fail to match the corresponding human improvements across the five NFI aspects. Among the evaluated aspects, logic patterns exhibit the largest gap: all evaluated agents remain below the human reference (0.0 - 1.3 vs. 1.5). These results suggest that, while current coding agents can successfully preserve functionality, they are still limited in performing the structural code improvements routinely made by human developers. Despite this overall gap, several agents demonstrate competitive performance on specific aspects. Codex achieves the highest valid- output rate (99.3%) and exceeds the human reference on coding style, error handling, and library constraints. Claude Code Agent achieves the highest functional correctness rate (70.0%) and exceeds the human reference on error handling and library constraints. Aider + GPT-4.1 achieves the highest documentation improvement (8.1 vs. 8.0) and also exceeds the human reference on error handling. Performance also varies considerably across NFI aspects. Doc- umentation and error handling are the strongest aspects for most configurations, whereas logic-pattern improvements consistently remain below the human reference. Library-constraint improve- ments show larger variation across agents: Codex, Claude Code Agent, and Mini-SWE + DeepSeek outperform the human refer- ence, whereas Aider + GPT-4.1 and Aider + Gemini 2.5 produce negative average improvements. The Mini-SWE configurations re- veal a functional-correctness bottleneck. Although they maintain relatively high valid-output rates (83.8% - 98.4%), their functional correctness rates remain between 4.8% and 11.5%, which leaves few outputs eligible for subsequent NFI evaluation. RQ1 Summary: Current coding agents remain substantially behind human developers in making non-functional improve- ments. Especially, all evaluated agents consistently underper- form on logic-pattern improvements (0.0 - 1.3 vs. 1.5), which suggests that structural code improvements remain a major challenge. RQ2: How does task scope affect the ability of coding agents to make non-functional improvements? Motivation. Real-world non-functional improvement tasks vary in scope. Some require localized modifications within a single file, whereas others involve coordinated changes across multiple files. Since modern coding agents are increasingly expected to handle repository-level development tasks, it is important to understand whether task scope influences their ability to make non-functional ArXiv, 2026, OnlinePengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang improvements. RQ2 compares agent performance on single-file and multi-file tasks to investigate how task scope affects NFI improve- ments across different quality aspects. Approach. We split SWE-NFI into its single-file and multi-file subsets. For each agent and each subset, we compute the average NFI improvement using Eq.(3)with푥= 푎. The reported values are computed over valid and functionally correct evaluated outputs only. For multi-file tasks, rule scores are aggregated across the modified files before computing task-level NFI improvement. Table 4 reports aspect-level improvements and their average across aspects. The S−M columns subtract the multi-file average from the single- file average. Negative values indicate higher average improvement on multi-file tasks, while positive values indicate higher average improvement on single-file tasks. Results. Table 4 shows that multi-file tasks remain more challeng- ing for current coding agents. Overall, the effect of task scope varies across NFI aspects. Documentation occasionally benefits from multi- file tasks (e.g., Claude Code Agent: 6.9→9.3 and Aider + Gemini 2.5: 6.4→9.9), suggesting that adding documentation across related files is relatively straightforward. In contrast, error-handling and logic- pattern improvements consistently decrease on multi-file tasks across all evaluated agents. For example, Codex’s logic-pattern improvement drops from 1.5 to 0.4, while Claude Code Agent’s error-handling improvement decreases from 2.4 to 0.7. Notably, the two Mini-SWE configurations paired with GPT-4.1 and Gemini-2.5 fail to produce functionally correct outputs on multi-file tasks and are therefore excluded from the comparison. These results suggest that current coding agents can often extend localized improvements such as documentation across multiple files, but still struggle to coordinate more complex non-functional improvements involving multiple related files. RQ2 Summary: Overall, multi-file tasks remain more chal- lenging than single-file tasks for current coding agents. While documentation occasionally benefits from additional cross-file context, multi-file tasks consistently reduce improvements in coding style (up to 1.4), error handling (up to 1.7), and logic patterns (up to 1.1). RQ3: How consistent are coding agents in making non-functional improvements across repeated executions of the same task? Motivation. Coding agents driven by large language models may exhibit stochastic behavior across repeated executions, even un- der the same benchmark task and agent configuration. For soft- ware maintenance tasks, stable behavior matters at multiple stages: producing valid outputs, preserving functional correctness, and achieving measurable non-functional improvement. RQ3 evaluates whether these stages remain stable across repeated runs. Approach. For RQ3, we use five repeated executions of each agent configuration to measure run-to-run consistency. We analyze three metrics: valid-output rate, functional pass rate, and aggregate agent NFI improvement. The aggregate improvement, denoted as퐼 NFI (푎), is computed using Eq.(3)with푥= 푎, over valid and functionally Table 5: Results over five repeated runs (RQ3). AgentMetricRun 1 Run 2 Run 3 Run 4 Run 5 Avg. Std. Codex Valid (%)98.4100.099.598.999.599.3 0.5 Func. Pass (%)62.768.663.664.564.764.8 2.0 퐼 NFI (푎)12.412.513.412.612.612.7 0.4 Claude Code Agent Valid (%)97.392.094.092.093.193.7 2.0 Func. Pass (%)67.171.069.870.171.970.0 1.6 퐼 NFI (푎)13.113.112.812.813.413.0 0.2 Aider+GPT Valid (%)96.396.895.796.896.396.4 0.4 Func. Pass (%)66.167.467.165.865.066.3 0.9 퐼 NFI (푎)13.713.713.313.314.013.6 0.3 Aider+DS Valid (%)76.074.072.975.671.373.9 1.8 Func. Pass (%)66.269.467.965.066.367.0 1.5 퐼 NFI (푎)11.611.712.011.511.811.7 0.1 Aider+Gemini Valid (%)96.896.395.394.795.895.8 0.7 Func. Pass (%)59.259.460.258.558.359.1 0.7 퐼 NFI (푎)11.712.411.912.012.912.2 0.4 Mini-SWE+GPT Valid (%)98.498.498.498.498.498.4 0.0 Func. Pass (%)4.84.84.84.84.84.80.0 퐼 NFI (푎)0.00.00.00.00.00.00.0 Mini-SWE+DS Valid (%)79.887.685.785.780.383.8 3.1 Func. Pass (%)10.712.011.113.89.811.5 1.4 퐼 NFI (푎)8.66.65.57.85.06.71.4 Mini-SWE+Gemini Valid (%)91.590.491.592.589.491.1 1.1 Func. Pass (%)5.15.15.64.95.75.30.3 퐼 NFI (푎)0.90.90.80.90.80.90.0 correct evaluated outputs. For each agent and metric, we report the five run values, their average, and standard deviation. Lower standard deviation indicates more stable behavior across repeated runs on the same tasks. Results. Table 5 shows that NFI improvements are generally stable across repeated runs. The best-performing configurations, includ- ing Codex, Claude Code Agent, and the three Aider-based agents, all exhibit relatively low standard deviations in퐼 NFI (푎)below 0.4, indicating that their measured NFI improvements remain consistent once outputs are valid and functionally correct. Greater variation appears before NFI evaluation. Valid-output and functional-pass rates fluctuate more noticeably for Aider + DeepSeek and Mini-SWE + DeepSeek. The results suggest that repeated executions primarily affect whether outputs become func- tionally correct and eligible for NFI evaluation rather than the mea- sured NFI improvements themselves. Finally, consistency should be interpreted together with effectiveness. For example, Mini-SWE + GPT and Mini-SWE + Gemini show almost no variation across runs, but their functional pass rates remain below 6% and result in little or no measurable NFI improvement. RQ3 Summary: Aggregate NFI improvements remain con- sistent across repeated runs once outputs pass validity and functional-correctness checks. Most run-to-run variation arises before NFI evaluation, primarily from valid-output and func- tional pass rates. High consistency alone does not necessarily imply effective non-functional improvements. SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional ImprovementsArXiv, 2026, Online RQ4: What is the practical cost of making non-functional improvements in terms of execution time and token consumption? Motivation. Beyond NFI performance, the practical utility of cod- ing agents also depends on their computational cost. An agent that achieves strong NFI improvements may require substantially more execution time or token consumption than alternative configura- tions. RQ4 evaluates the computational cost of different coding agents and examines the relationship between resource consump- tion and NFI performance. Approach. For each agent configuration, we measure two computational- cost metrics: execution time and token consumption. Execution time is measured from task initiation to completion, including model inference and agent execution overhead. Token consump- tion is computed as the total number of input and output tokens used during the interaction. We report the average values across all valid runs and compare them with the NFI performance reported in previous RQs. Results. Figure 2 shows that per-run execution time and token consumption vary by orders of magnitude across agent configura- tions. Claude Code Agent has the highest cost, averaging 14,330 seconds and 50.7 million tokens per run, followed by Mini-SWE + DeepSeek with 11,780 seconds and 32.1 million tokens. In contrast, Codex and the Aider configurations require fewer than one million tokens per run on average, while Mini-SWE + GPT-4.1 has the lowest measured cost, with 816 seconds and 327 thousand tokens. However, higher computational cost does not necessarily trans- late into proportionally better NFI performance. Claude Code Agent consumes approximately 100×more tokens than Codex and over 50×more than Aider + GPT-4.1, yet its functional pass rate as shown in Table 3 is only 5.2 and 3.7 percentage points higher, respectively. Moreover, Aider + GPT-4.1 achieves higher documentation (8.1 vs. 7.4) and error-handling improvements (2.3 vs. 2.1), while Codex outperforms Claude Code Agent in coding style (2.0 vs. 1.0) and library constraints (1.6 vs. 1.4). These results suggest diminishing returns in computational cost, where substantially higher resource consumption does not necessarily yield proportional gains in non- functional improvements. The results also show that computational cost is not explained by a simple commercial versus open-source distinction. Claude Code Agent is substantially more expensive than Codex despite both being commercial agents, while the open-source configurations exhibit a wide spectrum of resource consumption, ranging from the lightweight Aider framework to the considerably more expensive Mini-SWE framework. Overall, resource efficiency should be evalu- ated together with functional correctness and NFI performance. RQ4 Summary: Higher computational cost does not necessar- ily lead to proportionally better non-functional improvements. The cost pattern is also not explained by a simple commercial vs. open-source distinction. Resource cost should therefore be interpreted together with validity, functional correctness, and aggregate NFI improvement. 2450 14330 2854 3680 3654 816 11780 5801 504,160 50,741,364 991,751 438,519 870,624 327,391 32,139,360 7,518,460 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 100,000,000 1 10 100 1000 10000 100000 Codex (gpt-5.3) Claude Agent (sonnet-4-6) Aider + GPT- 4.1 Aider + DeepSeek Aider + Gemini 2.5 Mini-SWE + GPT-4.1 Mini-SWE + DeepSeek Mini-SWE + Gemini 2.5 Tokens Time(s) Agents Evaluation of Execution Time and Token Usage Across Different Agents Avg Time (s)Avg Token Figure 2: Execution time and token usage across agents Task description Add type annotations and docstrings to g4f/debug.py to improve clarity for maintainers and static analysis tools, without changing runtime behavior. Code before ... log_handler: callable = print logs: list = [] def log(text, file = None): if logging: log_handler(text, file=file) ... Sample 1 Claude r1| NFI_Documentation+21.05% ... log_handler: Callable[..., None] = print """Callable used to emit log messages.""" logs: list[str] = [] """Accumulated log messages.""" def log(text: str, file: Optional[TextIO] = None) -> None: """Write *text* via ``log_handler`` if enabled. The optional ``file`` stream is forwarded to ``log_handler`` (e.g. ``sys.stderr``). """ if logging: log_handler(text, file=file) ... Sample2 Codex r1 | NFI_Documentation+10.53% ... log_handler: Callable[..., None] = print logs: list = [] def log(text: str, file: Optional[TextIO] = None) -> None: """Log text through the configured handler when logging is enabled.""" if logging: log_handler(text, file=file) ... Figure 3: Comparison of different Agent outputs 6 Discussion 6.1 Case Study Figure 3 shows a task from SWE-NFI that is related to improving the aspect of Documentation. The task requires adding type annota- tions and docstrings to improve maintainability and static-analysis support without changing runtime behavior. For readability, the fig- ure shows abbreviated versions of the task description and source code. Both agents successfully improve the original implementation, but to different extents. Claude provides more comprehensive docu- mentation, introduces more precise type annotations (e.g., list[str]), and documents the logging behavior in greater detail, resulting in a 21.05 Documentation rule-score improvement. Codex also adds type annotations and docstrings, but its revisions are less com- prehensive, achieving a 10.53 Documentation improvement. This example illustrates that SWE-NFI can quantitatively distinguish the NFI produced by different coding agents. 6.2 Implications Implication 1: Current coding agents still lag behind human developers in non-functional improvements. The benchmark instances in SWE-NFI are constructed from real merged pull re- quests, reflecting non-functional improvements that developers ArXiv, 2026, OnlinePengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang actively make during software maintenance. These changes demon- strate that improving documentation, error handling, library usage, and code structure is an integral part of real-world software devel- opment rather than an auxiliary activity. However, our results show that current coding agents consistently fall short of the correspond- ing human reference improvements across multiple NFI aspects. This gap suggests that, although coding agents have achieved strong functional capabilities, their ability to perform high-quality non- functional improvements remains limited. Future research should place greater emphasis on developing coding agents that can better support software maintenance beyond functional correctness. Implication 2: Deterministic evaluation enables reproducible benchmarking. Unlike LLM-as-a-judge evaluation, SWE-NFI oper- ationalizes non-functional improvements using 92 executable rules. This enables deterministic, reproducible, and scalable evaluation without requiring evaluation-time human annotation. Although rule-based evaluation cannot capture every aspect of software qual- ity, it provides an objective foundation for comparing coding agents on developer-oriented non-functional improvements and is also flexible to extend. Implication 3: NFI capability of current coding agents is aspect-dependent and often favors localized additions over structural revisions. The aspect-level results show that agent performance is not uniform across NFI aspects. Documentation and error handling are the strongest sources of improvement for several configurations, while logic-pattern improvements remain below the human reference improvement for all evaluated agents. Library-constraint improvements also vary across configurations, including cases of negative average improvement despite functional correctness. These findings suggest that future coding agents should move beyond localized insertions toward reasoning about broader program structure and maintainability. Implication 4: SWE-NFI provides actionable guidance for non-functional improvements. The five-aspect framework and 92 executable rules provide a structured and practical foundation for improving non-functional aspects of software beyond functional correctness. By operationalizing common maintenance practices into measurable rules, SWE-NFI provides actionable guidance for both researchers and practitioners to evaluate and improve doc- umentation, coding style, error handling, library usage, and code structure. Beyond serving as an evaluation benchmark, the pro- posed framework can facilitate the development of coding agents and software engineering tools that better support real-world soft- ware maintenance. Implication 5: NFI outcomes depend on the agent system and resource cost must be interpreted with output quality. The results show that NFI performance is shaped by the full agent configuration, not only by the backbone model or resource budget. Agent scaffolds affect whether outputs pass validity and functional- correctness checks, and thus how many outputs become eligible for NFI scoring. Meanwhile, higher execution time or token consump- tion does not by itself imply stronger NFI improvement, and low- cost configurations may still produce weak evaluated outputs. Prac- tical use of coding agents for non-functional maintenance should therefore consider resource cost together with validity, functional correctness, and measured NFI improvement. 7 Threats to Validity Construct Validity. SWE-NFI operationalizes non-functional im- provements using 92 executable NFI rules derived from various sources, including prior studies, open-source artifacts, and coding guidelines. Although these rules cannot fully capture every aspect of software quality, we intentionally focus on improvements that can be measured automatically and reproducibly without requiring evaluation-time human annotation or LLM-based judgment. Our benchmark is also flexible to extend more rules. Internal Validity. The stochastic nature of coding agents may introduce variability into the experimental results. To mitigate this threat, we set the temperature to zero whenever supported and repeat each experiment five times. We report the average results across all runs to reduce the influence of randomness. Our bench- mark construction involves manual procedures, such as manually validating the data to be included in our benchmark, which may introduce subjectivity. To mitigate this threat, two authors inde- pendently performed these procedures, and disagreements were resolved through discussion until consensus was reached. Func- tional tests may also affect internal validity, because they may not cover all possible cases for each task. To mitigate this threat, we only apply NFI scoring to outputs that are valid and pass the task-specific tests. We report the functional pass rate separately, so outputs that fail the tests are not counted as negative NFI improvements. External Validity. Our benchmark focuses exclusively on Python projects. Although Python is one of the most widely used program- ming languages and has a rich open-source ecosystem, the findings may not directly generalize to other programming languages or software ecosystems. Future studies may extend our benchmark to study additional programming languages. Due to computational and API costs, we evaluate a limited set of coding agents. Although we include both commercial and open-source agents, our results may not fully represent the rapidly evolving landscape of coding agents. Future studies could expand the set of evaluated agents and foundation models to further validate our findings. 8 Conclusion In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on non-functional improvements beyond functional correctness. SWE-NFI contains 188 single-file and multi-file tasks constructed from real merged Python pull requests, and operational- izes five NFI aspects into 92 executable rules with a comprehensive evaluation suite combining functional correctness testing and deter- ministic rule-based NFI evaluation. Our results show that, although current coding agents can produce measurable NFIs after gener- ating functionally correct code, they still consistently fall short of human developers, particularly on structural improvements such as logic patterns. SWE-NFI provides a reproducible foundation for evaluating and advancing coding agents toward better support for real-world software development and maintenance. SWE-NFI : Studying and Benchmarking Coding Agents for Non-Functional ImprovementsArXiv, 2026, Online References [1]2026. Replication Package. https://figshare.com/s/f14692e227368f6c392b. Last accessed June 2026. [2]Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kim- berly Milner, Sofija Jancheska, John Yang, Carlos E Jimenez, Farshad Khor- rami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik R Narasimhan, Ramesh Karri, and Ofir Press. 2025. EnIGMA: Inter- active Tools Substantially Assist LM Agents in Finding Security Vulnerabil- ities. In Forty-second International Conference on Machine Learning.https: //openreview.net/forum?id=Of3wZhVv1R [3] Aider. 2026. Aider: An ai pair programming tool. https://aider.chat/. [4] Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023. Multi-lingual Evaluation of Code Generation Models. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=Bo7eeXm6An8 [5]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732 (2021). [6]Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of Useful Code Reviews: An Empirical Study at Microsoft. In IEEE/ACM Working Conference on Mining Software Repositories (MSR). [7] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200. [8] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al.2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49, 7 (2023), 3675–3691. [9]Junkai Chen, Xing Hu, Zhenhao Li, Cuiyun Gao, Xin Xia, and David Lo. 2024. Code search is all you need? improving code suggestions with code search. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). 1–13. doi:10.1145/3597503.3639085 [10] Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, and David Lo. 2026. SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Maria Liakata, Viviane P. Moreira, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 24144–24168. https://aclanthology.org/2026.acl-long.1107/ [11] Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. 2026. Nlperturbator: Studying the robustness of code llms to natural language variations. ACM Transactions on Software Engineering and Methodology 35, 4 (2026), 1–20. [12] Junkai Chen, Zhenhao Li, Qiheng Mao, Xing Hu, Kui Liu, and Xin Xia. 2025. Understanding practitioners’ expectations on clear code review comments. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1257–1279. doi:10.1145/3728931 [13] Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning Runtime Behavior of a Program with LLM: How Far Are We?. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). [14]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al.2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [15]Claude Code. 2026. Claude Code Docs. https://docs.anthropic.com/en/docs/ claude-code. Last Access: 2026. [16] Deepseek. 2026. DeepSeek. https://w.deepseek.com/en/. Last Access: 2026. [17] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al.2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941 (2025). [18]Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al.2026. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. arXiv preprint arXiv:2512.12730 (2026). [19]Zishuo Ding, Yiming Tang, Yang Li, Heng Li, and Weiyi Shang. 2023. On the temporal relations between logging and code. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 843–854. [20]Mingzhe Du, Luu A Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models. Advances in Neural Information Processing Systems 37 (2024), 16601–16622. [21]Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. [22] Yongkang Du, Jen-tse Huang, Jieyu Zhao, and Lu Lin. 2025. Faircoder: Evaluating social bias of llms in code generation. arXiv preprint arXiv:2501.05396 (2025). [23] Ryo Fujii, Makoto Morishita, Kazuki Yano, and Jun Suzuki. 2026. TimeMachine- bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks. arXiv preprint arXiv:2601.22597 (2026). [24]David Goodger and Rossum Guido. 2001. PEP 257 - Docstring Conventions. https://peps.python.org/pep-0257/. [25]Google. 2026. Gemini API. https://ai.google.dev/gemini-api/docs/models. Last Access: 2026. [26]Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories? arXiv preprint arXiv:2507.12415 (2025). [27]Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, and David Lo. 2026. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios. arXiv preprint arXiv:2604.06742 (2026). [28]Xing Hu, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. Assessing and advancing benchmarks for evaluating large language models in software engineering tasks. ACM Transactions on Software Engineering and Methodology (2025). [29]Xing Hu, Xin Xia, et al.2022. Practitioners’ Expectations on Automated Code Comment Generation. In IEEE/ACM International Conference on Software Engi- neering (ICSE). [30]Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. 2025. Ef- fibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems 37 (2025), 11506–11544. [31] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world GitHub Issues? arXiv preprint arXiv:2310.06770 (2024). [32]Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettle- moyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning. PMLR, 18319–18345. [33] Lukasz Langa. 2019. PEP 585 – Type Hinting Generics In Standard Collections. https://peps.python.org/pep-0585/. [34] Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. 2025. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 17160–17176. [35] Zhenhao Li, An Ran Chen, Xing Hu, Xin Xia, Tse-Hsun Chen, and Weiyi Shang. 2023. Are they all good? studying practitioners’ expectations on the readability of log messages. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 129–140. doi:10.1109/ASE56229.2023.00136 [36]Zhenhao Li, Tse-Hsun Chen, and Weiyi Shang. 2020. Where Shall We Log? Studying and Suggesting Logging Locations in Code Blocks. In 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. 361–372. doi:10.1145/3324884.3416636 [37]Zhenhao Li, Tse-Hsun Chen, Jinqiu Yang, and Weiyi Shang. 2019. DLFinder: Characterizing and Detecting Duplicate Logging Code Smells. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). doi:10.1109/ICSE. 2019.00032 [38] Zhenhao Li, Heng Li, Tse-Hsun Chen, and Weiyi Shang. 2021. Deeplv: Sug- gesting log levels using ordinal based neural networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1461–1472. doi:10.1109/ICSE43902.2021.00131 [39]Zhengquan Li, Zhenhao Li, and Zishuo Ding. 2025. From Feedback to Fail- ure: Automated Android Performance Issue Reproduction. arXiv preprint arXiv:2508.11147 (2025). [40]Zhenhao Li, Chuan Luo, Tse-Hsun Chen, Weiyi Shang, Shilin He, Qingwei Lin, and Dongmei Zhang. 2023. Did we miss something important? Studying and exploring variable-aware log abstraction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 830–842. [41]Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2025. Large language model-based agents for software en- gineering: A survey. ACM Transactions on Software Engineering and Methodology (2025). [42]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Lan- guage Models for Code Generation. In Advances in Neural Information Processing Systems (NeurIPS). [43]Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408.06450 (2024). ArXiv, 2026, OnlinePengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, and Weiyi Shang [44]Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. 2025. SWE- fficiency: Can Language Models Optimize Real-World Repositories on Real Work- loads? arXiv preprint arXiv:2511.06090 (2025). [45]Qiheng Mao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia, and Jianling Sun. 2025. Towards explainable vulnerability detection with large language models. IEEE Transactions on Software Engineering (2025). [46]Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2016. An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. In Empirical Software Engineering, Vol. 21. 2146–2189. [47]Tam Nguyen, Phong Vu, and Tung Nguyen. 2020. Code Recommendation for Exception Handling. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). [48]OpenAI. 2025. Codex CLI: A Lightweight Coding Agent That Runs in Your Terminal. https://github.com/openai/codex. Accessed: 2026-06-28. [49]OpenAI. 2026. OpenAI Developer. https://developers.openai.com/api/docs/ models. Last Access: 2026. [50] Khouloud Oueslati, Maxime Lamothe, and Foutse Khomh. 2026. RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring. In IEEE/ACM International Conference on Software Engineering (ICSE). [51]Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Commun. ACM 68, 2 (2025), 96–105. [52]Norman Peitek, Sven Apel, Chris Parnin, André Brechmann, and Janet Siegmund. 2021. Program Comprehension and Code Complexity Metrics: An fMRI Study. In IEEE/ACM International Conference on Software Engineering (ICSE). [53]Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cwe- val: Outcome-driven evaluation on functionality and security of llm code gener- ation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE, 33–40. [54]Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code generation. Proceedings of the ACM on Software Engineering 2, FSE (2025), 242–265. [55]Veronica Pimenova, Sarah Fakhoury, Christian Bird, Margaret-Anne Storey, and Madeline Endres. 2025. Good Vibrations? A Qualitative Study of Co-Creation, Communication, Flow, and Trust in Vibe Coding. arXiv:2509.12491 [cs.SE] [56] Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie Zhang, Heming Cui, Siu Ming Yiu, Dong Huang, et al.2026. Effibench- x: A multi-language benchmark for measuring efficiency of llm-generated code. Advances in Neural Information Processing Systems 38 (2026). [57] Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buch- holz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al.2026. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703 (2026). [58] Guido Rossum, Jukka Lehtosalo, and Lukasz Langa. 2001. PEP 484 - Type Hints. https://peps.python.org/pep-0484/. [59]SciTools.2026.UnderstandingMcCabeCyclomaticComplexity. https://support.scitools.com/support/solutions/articles/70000582297- understanding-mccabe-cyclomatic-complexity. [60] Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2026. SecRepoBench: Benchmarking Code Agents for Secure Code Com- pletion in Real-World Repositories. arXiv preprint arXiv:2504.21205 (2026). [61] Xinyu Shi, Zhenhao Li, and An Ran Chen. 2025. Enhancing LLM-based Fault Localization with a Functionality-Aware Retrieval-Augmented Generation Frame- work. arXiv preprint arXiv:2509.20552 (2025). [62]Anthony Sottile. 2021. pyupgrade Documentation. https://github.com/asottile/ pyupgrade. Last Access: 2026. [63]SWE-agent Team. 2026. mini-SWE-Agent. https://mini-swe-agent.com/latest/. [64]Tufts University. 2026. CS 40 Coding Standards. https://w.cs.tufts.edu/comp/ 40/docs/coding-standards.html. [65]Guido Van Rossum, Barry Warsaw, and Nick Coghlan. 2001. PEP 8 - Style Guide for Python Code. https://peps.python.org/pep-0008/. [66]Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2025. LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion. In IEEE/ACM International Conference on Software Engineering (ICSE). arXiv:2406.09834. [67]Xin Wang, Yang Feng, Jiaoxiao Qian, Yang Zhang, Zhenhao Li, and Zishuo Ding. 2026. Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback. arXiv preprint arXiv:2603.29122 (2026). [68]Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al.2025. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025. 65882–65919. [69]Xin Wang, Zhenhao Li, and Zishuo Ding. 2025. Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 1931–1942. doi:10.1109/ASE63991.2025.00161 [70]Xin Wang, Zhenhao Li, and Zishuo Ding. 2026. LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE). [71]Tongtong Wu, Weigang Wu, Xingyu Wang, Kang Xu, Suyu Ma, Bo Jiang, Ping Yang, Zhenchang Xing, Yuan-Fang Li, and Gholamreza Haffari. 2024. Versicode: Towards version-controllable code generation. arXiv preprint arXiv:2406.07411 (2024). [72]Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agent- less: Demystifying LLM-based Software Engineering Agents. arXiv preprint arXiv:2407.01489 (2024). [73]Weiwei Xu, Kai Gao, Hao He, and Minghui Zhou. 2025. Licoeval: Evaluating llms on license compliance in code generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1665–1677. [74]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems (NeurIPS). [75]John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. 2025. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?. In The Thirteenth International Conference on Learning Representations. [76]John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al.2026. ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv preprint arXiv:2605.03546 (2026). [77]Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. 2023. Evacrc: Evaluating code review comments. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 275–287. [78]Ka-Ping Yee. 2005. PEP 3134 – Exception Chaining and Embedded Tracebacks. https://peps.python.org/pep-3134/. [79] Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. 2025. Utboost: Rigorous evaluation of coding agents on swe-bench. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 3762–3774. [80] He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li. 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs. Proceedings of the ACM on Software Engineering 3, FSE (2026), 745–765. [81]Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, et al.2026. Multi-swe-bench: A multilingual benchmark for issue resolving. Advances in Neural Information Processing Systems 38 (2026). [82]Beiqi Zhang, Peng Liang, Qiong Feng, Yujia Fu, and Zengyang Li. 2024. Copilot- in-the-Loop: Fixing Code Smells in Copilot-Generated Python Code using Copilot. In IEEE/ACM International Conference on Automated Software Engineering (ASE), NIER Track. arXiv:2401.14176. [83]Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2471–2484. [84] Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al.2026. Swe-bench goes live! Advances in Neural Information Processing Systems 38 (2026). [85] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- tocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604. [86]Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, and Jiao Sun. 2026. SWE-IF: Aligning Code Evaluation with Human Preference. In Proceedings of the 43rd International Conference on Machine Learning (ICML) (PMLR, Vol. 306). [87]Renyi Zhong, Yichen Li, Yulun Wu, Jinxi Kuang, Yintong Huo, and Michael R Lyu. 2026. Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs. arXiv preprint arXiv:2604.17529 (2026). [88]Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, and Michael R Lyu. 2026. Larger is not always better: Exploring small open-source language models in logging statement generation. ACM Transactions on Software Engineering and Methodology 35, 7 (2026), 1–41. [89] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2025. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In International Conference on Learning Representations, Vol. 2025. 66602–66656.