Paper deep dive

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 102

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/1/2026, 4:40:38 AM

Summary

The paper introduces RealSWE, a compositional evaluation framework for coding agents that bridges the gap between structured benchmark tasks (SWE-Bench) and realistic, sparse user requests. By analyzing real user prompts from SWE-Chat, the authors define an information taxonomy and linguistic dimensions to create 381 multi-variant task families. Evaluations show that realistic inputs reduce resolution rates by 6.4 percentage points on average. Crucially, the study finds that 'Desired Behavior' and 'Motivation' are the most valuable information fields for LLM performance, while linguistic style has minimal impact.

Entities (8)

RealSWE · benchmarkframework · 100%Desired Behavior · information-category · 95%Motivation · information-category · 95%SWE-BENCH · benchmark · 95%LLMs · model-class · 90%Resolution Rate · metric · 90%SWE-chat · dataset · 90%DeepSeek V4 Pro · model · 85%

Relation Signals (5)

RealSWE → derivedfrom → SWE-BENCH

confidence 95% · RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro.

Realistic Inputs → reduces → Resolution Rate

confidence 95% · realistic inputs reduce resolution rates by 6.4 pp on average

RealSWE → analyzesdatafrom → SWE-chat

confidence 90% · apply them to real user prompts from SWE-chat

RealSWE → evaluates → LLMs

confidence 90% · Evaluating seven contemporary LLMs with RealSWE

Desired Behavior → improvesperformanceof → LLMs

confidence 90% · including Desired Behavior and Motivation significantly affects performance

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

98,750 characters extracted from source content.

Expand or collapse full text

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests Gyuhyeong KimHyojung GwonJeonghyeon KimKyuhong ShimSunjae Lee ∗ Sungkyunkwan University gyuhyeong, gywndgywnd12, jeonghyeon12, khshim, sunjae.lee@skku.edu § GitHub Abstract Coding agents are now commonly evaluated on the SWE-BENCH family of bench- marks, whose tasks are built from curated GitHub issues—long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-CHAT and problem statements from SWE-BENCH VERIFIED and PRO. We find that requests carrying only a problem statement, alone or with limited addi- tional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce REALSWE, 381 multi-variant task families derived from SWE-BENCH VERIFIED and PRO. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with REALSWE, we find that i) realistic inputs reduce res- olution rates by 6.4 p on average and can change model rankings. Controlled analysis further shows that i) including Desired Behavior and Motivation signifi- cantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; i) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation—which most real prompts omit—substantially improves the LLM’s software engineering performance. 1 Introduction Large language models (LLMs) are rapidly transforming software engineering, advancing beyond function-level code generation toward coding agents that autonomously resolve repository-level issues. Their progress is commonly measured using SWE-BENCH families [15,20,33,25,8,34,1], which harvest executable tasks from GitHub issues and their corresponding fixes (e.g., commits and pull requests) in well-known open-source GitHub repositories. Today, leaderboard scores on these benchmarks serve as the de facto standard for comparing LLMs’ coding capabilities. Yet a growing body of evidence indicates that the inputs these benchmarks feed to agents are far from what agents receive in practice. Curated GitHub issues are typically detailed, well-structured, and long, whereas everyday user requests are short, informal, and sparse [6,10,4]. For example, a GitHub issue may describe a failure, provide reproduction steps and environment information, specify the desired behavior, and even suggest a solution. A user may express the same intent simply as ∗ Corresponding author. Preprint. arXiv:2608.27831v2 [cs.AI] 31 Aug 2026 SWE-bench (Verified + Pro) RealSWE-bench (Ours) 53.9 45.9 49.7 41.6 49.1 44.0 48.4 40.7 42.1 36.7 50.1 43.5 34.1 30.1 DeepSeek V4 Pro DeepSeek V4 Flash MiMo V2.5 Pro MiMo V2.5 Claude Haiku 4.5 Qwen3.7 Plus MiniMax M3 Resolution Rate (%) Figure 1: Resolution rates on the original SWE-BENCH tasks and REALSWE-BENCH. All models decline under realistic inputs. “this crashes on empty input—fix it.” Although the underlying task is identical, the latter provides far less evidence, requiring the agent to infer missing requirements from the repository and context. Benchmark performance may therefore depend not only on task difficulty, but also on how the task is communicated. Recent work has begun to close this gap with new realistic benchmarks. CURSORBENCH [6] evaluates prompts drawn from real coding sessions and others synthesize underspecified tasks [5,35,29] or mutate GitHub issues [10,24,28] to simulate user inputs. However, these efforts fall short in two ways. Benchmarks grounded in genuine user data, such as CURSORBENCH, are closed-source and unavailable for independent use or inspection. Open alternatives approximate realism heuristically, for example by truncating problem statements or injecting ambiguity, without systematic analysis or empirical grounding in actual user data. Consequently, while they reveal that a benchmark–reality gap exists, they fail to provide insight into what causes it or how it should be measured. In this paper we address this gap through a systematic, data-grounded approach. We begin by analyzing user prompts from SWE-CHAT, a large-scale dataset of real interactions between developers and coding agents [4]. We characterize each request along two orthogonal axes: the information it conveys (Table 1) and the language in which it is conveyed (Figure 2). Our analysis reveals a substantial mismatch between real user requests and benchmark problems on both axes. Real requests are information-sparse: prompts consisting only of a problem statement (i.e., which bug to fix or which feature to implement) alone or with limited additional context (e.g., source URLs, dates) account for 88% of user prompts, compared with only 7% of tasks in SWE-BENCH VERIFIED and SWE-BENCH PRO. They also differ linguistically: 87% of user prompts use a casual tone and 51% contain imperative sentences, whereas 94% of benchmark problems use a formal tone and 89% rely on declarative sentences. These differences provide an empirical basis for building realistic coding-agent evaluation. Guided by these observations, we introduce REALSWE, an open benchmark and configurable evaluation framework grounded in the characteristics of real user inputs. It contains 381 multi- variant task families derived from SWE-BENCH VERIFIED and PRO. Each family shares the same underlying task while varying information composition and linguistic style. We release it as REALSWE-BENCH, a fixed evaluation set matching the distributions observed in SWE-CHAT, and REALSWE-FRAMEWORK, which exposes the full variant suite for custom configurations and controlled ablations. We evaluate seven contemporary LLMs under REALSWE. The results reveal substantial discrepancies between performance on original SWE-BENCH-style problems and realistic user inputs: resolution rates drop by 6.4 percentage points (p) on average, and the gap between stronger and weaker models narrows (Figure 1). More importantly, our controlled analysis under REALSWE-FRAMEWORK reveals a sharply uneven value of information. For bug fixes, the presence of Desired Behavior strongly affects success (+8 p, 17% relative), while Reproduction Steps and Environment Information add input tokens with no measurable benefit; for feature requests, adding Motivation improves 2 Table 1: Information taxonomy and field abbreviations for bug-fix and feature-request prompts. Bug fixFeature request [P] Problem Statement (Observed Failure) [P] Problem Statement (Requested Feature) [D] Desired Behavior [M] Motivation [R] Reproduction Steps [A] Additional Information [E] Environment Information [A] Additional Information performance by up to+7 p. Linguistic style changes, in contrast, produce only small, model- dependent effects. These findings expose an actionable mismatch between what users provide and what agents need: Desired Behavior of a bug fix and Motivation behind the new feature are among the most valuable signals an agent can receive, yet only 5% of real user prompts state it. Explicitly stating these can thus substantially improve the performance users experience from coding agents. In summary, our contributions are threefold: •Empirical characterization of real SWE requests. Using a structured information taxonomy and linguistic dimensions, we analyze how real user requests in SWE-CHAT are composed, and quantify how their information composition and linguistic style differ from those of SWE-BENCH VERIFIED and SWE-BENCH PRO. •A data-grounded benchmark and configurable framework. Guided by this analysis, we construct 381 multi-variant task families and release them in two forms: REALSWE-BENCH, an open-source benchmark reflecting the empirical distribution of real user inputs, and REALSWE- FRAMEWORK, a configurable framework that allows researchers to customize the information composition and linguistic style of the evaluation. • Controlled evaluation and actionable findings. Evaluating seven LLMs, we i) quantify the benchmark–reality gap, i) identify Desired Behavior and Motivation as key signals for software engineering tasks, and i) distill actionable guidance for users of coding agents. 2 Related Work Coding agent benchmarks.SWE-BENCH established repository-level issue resolution as a stan- dard evaluation setting by pairing real GitHub issues with executable tests [15]. Subsequent bench- marks have improved reliability and expanded language, repository, modality, task, and complexity coverage [20,33,25,32,8,18]. However, these benchmarks generally associate each executable task with a single canonical issue description. They therefore broaden and strengthen the underlying software engineering tasks, but do not examine how performance changes when the same task is communicated with different information or linguistic forms. Realistic and communication-aware evaluation.Recent work has begun to incorporate realistic tasks using naturally occurring developer requests or transformed benchmark inputs. Benchmarks built from real coding sessions provide authentic user inputs, but communication varies together with the task, repository, and difficulty, making its independent effect difficult to isolate [6,14]. Observational datasets such as SWE-CHAT further characterize how developers communicate with coding agents in practice [4]. Other work introduces underspecified or interactive variants to study ambiguity and clarification behavior [28, 9, 16]. Most closely related to our work, SAVING SWE-BENCH uses patterns observed in real developer interactions to transform existing repository tasks into user-style inputs [10]. However, its trans- formations jointly alter multiple properties of the task specification, making it difficult to attribute performance changes to particular information components or linguistic properties. In contrast, REALSWE represents each task as a multi-variant family grounded in the information compositions and linguistic dimensions observed in real requests. This design independently controls the infor- mation content and linguistic style of a fixed repository task, enabling per-field ablations, arbitrary compositions, and distribution-matched evaluation. 3 Figure 2: Distributions of (a) information composition and (b) linguistic style across SWE-CHAT, SWE-BENCH VERIFIED, and SWE-BENCH PRO. See Appendix D for detailed results. Classification (Task Types) Restructuring (Information Decomposition) Rephrasing (Real-User Style) Multi-Variant Task Families dump_json() crashes on serializing empty dictionary. Expected behavior is to return "" Example code: dump_json() on thread 1. Environment: Python 3.11, Linux Non-empty dictionaries work correctly. Bug Fix Feature Request V dump_json() crashes on serializing empty dictionary. Expected behavior is to return "“ ... Information Taxonomy LLM PDREA LLM PDREA BugFix FeatureRequest PMA P When I pass empty string, dump_json() crashes. D It should return "" R dump_json() on thread 1. E Python 3.11, Linux A It works fine when I pass non-empty dictionaries. RealSWE-Bench (Fixed Benchmark) 〮 381 tasks 〮 Reflecting SWE-chat distribution Distribution [P][PA] Bug74%26% Feature72%28% RealSWE-Framework (Configurable Framework) Task Type: BugFeature Information Composition: [P]/ [PA]/[PD]/ [PRA] / [PDRE] Tone: Formal/ Casual Sentence Type: Imperative / Declarative / Interrogative Task Variants (Same Underlying Software Engineering Task) PDE PRA PDR PDREA PE PR PRA PREA PD PA PDA PDE Casual Imperative Casual Declarative Formal Imperative P dump_json() crashes on serializing ... D Expected behavior is to return "" R Example code: dump_json() on ... E Environment: Python 3.11, Linux A Non-empty dictionaries work correctly. P dump_json() crashes on serializing ... D Expected behavior is to return "" R dump_json() on thread 1. E Python 3.11, Linux A Non-empty dictionaries work correctly. LLM Figure 3: Overview of the REALSWE construction pipeline. 3 Method We introduce REALSWE (Figure 3), a benchmark and configurable framework for evaluating the same software engineering task under systematically varied task specifications, including REALSWE- BENCH, a fixed configuration that reflects the characteristics of real user requests. We construct REALSWE in four stages. First, drawing on established software engineering practices and prior literature, we define task-specific information taxonomies and linguistic dimensions for character- izing software engineering requests (§3.1). Second, we apply this scheme to real user requests from SWE-CHAT and problem statements from SWE-BENCH VERIFIED and SWE-BENCH PRO, quantifying their differences in information composition and linguistic style (§3.2). Third, guided 4 by these measurements, we transform the benchmark problems into multi-variant task families that represent the same software engineering task under different information compositions and linguistic styles (§3.3). Fourth, we validate each LLM-driven stage of the construction pipeline against human judgment (§3.4). Finally, we release a fixed benchmark reflecting dominant real- user input patterns (REALSWE-BENCH) and a configurable framework exposing all task variants (REALSWE-FRAMEWORK) (§3.5). 3.1 Characterizing SWE Requests Data. We use two complementary data sources. For real user inputs, we analyze SWE-CHAT [4], a public dataset containing more than 6,000 real developer–agent sessions. Because SWE-BENCH typically evaluates an agent from a single problem statement without further interaction, we retain only the first user request from each session. We further remove prompts that do not express an actionable SWE task, including conversational dialogue, inputs generated by other tools, and pasted LLM outputs, leaving 718 user-authored prompts (Appendix C). We compare these real requests against problem statements from SWE-BENCH VERIFIED [20] and PRO [8]. We choose these two benchmarks because they represent widely used repository-level coding-agent evaluation. SWE-BENCH VERIFIED is a human-validated benchmark for standardized model comparison. SWE-BENCH PRO extends this setting to more difficult, longer-horizon tasks drawn from larger and more complex repositories. Task types.Following common practice in issue-tracking systems [12], we categorize each request as either a bug fix—existing behavior is incorrect and needs a fix—or a feature request—the user asks for new or changed functionality. These two types cover nearly all requests: only 2 of the 1,231 problem statements in SWE-BENCH VERIFIED and PRO fall into neither, and we exclude them before the rest of the pipeline (Table 7). Information taxonomy. For each task type, we define an information taxonomy: the set of in- formation types that a request may contain. The taxonomy is grounded in GitHub’s default issue templates [11,26] (Table 1).[A]captures information outside the other categories, such as source URLs, issue-author metadata, and dates. A request is then described by the set of information types it contains—e.g., [P], [PA], or [PDR]. Linguistic properties. To characterize how a request is written, we categorize its linguistic style along four dimensions: Formality, Sentence type, Certainty, and Perspective (Figure 2). Adapted from [27], these dimensions are chosen to separate GitHub issue-style prose from the conversational language of coding-agent chats. 3.2 Measuring the Benchmark–Reality Mismatch Annotation procedure. We apply the information taxonomy and linguistic dimensions to the 718 user-authored SWE-CHAT requests and to the 1,229 problem statements from SWE-BENCH VERIFIED and PRO. Using an LLM-assisted pipeline, GPT-5.4 [21] identifies each piece of informa- tion in a prompt, assigns it to a taxonomy category, and classifies the prompt’s style along the four dimensions. Results: the mismatch. Figure 2 summarizes the resulting distributions. Most notably, the two sources diverge sharply in information composition. Real requests are compositionally sparse:[P] and[PA]alone account for 88% of prompts (85.5% of bug-fix and 91.1% of feature requests), indicating that in real-world practice, users often rely on simple, underspecified requests such as “Server crashes on empty input, fix it.” or “Implement new feature that does ...” In comparison, problems in SWE-BENCH VERIFIED and PRO are information-rich, with only 7% consisting of [P]/[PA](8.0% for bug fixes, 3.9% for feature requests). They often include additional fields such as Reproduction Steps, Environment Information, and other contextual details that real users rarely provide. This gap likely leads benchmarks to overestimate LLMs’ coding performance in real-world settings. The linguistic dimensions exhibit a less uniform pattern (Figure 2(b)). SWE-CHAT and the SWE- BENCH datasets differ most strongly in formality and sentence type: 86.8% of real user requests 5 are casual and 51.3% are imperative. In contrast, 84.8% and 100% of prompts in SWE-BENCH VERIFIED and PRO, respectively, are formal, and approximately 89% of prompts in both benchmarks are declarative. This suggests that SWE-BENCH prompts resemble polished issue reports rather than conversational user requests. Certainty and perspective show no consistent separation between real requests and benchmark prompts. These distributions serve as empirical targets for realistic evaluation and directly guide the construction described below. 3.3 Constructing Multi-Variant Task Families Guided by the observed distributions, we transform each benchmark problem from SWE-BENCH VERIFIED and PRO into a multi-variant task family through a three-step, LLM-driven pipeline (task-type classification, information decomposition, and real-user-style rephrasing). The complete prompts for all three steps are provided in Appendix H. Decomposing problem statements. First, we leverage GPT-5.4 to classify each task as a bug fix or feature request. Before decomposition, we also strip scaffolding inherited from GitHub issue templates, such as Markdown headings and HTML tags, while preserving all user-written text (Appendix B.2). Then, we segment each original problem statement into sentences and assign each sentence to an information taxonomy category. We redistribute the original text across these fields without rewriting it so that decomposition does not alter the information it carries. Rephrasing in real-user style. We rewrite the restructured problems into the style observed in SWE-CHAT using GPT-5.4, conditioning on the majority category of each linguistic dimension measured in §3.2. This process changes only the manner of expression while preserving technical content, such as code blocks, error messages, tracebacks, and file paths (Appendix B.3). Collecting task families with configurable variants.REALSWE represents each software engi- neering problem using arbitrary combinations of information categories. Supporting such configu- rations requires every taxonomy field to be present in the source problem statement. Of the 1,229 problems in SWE-BENCH VERIFIED and SWE-BENCH PRO, 403 contain all required information categories, forming our initial candidate set (Appendix B.2). 3.4 Validation & Quality Control We assess both the reliability of the construction pipeline and the quality of the resulting task families through human validation. For each LLM-driven stage, two annotators independently evaluate 100 sampled instances and resolve disagreements by consensus to establish human ground truth. Appendix F reports the full rubrics, inter-annotator agreement, and validation results. Pipeline validation and task quality. For task-type classification, we directly measure accuracy against the human ground truth; the classification pipeline achieves an accuracy of 0.95. For field decomposition and linguistic rephrasing, we use GPT-5.6 Terra [22] as an LLM judge to audit all candidate tasks using the same three-point rubrics as the human annotators. First, we assess the quality of the task set by verifying whether the field-decomposition pipeline assigns each content unit to the correct taxonomy field and whether the resulting task specification remains complete and coherent. Second, we assess the realism of the transformed requests by verifying whether the rephrasing pipeline produces the target real-user style while preserving the original information, meaning, implementation intent, and technical literals. The LLM judge shows high agreement with human judgment (decomposition: accuracy 0.97, macro- F 1 0.83; rephrasing: accuracy 0.99, macro-F 1 0.75). We exclude candidates receiving the lowest score on any critical criterion, removing 22 of the 403 candidates and leaving 381 task families (192 bug fixes and 189 feature requests). Selection bias. Our selection process reduces the original pool of 1,229 tasks to 381. To assess potential selection bias, we compare the selected 381 tasks with the excluded 848 tasks in terms of resolution rate, patch size, and repository distribution. We find that the selected task families are not easier, smaller, or concentrated in particular repositories. Appendix A.4 reports the complete comparison. 6 Figure 4: Mean task-description length across coding benchmarks and the real user prompts in SWE- CHAT. External documents (e.g.,AGENTS.mdand documentation) are excluded. REALSWE-BENCH closely matches SWE-CHAT, whereas conventional benchmarks are substantially longer. 3.5REALSWE Benchmark and Framework To reflect the information distribution observed in SWE-CHAT (§3.2), REALSWE-BENCH samples one variant per task family in the following proportions: [P] 74% and [PA] 26% for bug fixes, and [P] 72% and[PA]28% for feature requests. This yields 142[P]and 50[PA]bug-fix tasks, and 136 [P] and 53 [PA] feature-request tasks (total 381 tasks). The resulting benchmark has an average task-description length of 1,417 characters, closely matching the 1,427 average observed in SWE-CHAT and well below the 1,672–2,776 characters of conventional SWE benchmarks (SWE-BENCH VERIFIED, MULTILINGUAL, PRO, and DEEPSWE; Figure 4). Given that prior work identifies description length as a key benchmark–reality distinction [6], this alignment corroborates the realism of REALSWE-BENCH. REALSWE-FRAMEWORK exposes all 381 task families through a configuration interface. Re- searchers specify a task type, information composition, and linguistic style—for example, bug fixes containing only[P]and[D]in a casual style—and the framework assembles the corresponding dataset on demand. 4 Experiments Using REALSWE (both benchmark and framework), we conduct a controlled evaluation of how realistic user inputs affect coding agents, organized around three research questions: RQ1.How does agent performance change when benchmark problems are replaced with realistic user inputs? RQ2. Does the linguistic style of a request alone affect coding-agent performance? RQ3. Which information fields most affect task resolution, and how does their value differ? Table 2: Resolution rate (%) on the original problem statements and REALSWE-BENCH.∆denotes REALSWE-BENCH minus Original; the largest drop in each ∆ column is in bold. All (n=381)Bug fix (n=192)Feature request (n=189) ModelOriginalREALSWE∆OriginalREALSWE∆OriginalREALSWE∆ DeepSeek V4 Pro53.9 ±0.6 45.9 ±2.0 −8.059.4 ±1.4 49.0 ±2.3 −10.448.3 ±0.3 42.9 ±2.3 −5.5 DeepSeek V4 Flash49.7 ±1.7 41.6 ±0.9 −8.056.2 ±1.0 44.4 ±1.8 −11.843.0 ±2.5 38.8 ±0.3 −4.2 MiMo V2.5 Pro49.1 ±1.2 44.0 ±1.0 −5.154.9 ±1.6 47.2 ±2.1 −7.643.2 ±2.1 40.7 ±1.1 −2.5 MiMo V2.548.4 ±1.4 40.7 ±1.5 −7.754.5 ±1.3 45.1 ±1.2 −9.442.2 ±1.6 36.2 ±2.1 −6.0 Claude Haiku 4.542.1 ±1.1 36.7 ±2.9 −5.450.3 ±1.6 40.1 ±2.4 −10.233.7 ±1.9 33.2 ±3.4 −0.5 Qwen3.7 Plus50.1 ±0.5 43.5 ±0.8 −6.654.7 ±1.8 45.5 ±2.2 −9.245.5 ±2.4 41.4 ±1.3 −4.1 MiniMax M334.1 ±0.5 30.1 ±1.7 −4.040.5 ±0.3 35.2 ±0.3 −5.227.7 ±1.3 24.9 ±3.2 −2.8 7 Table 3: Average per-task cost and steps on the original problem statements and REALSWE-BENCH. Cost (¢)Steps ModelOriginalREALSWEOriginalREALSWE DeepSeek V4 Pro3.02 ±0.02 3.16 ±0.08 42.6 ±0.4 44.2 ±0.3 DeepSeek V4 Flash1.27 ±0.01 1.33 ±0.00 47.7 ±0.4 49.0 ±0.2 MiMo V2.5 Pro5.55 ±0.07 6.52 ±0.18 47.9 ±0.4 48.6 ±0.5 MiMo V2.52.24 ±0.06 2.31 ±0.04 47.3 ±1.1 47.5 ±0.5 Claude Haiku 4.533.89 ±0.21 33.23 ±0.45 66.9 ±0.4 66.7 ±0.3 Qwen3.7 Plus15.26 ±0.67 16.12 ±0.23 56.2 ±0.7 56.8 ±0.4 MiniMax M317.78 ±0.17 18.05 ±0.14 82.2 ±0.7 83.5 ±0.4 4.1 Experimental Setup Models.We evaluate seven LLMs with varying sizes and families: DeepSeek V4 Pro and DeepSeek V4 Flash [7], MiMo V2.5 Pro and MiMo V2.5 [30], Claude Haiku 4.5 [2], Qwen3.7 Plus [23], and MiniMax M3 [17]. The set includes both open-weight and commercial models and spans a broad range of baseline performance. We enable reasoning for every model to match contemporary coding- agent use. Exact model snapshots, providers, and inference settings are reported in Appendix G. Agent scaffold and execution environment. Following common practice in SWE benchmark evaluation, we run every model with the same mini-SWE-agent v2 scaffold [31]. Its minimal Bash-only interface provides a consistent agent setup across model providers without relying on provider-specific tool-calls. Each task runs in the execution container released with its source benchmark, with a maximum of 100 agent steps and no cost limit. We run every model–condition pair three times and report the mean and standard deviation across runs. Details of the statistical analysis are provided in Appendix E. 4.2 Main Results: The Benchmark–Reality Gap Table 2 compares performance on the original problem statements with performance on REALSWE- BENCH. As a result, all seven models resolve fewer tasks under REALSWE-BENCH. The resolution rate decreases by 6.4 p on average, with absolute drops ranging from 4.0 p for MiniMax M3 (34.1% to 30.1%) to 8.0 p for DeepSeek V4 Pro (53.9% to 45.9%). Relative to each model’s original score, the declines fall within a range of 10.3–16.2%, with a mean of 13.6%. The degradation is not specific to one model family; all models lose a similar fraction of their baseline performance. Conventional leaderboard scores should therefore be interpreted as optimistic estimates of performance under realistic, sparse requests. The performance drop is accompanied by a modest increase in computation (Table 3). Six of the seven models incur higher cost under REALSWE-BENCH (6.2% on average) and take more steps (1.8% on average). Claude Haiku 4.5 is the only exception, showing marginal decreases in both. These results suggest that agents compensate for missing information through additional exploration, but that this extra effort is insufficient to recover the lost resolution rate. For most models, realistic requests are therefore both less successful and more expensive. More importantly, model rankings are not preserved (Figure 1). MiMo V2.5 Pro moves from fourth place on the original inputs to second place under REALSWE-BENCH, overtaking Qwen3.7 Plus and DeepSeek V4 Flash; the shift in the MiMo–Qwen gap between the two conditions is significant (+3.7 p, 95% CI[+0.7, +7.3]). MiMo V2.5 Pro is also roughly 2.5×cheaper per task than Qwen3.7 Plus (6.5 versus 16.1 cents). This suggests that the original leaderboard, taken at face value, would have steered users toward a model that is more expensive without being measurably better under realistic requests. Realistic inputs also reshape the performance distribution in two ways. Among the four models clustered near the top of the original leaderboard (Qwen3.7 Plus, DeepSeek V4 Flash, MiMo V2.5 Pro, and MiMo V2.5), the performance range nearly doubles from 1.7 to 3.3 p. At the same time, because the strongest models lose more in absolute terms, the overall strongest-to-weakest range narrows from 19.8 to 15.8 p. Realistic inputs thus alter model separation rather than scaling all 8 Table 4: Resolution rates and agent steps across information-field configurations for four mod- els.∆ orig denotes the change from the original input, and∆ prev the change from the preceding configuration. DeepSeek V4 ProDeepSeek V4 FlashMiMo V2.5 ProMiMo V2.5 FieldsRate∆ orig ∆ prev StepsRate∆ orig ∆ prev StepsRate∆ orig ∆ prev StepsRate∆ orig ∆ prev Steps Bug fix (n=192) Orig. 59.4 ±1.4 –42.2 56.2 ±1.0 –46.1 54.9 ±1.6 –45.4 54.5 ±1.3 –45.4 PDREA57.8 ±3.3 −1.6 −1.6 43.1 56.4 ±0.3 +0.2 +0.2 45.7 55.7 ±0.9 +0.9 +0.9 45.1 55.0 ±0.6 +0.5 +0.5 44.5 PDRE 55.9 ±1.1 −3.5 −1.9 42.2 57.1 ±1.3 +0.9 +0.7 45.8 53.3 ±1.3 −1.6 −2.4 44.9 51.6 ±0.5 −3.0 −3.5 45.0 PDR56.6 ±1.5 −2.8 +0.7 42.0 54.9 ±0.6 −1.4 −2.3 46.9 54.2 ±1.4 −0.7 +0.9 45.0 54.9 ±0.8 +0.3 +3.3 44.4 PD56.6 ±2.0 −2.80.0 42.7 52.8 ±1.6 −3.5 −2.1 47.7 55.4 ±1.7 +0.5 +1.2 46.3 52.8 ±1.1 −1.7 −2.1 45.5 P48.1 ±1.3 −11.3−8.5 44.7 45.3 ±0.9 −10.9−7.5 48.1 46.5 ±1.8 −8.3−8.8 45.9 45.7 ±1.7 −8.9−7.2 45.7 PA50.5 ±2.4 −8.9 +2.4 42.8 47.4 ±0.9 −8.9 +2.1 46.9 51.6 ±0.9 −3.3 +5.0 45.5 47.9 ±1.0 −6.6 +2.3 44.5 Feature request (n=189) Orig. 48.3 ±0.3 –43.0 43.0 ±2.5 –49.4 43.2 ±2.1 –50.5 42.2 ±1.6 –49.4 PMA45.5 ±1.8 −2.8−2.8 43.9 43.7 ±2.0 +0.7 +0.7 50.3 42.0 ±0.3 −1.2 −1.2 52.6 38.3 ±1.3 −3.9−3.9 49.4 PM45.7 ±1.3 −2.6 +0.2 44.9 44.1 ±1.6 +1.1 +0.4 49.5 42.2 ±2.9 −1.1 +0.2 51.4 40.2 ±1.9 −1.9 +1.9 48.7 P42.9 ±0.9 −5.5−2.8 44.7 37.0 ±0.5 −6.0−7.1 51.3 40.2 ±1.4 −3.0−1.9 52.1 38.4 ±0.8 −3.7 −1.8 49.8 PA43.6 ±0.8 −4.8 +0.7 44.9 41.1 ±2.0 −1.9 +4.1 50.1 40.7 ±1.9 −2.5 +0.5 52.4 36.7 ±0.6 −5.5 −1.8 50.5 scores uniformly: models that looked interchangeable pull apart, while the field as a whole becomes more compressed. Finally, the benchmark–reality gap varies substantially by task type. Bug-fix resolution decreases by 9.1 p on average, whereas feature-request resolution decreases by only 3.7 p. This suggests that underspecified user requests are especially challenging for bug repair, for which a precise description of the intended behavior may be critical. We test this explanation directly through the field-level analysis below. 4.3 Controlled Analysis of REALSWE Results To further investigate why realistic requests reduce coding-agent performance, we conduct a controlled analysis using the variants provided by REALSWE-FRAMEWORK (Table 4). This analysis uses four models: DeepSeek V4 Pro, DeepSeek V4 Flash, MiMo V2.5 Pro, and MiMo V2.5, representing two model families at different scales. Ablation design. Since the number of possible field combinations grows combinatorially, we organize the main analysis around cumulative ablation levels that begin with the most complete request and remove one field at a time. For bug fixes, the sequence is[PDREA]→ [PDRE]→ [PDR] → [PD] → [P] ; for feature requests, it is[PMA] → [PM] → [P]. Each transition removes the rightmost field, allowing us to measure the effect of progressively reducing the available information. The all-field variants,[PDREA]and[PMA], preserve the information contained in the original problem statement and differ only in linguistic style. Finally, because[PA]accounts for a large share of real requests in SWE-CHAT, we include it as an additional comparison condition. To verify that the observed effects are not specific to this cumulative ablation path, we evaluate a broader set of field combinations in Appendix E.1.2, focusing in particular on whether the effects of[D]and[M]persist when combined with other fields. Effect of linguistic style. Comparing the original problem statements with the all-field rephrased variants (i.e.,[PDREA]and[PMA]) isolates the effect of linguistic style while holding information constant. Linguistic style has only a small, model-dependent effect: for bug fixes, the change ranges from−1.6to+0.9p and averages 0.0 p; for feature requests, it ranges from−3.9to+0.7p and averages−1.8p. None of the eight style contrasts is significant (Holm-adjustedp≥.35). Some models even improve after rephrasing, and no consistent direction emerges across models or task types. Overall, linguistic rephrasing alone does not systematically affect coding-agent performance; rather, as shown below, what a request contains matters substantially more than how it is expressed. Information-field ablation. For bug fixes, progressively removing[A],[E], and[R]changes resolution rate by only 1.8 p on average. No removal step yields a significant decrease (Holm- adjustedp≥.44), and the only significant change is an improvement (+6.2 p for MiMo V2.5,p=.017; Table 9), indicating that there is no noticeable or consistent effect of these information fields. 9 On the other hand, Desired Behavior[D]is markedly different. Removing[D]from[PD]lowers the resolution rate by 7.1–8.9 p (8.0 p on average; significant for all four models, Holm-adjusted p<.01). This loss is more than four times the combined decrease from removing[A],[E], and [R]. Although reproduction code and environment details may be essential for particular bugs, their average contribution along this ablation path is small, whereas Desired Behavior consistently provides substantial value. Feature requests show similar concentration around Motivation[M]. Comparing[PM]with[P], removing Motivation reduces resolution for all four models by 3.4 p on average, and[M]is the only feature-request field whose average effect is distinguishable from zero (95% CI[+0.2, +8.2] p). The effect is smaller and more model-dependent than that of Desired Behavior for bug fixes: DeepSeek V4 Flash loses 7.1 p (Holm-adjustedp=.001), whereas the other models lose 1.8–2.8 p (n.s.). Motivation is therefore valuable on average, but not as uniformly decisive as Desired Behavior. We attribute this pattern to the nature of the task type. In a feature request, the problem statement itself already describes the desired behavior, so the benefit that[D]provides for bug fixes comes built into[P], and Motivation is the main information a user can still add, with a correspondingly smaller marginal value. Additional combinations reported in Appendix E.1.2 confirm that the benefits of[D] and [M] persist across different surrounding information fields. What users provide vs. what agents need. As established in §3.2, real users overwhelmingly submit[P]or[PA]requests (Figure 2), often omitting high-value fields such as Desired Behavior and Motivation while providing lower-value residual context. In fact,[PD]outperforms[PA]across all four models by 3.8–6.1 p on bug-fix tasks, while[PM]outperforms[PA]by 1.4–3.5 p on feature-request tasks. These results show that prompt length alone cannot precisely characterize the benchmark–reality gap: performance depends more on which information a request contains than on how much information it contains. 5 Discussion and Limitations Actionable guidance for information-sparse prompts.The practical implication of our analysis is not that users should write benchmark-style issue reports for every software engineering request. Rather, users should specify the Desired Behavior when requesting a bug fix and the Motivation when requesting a new feature. In practice, however, users may omit this information because they do not recognize its value or because the intended behavior or motivation is itself unclear. Coding-agent interfaces can address this gap by explicitly asking targeted clarification questions before implementation. Alternatively, an agent may infer the missing Desired Behavior or Motivation from the request and repository context, incorporate it into the task specification, and then begin implementation. Such mechanisms could help coding agents better handle information-sparse requests and improve the performance users experience. Model coverage. Although DeepSeek V4 Pro and MiMo V2.5 Pro are trillion-parameter-scale models, our evaluation does not include frontier systems that currently achieve leading performance on software engineering tasks, such as GPT-5.6 [22], Opus 5 [3], and Kimi K3 [19]. Our findings may therefore not fully generalize to the strongest available coding models. Multi-turn evaluation.Following SWE-BENCH, our evaluation is single-turn: the agent receives one task specification and completes the task without further interaction. It therefore does not capture how iterative clarification might mitigate sparse or ambiguous requests—a constraint of SWE-BENCH-style evaluation in general. Nevertheless, REALSWE enables a more structured analysis around which information should be prioritized during such interactions. Extending this to controlled conversational settings is important future work. 6 Conclusion We introduced REALSWE, a benchmark and configurable framework for evaluating coding agents under realistic user requests. By characterizing the gap between real prompts and conventional SWE-BENCH problems, REALSWE enables controlled analysis of how information and linguistic style affect coding performance. Our results show that realistic inputs can substantially change 10 measured performance, and that Desired Behavior and Motivation are particularly valuable signals. We hope REALSWE supports more realistic evaluation and better-informed coding-agent interfaces. References [1] Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitrii Babaev, Alena Fenogenova, and Valentin Malykh. SWE-MERA: A dynamic benchmark for agenticly evaluating large language models on software engineering tasks. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 440–452. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.emnlp-demos.30. [2]Anthropic. System card: Claude Haiku 4.5.https://w.anthropic.com/claude-haiku-4-5-sys tem-card, 2025. [3] Anthropic. Introducing Claude Opus 5. https://w.anthropic.com/news/claude-opus-5, 2026. [4]Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, and Sanmi Koyejo. SWE-chat: Coding agent interactions from real users in the wild, 2026. URLhttps://arxiv.org/abs/2604.207 79. [5]Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. NLPerturbator: Studying the robustness of code LLMs to natural language variations. ACM Transactions on Software Engineering and Methodology, 35(4): 89:1–89:20, 2026. doi: 10.1145/3745764. [6] Cursor. How we compare model quality in Cursor. https://cursor.com/blog/cursorbench, 2026. [7]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URL https://arxiv.org/abs/2606.19348. [8]Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?, 2025. URL https://arxiv.org/abs/2509.16941. [9]Nicholas Edwards and Sebastian Schuster. Ask or assume? uncertainty-aware clarification-seeking in coding agents, 2026. URL https://arxiv.org/abs/2603.26233. [10] Spandan Garg, Benjamin Steenhoek, and Yufan Huang. Saving SWE-Bench: A benchmark mutation approach for realistic agent evaluation, 2026. URL https://arxiv.org/abs/2510.08996. [11] GitHub. Configuring issue templates for your repository. GitHub Docs,https://docs.github.com/en /communities/using-templates-to-encourage-useful-issues-and-pull-requests/conf iguring-issue-templates-for-your-repository, 2026. [12] Kim Herzig, Sascha Just, and Andreas Zeller. It’s not a bug, it’s a feature: How misclassification impacts bug prediction. In 2013 35th International Conference on Software Engineering (ICSE), pages 392–401, 2013. doi: 10.1109/ICSE.2013.6606585. [13] Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. https://deepswe.datacurve.ai/blog/deepswe, 2026. [14]Smriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, and Satish Chandra. REAP: Automatic curation of coding agent benchmarks from interactive production usage, 2026. URL https://arxiv.org/abs/2604.01527. [15] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, pages 54107–54157, 2024. URLhttps://open review.net/forum?id=VTF8yNQM66. [16]Brendan King and Jeffrey Flanigan. Dialogue SWE-Bench: A benchmark for dialogue-driven coding agents, 2026. URL https://arxiv.org/abs/2606.13995. [17]Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, Zekun Li, Songquan Zhu, Jingkai Zhou, and Pengyu Zhao. MiniMax sparse attention, 2026. URL https://arxiv.org/abs/2606.13392. 11 [18]Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. FEA-Bench: A benchmark for evaluating repository-level code generation for feature implementation, 2025. URL https://arxiv.org/abs/2503.06680. [19] Moonshot AI. Kimi K3. https://huggingface.co/moonshotai/Kimi-K3, 2026. [20]OpenAI. Introducing SWE-bench Verified.https://openai.com/index/introducing-swe-bench -verified/, 2024. [21] OpenAI. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/, 2026. [22]OpenAI. GPT-5.6 preview system card.https://deploymentsafety.openai.com/gpt-5-6-prev iew, 2026. [23] Qwen Team. Qwen3.7-Plus. https://qwen.ai/blog?id=qwen3.7-plus, 2026. [24]Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, and Yunzhong He. SWE-INTERACT: Reimagin- ing SWE benchmarks as user-driven long-horizon coding sessions, 2026. URLhttps://arxiv.org/ab s/2606.30573. [25]Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents, 2025. URL https://arxiv.org/abs/2504.08703. [26]Emre Sülün, Metehan Saçakçı, and Eray Tüzün. An empirical analysis of issue templates usage in large- scale projects on GitHub. ACM Transactions on Software Engineering and Methodology, 33, 2024. doi: 10.1145/3643673. [27]Kimberly Truong, Riccardo Fogliato, Hoda Heidari, and Steven Wu. Persona-augmented benchmarking: Evaluating LLMs across diverse writing styles. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22676–22709, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1155. URL https://aclanthology.org/2025.emnlp-main.1155/. [28]Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. Ambig- SWE: Interactive agents to overcome underspecificity in software engineering, 2026. URLhttps: //arxiv.org/abs/2502.13069. [29] Jie JW Wu and Fatemeh H. Fard. HumanEvalComm: Benchmarking the communication competence of code generation for LLMs and LLM agents. ACM Transactions on Software Engineering and Methodology, 34(7):189:1–189:42, 2025. doi: 10.1145/3715109. [30]Xiaomi. Xiaomi MiMo-V2.5 series open-sourced.https://mimo.mi.com/docs/en-US/news/lates t/v2.5-open-sourced, 2026. [31] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps: //arxiv.org/abs/2405.15793. [32] John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. SWE-bench Multimodal: Do AI systems generalize to visual software domains?, 2024. URLhttps: //arxiv.org/abs/2410.03859. [33] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Ming Ding, and Liang Xiang. Multi-SWE-bench: A multilingual benchmark for issue resolving. In Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc., 2025. URLhttps://proceedings.neurips.c/paper_files/paper/2025/file/5afa9cb1e91 7b898ad418216dc726fbd-Paper-Datasets_and_Benchmarks_Track.pdf. [34] Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. SWE-bench goes live! In Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc., 2025. URLhttps://proceedings.neurips.c/paper_files/paper/202 5/file/d83c4a745789690f82e86d0ef752ae7c-Paper-Datasets_and_Benchmarks_Track.pdf. 12 [35]Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, pages 66602–66656, 2025. URLhttps://proceedings.iclr.c/pape r_files/paper/2025/file/a6a90bcc2a470c3871b2d39a67d26e8-Paper-Conference.pdf. AREALSWE-BENCH Details and Examples (a) Repository Python Go JavaScript TypeScript (b) Language flipt-io/flipt ansible/ansible django/django sphinx-doc/sphinx navidrome/navidrome internetarchive/openlibrary element-hq/element-web gravitational/teleport qutebrowser/qutebrowser protonmail/webclients NodeBB/NodeBB future-architect/vuls matplotlib/matplotlib scikit-learn/scikit-learn astropy/astropy pydata/xarray pylint-dev/pylint pytest-dev/pytest sympy/sympy tutao/tutanota mwaskom/seaborn 0 10 20 30 40 Number of tasks 55.1% 28.6% 14.7% 1.6% Figure 5: Coverage of REALSWE-BENCH across repositories and programming languages. This section describes what REALSWE-BENCH contains: the benchmarks it is derived from, the repositories and programming languages it spans, the size of the gold patches its tasks require, a comparison of the selected tasks with the excluded ones, and examples of both the construction pipeline and the resulting tasks. A.1 Source Benchmarks REALSWE-BENCH is derived from SWE-BENCH VERIFIED [20] and SWE-BENCH PRO [8]. We choose these two benchmarks because they represent widely used repository-level coding-agent evaluation. SWE-BENCH VERIFIED.SWE-BENCH VERIFIED is a human-validated benchmark for stan- dardized model comparison. Ninety-three professional software developers experienced in Python reviewed a random sample of 1,699 tasks from the SWE-BENCH test set [15] and removed those whose issue description is not well-specified or whose unit tests are inappropriately scoped, leaving 500 tasks. It contributes all 500, which come entirely from Python repositories. SWE-BENCH PRO.SWE-BENCH PRO contributes 731 tasks that are more difficult and longer- horizon, drawn from larger and more complex repositories. Unlike SWE-BENCH VERIFIED, it covers languages beyond Python (§A.2). Every task carries Requirements alongside the problem statement, and some also carry an Interface; §B.2 describes how we handle them. Why not DEEPSWE? We also considered DEEPSWE [13] as a source benchmark but could not use it. Each variant in REALSWE is built by redistributing the text of the original problem statement across the taxonomy fields without rewriting it (§3.3), so a field the original does not contain cannot be produced. Supporting arbitrary combinations of information categories therefore requires every taxonomy field to be present in the source problem statement: the five fields[P][D][R][E][A]for bug fixes and the three fields[P][M][A]for feature requests (Table 1). Applying this criterion to the 13 Table 5: Size of the gold patch in REALSWE-BENCH. Files touchedLines edited TasksMeanMedianMeanMedian REALSWE-BENCH3814313462 ,→ bug fixes192328236 ,→ feature requests18954187118 113 DEEPSWE tasks leaves only 19 candidates, all of them feature requests. Moreover, DeepSeek V4 Pro-preview, the model with the highest resolution rate among the four models in Table 4, achieves a resolution rate of 0% on these tasks even from the original problem statements, leaving no baseline against which to measure the effect of realistic requests. A.2 Repositories and Programming Languages REALSWE-BENCH spans 21 repositories (Figure 5(a)) and four programming languages (Figure 5(b)). Python accounts for 55.1% of the tasks. Because SWE-BENCH VERIFIED contains only Python repositories, every task in the other three languages comes from SWE-BENCH PRO. Repository and language metadata follow the SWE-BENCH VERIFIED and SWE-BENCH PRO releases on Hugging Face. A.3 Gold Patch Size The size of the gold patch—the number of files it touches and the number of lines it adds or removes— indicates how much work a task requires. The gold patch for a task in REALSWE-BENCH touches 4 files and edits 134 lines on average (Table 5). Feature requests require larger patches than bug fixes: 5 files and 187 lines on average, compared with 3 files and 82 lines. This suggests that feature requests are the more difficult of the two task types, and indeed every model in Table 2 has a lower resolution rate on feature requests than on bug fixes. As discussed in §A.4, this indicates that the tasks selected for REALSWE are not meaningfully easier and do not contain smaller gold patches compared to other tasks in SWE-BENCH, and that the selection is therefore not biased. A.4 Selection Bias Of the 1,229 tasks in the original pool—the 500 SWE-BENCH VERIFIED tasks and the 729 SWE- BENCH PRO tasks that remain after removing the two whose task type is other (Appendix B)—403 contain all required information categories, and excluding the 22 candidates that receive the lowest score on any critical criterion leaves 381. To check whether this selection favors easier or smaller tasks, or concentrates the benchmark on a few repositories, we compare the selected 381 tasks with the excluded 848 tasks on resolution rate, patch size, and repository distribution (Table 6). The resolution rate is that of DeepSeek V4 Pro on the original problem statements, reported as the mean over three runs with one standard deviation. We measure repository distribution by the effective number of repositories, N eff = 1 P k i=1 p 2 i , wherep i is the share of the group’s tasks drawn from repositoryiandkis the number of repositories the group spans. The denominator is the Herfindahl–Hirschman index, soN eff is the number of equally sized repositories that would give the same concentration; a larger value means the tasks are spread more evenly. On none of the three measures are the selected tasks easier, smaller, or more concentrated in particular repositories. They are in fact harder (53.9% against 63.0%), larger (134 lines against 94), and spread more evenly (13.9 effective repositories against 11.0). Figure 6 shows where that last difference comes from: django/django alone accounts for 22.5% of the excluded tasks, while no repository exceeds 12.3% of the selected ones. 14 Table 6: The selected 381 tasks compared with the excluded 848 tasks. Lines editedRepositories TasksResolution rate (%)MeanMedianCountEffective, N eff Selected38153.9 ±0.6 134622113.9 Excluded84863.0 ±1.2 94382311.0 flipt-io/flipt ansible/ansible django/django sphinx-doc/sphinx navidrome/navidrome internetarchive/openlibrary gravitational/teleport element-hq/element-web qutebrowser/qutebrowser protonmail/webclients future-architect/vuls NodeBB/NodeBB matplotlib/matplotlib scikit-learn/scikit-learn sympy/sympy astropy/astropy pydata/xarray tutao/tutanota pytest-dev/pytest pylint-dev/pylint mwaskom/seaborn psf/requests pallets/flask 0 5 10 15 20 Share of tasks in each group (%) Selected (381 tasks) Excluded (848 tasks) Figure 6: Each repository’s share of the selected tasks and of the excluded tasks. A.5 Example Tasks Figure 7 shows the input and output of the construction pipeline for a single task (§3.3), and Figure 8 gives four example tasks from REALSWE-BENCH. Nitpick flags Literal annotation values as missing py:class ### Describe the bug When a value is present in a type annotation as `Literal`, sphinx will treat the value as a `py:class`. With nitpick enabled, values like `Literal[True]` end up failing, because `True` is not a class. This is a problem for builds which want to use `-n -W` to catch doc errors. ### How to Reproduce Setup a simple function which uses Literal, then attempt to autodoc it. e.g. ```python import typing @typing.overload def foo(x: "typing.Literal[True]") -> int: ... @typing.overload def foo(x: "typing.Literal[False]") -> str: ... def foo(x: bool): """a func""" return 1 if x else "foo" ``` I've pushed an example [failing project](https://github.com/sirosen/repro/tree/master/sphinxdoc/literal) to [my repro repo](https://github.com/sirosen/repro). Just run `./doc.sh` with `sphinx-build` available to see the failing build. ### Expected behavior ... SWE-bench (Input) Fix nitpick so it stops flagging `Literal` annotation values as missing `py:class`. When a value shows up in a type annotation as `Literal`, Sphinx currently treats that value like a `py:class`. With nitpick enabled, cases like `Literal[True]` fail because `True` is not a class. This breaks builds that use `-n -W` to catch doc errors. Keep `Literal[True]` (or any other literal value) in the type annotation, but do not trigger a nitpick warning for it. Set up a simple function that uses `Literal`, then autodoc it. e.g. ```python import typing @typing.overload def foo(x: "typing.Literal[True]") -> int: ... @typing.overload def foo(x: "typing.Literal[False]") -> str: ... def foo(x: bool): """a func""" return 1 if x else "foo" ``` Use this example [failing project](https://github.com/sirosen/repro/tree/master/sphinxdoc/literal) from [my repro repo](https://github.com/sirosen/repro). Just run `./doc.sh` with `sphinx-build` available to reproduce the failing build. Linux 3.8, 3.9 4.1.2 autodoc https://github.com/sirosen/repro/tree/master/sphinxdoc/literal RealSWE (Output) instance_id: sphinx-doc__sphinx-9602 Figure 7: The input and output of the construction pipeline for a single task. The original carries all five bug-fix fields of Table 1, and the all-field variant[PDREA]preserves that information. It differs only in linguistic style—casual, imperative, confident, and non-first-person—as set in Appendix B.3. 15 RealSWE-bench （Bug fix） Information Composition [P][PA] instance_id: sphinx-doc__sphinx-7440 Fix the glossary duplicate-term error caused by different casing. ``` Warning, treated as error: doc/glossary.rst:243:duplicate term description of mysql, other instance in glossary ``` instance_id: matplotlib__matplotlib-20676 Fix interactive SpanSelector forcing the axes limits to include 0. The x-axis limits get expanded to include x=0. attn @ericpre RealSWE-bench （Feature request） instance_id: django__django-13809 Add a `--skip-checks` option to the `runserver` command. instance_id: django__django-16116 Make `makemigrations --check` just exit without creating migrations. The decision to still write migrations doesn't seem to have been discussed on ticket:25604 or https://groups.google.com/g/django- developers/c/zczdY6c9KSg/m/ZXCXQsGDDAAJ. This came up while reading a PR that adjusts the documentation for `migrate --check`. The current documentation seems silent on this behavior. Figure 8: Four tasks from REALSWE-BENCH, one for each combination of task type (bug fix, feature request) and information composition ([P], [PA]). B Construction Pipeline Details This section reports what each step of the construction pipeline described in §3.3 produced, and why it is designed the way it is. Figure 3 shows the pipeline as a whole; Appendix F validates each step and the resulting task quality. B.1 Task-Type Classification We classify each task as a bug fix or a feature request using GPT-5.4 [21]. In SWE-BENCH PRO, a model is normally given the problem statement together with its Requirements and Interface; we classify from the problem statement alone, so that the classification reflects what the issue reports rather than the specification added for evaluation. Because some tasks are neither a bug fix nor a feature request, we offer other as a third option. Table 7 reports the counts. The two benchmarks differ substantially: 86.4% of the SWE-BENCH VERIFIED tasks are bug fixes, compared with 56.5% for SWE-BENCH PRO. Only two of the 1,231 tasks are assigned other, both from SWE-BENCH PRO. Both are refactorings: they neither fix incorrect behavior nor add new or changed functionality. We exclude them and carry the remaining 1,229 tasks into the next step. That so few fall outside the two types indicates that bug fixes and feature requests cover essentially all tasks in both benchmarks. B.2 Information Decomposition This step divides each problem statement into the information fields of Table 1. It redistributes the original text across those fields without rewriting it, so that decomposition does not alter the information the problem statement carries. Appendix H gives the prompt; Appendix D reports the resulting field distributions in full. Table 7: Distribution of task types in SWE-BENCH VERIFIED and SWE-BENCH PRO. VERIFIEDPROTotal Bug fix432 (86.4%)413 (56.5%)845 (68.6%) Feature request68 (13.6%)316 (43.2%)384 (31.2%) Other0 (0.0%)2 (0.3%)2 (0.2%) Total5007311,231 16 Problem statements often retain template artifacts such as Markdown headings and HTML tags. These are not what actual users type, and leaving them in misdirects the model: when a user has written something that does not match what the template asked for, a section heading still in place leads the model to assign the line by the heading rather than by its content. We therefore remove these artifacts before decomposition, keeping everything the user wrote. Unlike tasks in SWE-BENCH VERIFIED, tasks in SWE-BENCH PRO are divided into three separate components: Problem Statement, Requirements, and Interface. The Problem Statement describes the problem to be solved; the Requirements specify the expected behavior that the unit tests will check, and never include specific code implementation or leak the solution; and the Interface provides code metadata, such as function signatures, that helps reduce false negatives during unit-test evaluation [8]. To heuristically approximate a realistic coding environment, our pipeline merges the Problem State- ment and Requirements into a single task description before processing it. The Interface is excluded from the transformation pipeline and instead provided to the agent, when available, as an unmodified external document. A task family can express arbitrary combinations of information categories only if every field of Table 1 is present—five for a bug fix and three for a feature request. The original problem statement supplies all of them in 302 problems; Appendix D reports how often each field appears. Unlike the other fields, Environment Information[E]does not depend on what the issue author chose to write down: it is an objective property of the execution environment. For the 101 bug fixes whose statements carry only[PDRA]we therefore take it from the execution container released with its source benchmark, bringing the candidate set to 403. We add it to the original problem statement of those tasks as well, so the original carries the same information as the variants. Quality control then reduces the 403 to the 381 task families we release (Appendix F). B.3 Rephrasing in Real-User Style Every variant in REALSWE-BENCH and in Table 4 shares one style setting. Following the SWE- CHAT distributions in Figure 2, we target the majority category of each linguistic dimension: casual for formality, imperative for sentence type, confident for certainty, and non-first-person for perspective. Appendix H gives the prompt. We fix all four dimensions this way, but REALSWE-FRAMEWORK leaves them open: researchers specify a task type, an information composition, and a linguistic style, and the framework assembles the corresponding dataset on demand (Figure 3), supporting custom configurations and controlled ablations. CSWE-CHAT: Prompt Collection and Filtering SWE-CHAT [4] collects the session transcripts that the Entire CLI records, with the developer’s consent, for coding agents including Claude Code, OpenCode, Gemini CLI, Cursor, and Factory AI Droid. The snapshot we use contains 5,851 sessions (the SWE-CHAT paper reports more than 6,000). Because SWE-BENCH typically evaluates an agent from a single problem statement without further interaction, we retain only the first user request from each session. SWE-CHAT annotates each user (a) Selection selected1,589 (27.2%) other4,262 (72.8%) (b) User-authored filter user-authored718 (45.2%) removed871 (54.8%) (c) What was removed LLM-generated 542 (62.2%) Tool-injected 273 (31.3%) other56 (6.4%) Figure 9: Filtering SWE-CHAT down to the 718 prompts this paper analyzes. 17 SWE-chat Verified Pro Bug fix P 270 PA 95 PD 15 PE 1 PR 20 PDA 3 PDE 0 PDR 1 PEA 4 PRA 6 PRE 4 PDEA 1 PDRA 2 PDRE 0 PREA 3 PDREA 1 R 1 P 23 PA 44 PD 21 PE 2 PR 49 PDA 18 PDE 1 PDR 31 PEA 3 PRA 57 PRE 23 PDEA 2 PDRA 25 PDRE 40 PREA 21 PDREA 72 R 0 P 0 PA 1 PD 109 PE 0 PR 1 PDA 51 PDE 3 PDR 120 PEA 0 PRA 0 PRE 0 PDEA 2 PDRA 76 PDRE 18 PREA 0 PDREA 32 R 0 Feature request P 191 PA 74 PM 14 PMA 11 MA 1 P 6 PA 4 PM 22 PMA 36 MA 0 P 2 PA 3 PM 149 PMA 162 MA 0 Figure 10: Full distributions of information composition across SWE-CHAT, SWE-BENCH VERI- FIED, and SWE-BENCH PRO; each cell gives the number of prompts, and darker cells indicate larger shares. prompt with one of eight intents; we keep the two that correspond to our task types, create new code and debug, which leaves 1,589 requests (Figure 9(a)). Reviewing these requests, we found that a prompt recorded in a user turn is not necessarily written by the user. Text produced by an agent and wrapping injected by a tool sit in the user’s place, and together they account for more than half of the set (Figure 9(b)). We are measuring how users actually write their requests, so we remove them. Of the 871 removed requests, 542 (62.2%) are LLM-generated and 273 (31.3%) are tool-injected (Figure 9(c)). A single source dominates each: 523 of the 542 are plan-mode output from Claude Code, and 221 of the 273 are prompts wrapped by Conductor. The remaining 56 are personal templates whose source we could not verify and requests that give only an issue URL or a slash command, leaving the task itself outside the prompt. The remaining 718 are the user-authored prompts this paper analyzes. We apply the information taxonomy and the linguistic dimensions to them to measure what real requests contain and how they are written; Appendix D reports the resulting distributions. D Benchmark–Reality Mismatch D.1 Information Composition Problems in SWE-BENCH VERIFIED and PRO are information-rich (Figure 10). Among the 845 bug fixes the most common combinations are[PDR](17.9%) and[PD](15.4%); the top two cover only a third, and the remainder spreads over fourteen further combinations. The 384 feature requests are almost entirely[PMA](51.6%) and[PM](44.5%), which together account for 96.1%. Only 7% of problems consist of [P] or [PA] alone (8.0% for bug fixes, 3.9% for feature requests). Real requests run the other way.[P]and[PA]account for 88% of the 718 prompts (85.5% of bug-fix and 91.1% of feature requests). A prompt carries 1.4 information fields on average, half the 2.9 of a benchmark problem, and 64.2% consist of the problem statement[P]and nothing else; the same figure for the benchmark is 2.5%. Reading field by field shows where the gap sits (Table 8).[P]is present in almost every prompt on both sides, so the difference lies entirely in the supporting fields. Reproduction Steps[R]appear 18 Table 8: Share of prompts (%) containing each information field. Fields co-occur, so a column does not sum to 100. Bug fixFeature request SWE-CHATVerified + ProSWE-CHATVerified + Pro Prompts427845291384 [P] Problem Statement99.8100.099.7100.0 [D] Desired Behavior5.473.5– [R] Reproduction Steps8.966.9– [E] Environment Information3.325.9– [M] Motivation–8.996.1 [A] Additional Information26.947.829.653.4 SWE-chatVerifiedPro 01050100 8.2 84.8 100.0 Formal 86.8 13.4Casual 5.0 1.8Other Formality 01050100 18.9 89.6 89.0 Declarative 51.3 0.6 0.8 Imperative 16.4 0.2 Interrogative 13.4 9.6 10.2 Other Sentence type 01050100 80.4 74.2 99.3 Confident 15.5 25.0 0.5 Uncertain 4.2 0.8 0.1 Other Certainty 01050100 39.8 57.8 7.4 First-person 60.0 42.2 92.6 Non-first-person 0.1 Other Perspective Figure 11: Distributions of linguistic style along the four dimensions, as the share of prompts (%). Blank cells are zero. in 8.9% of real requests against 66.9% of benchmark problems, and Environment Information[E] in 3.3% against 25.9%. The widest gaps belong to Desired Behavior[D]and Motivation[M], the fields that raise the resolution rate most for bug fixes and for feature requests respectively (Table 4): 5.4% against 73.5% for[D], and 8.9% against 96.1% for[M]. This gap likely leads benchmarks to overestimate coding performance in real-world settings. D.2 Linguistic Style The linguistic dimensions exhibit a less uniform pattern (Figure 11). Formality and sentence type divide the two sources most sharply: 86.8% of real user requests are casual and 51.3% are imperative, whereas 84.8% of SWE-BENCH VERIFIED and 100% of PRO prompts are formal and approximately 89% of prompts in both are declarative. SWE-BENCH prompts read as polished issue reports rather than conversational user requests. Certainty and perspective show no consistent separation. E Additional Results and Analyses E.1 Statistical Analysis Our statistical analysis examines whether the performance differences underlying the main compar- isons are robust to variation across repeated runs and repositories, while quantifying their magnitude and uncertainty. We distinguish the roles of the reported statistics as follows. 19 StatisticWhat it shows Effect size ∆Direction and magnitude Repository-clustered 95% CIEffect-size uncertainty across repositories Exact McNemar + Holm-adjusted pModel-level paired inference Repository sign-flip pAggregate and model-gap inference All analyses use matched comparisons in which different input conditions are applied to the same tasks. The main text reports the mean and sample standard deviation of three independent runs. Ranges and averages quoted in the main text are computed from unrounded values; table entries are rounded to one decimal place, so recomputing from the displayed entries may differ by 0.1 p. For inference, we define a task as resolved under a condition when at least two of the three runs are resolved. For a given model, lety icr ∈0, 1denote the grader outcome for taski, conditionc, and run r. The task-level outcome and paired effect are Y ic = 1 " 3 X r=1 y icr ≥ 2 # , b ∆ A→B = 100 n n X i=1 (Y iB − Y iA ). For model-level paired comparisons, we use a two-sided exact McNemar test and apply Holm correction when multiple comparisons jointly support one claim. Effects aggregated across models and the selected model-gap contrast instead use repository-level sign-flip tests. We obtain all 95% confidence intervals from 10,000 repository-clustered bootstrap samples, preserving dependence among tasks from the same repository. The intervals are not multiplicity-adjusted and are not dual to either hypothesis test, so exclusion of zero need not coincide with p < .05. E.1.1 Statistical Support for the Main Findings Table 9 summarizes the statistical evidence for the benchmark–reality gap, linguistic style, and information-field ablation. The paired Original–REALSWE-BENCH comparisons show that the benchmark–reality gap persists when evaluated on the same tasks. Both task types exhibit a decrease, with a larger point estimate for bug fixes. The selected model contrast further supports the finding that realistic inputs can alter relative model comparisons, rather than merely lowering all scores uniformly. Changing linguistic style while holding information fixed produces no consistent direction across models or task types. The information-field results instead distinguish the value of request content: removing[A],[E], or[R]produces no consistent degradation, whereas Desired Behavior[D] contributes robustly across all four models. Motivation[M]has a positive average contribution, but its evidence is more dependent on the model and repository. Taken together, the lack of a systematic style effect and the field-specific ablation pattern indicate that performance is more sensitive to which information a request provides than to how it is expressed or how much information it provides. E.1.2 Robustness Across Information Contexts The cumulative ablation in the main text isolates each transition, but it does not by itself establish whether the key-field effects depend on that particular path. We therefore compare additional matched pairs that differ only in Desired Behavior or Motivation while holding the surrounding information fixed (Table 10). Desired Behavior has a positive point estimate in every surrounding information context considered. Although the strength of individual comparisons varies after multiplicity correction, its benefit is not confined to the cumulative ablation path used in the main analysis. Motivation likewise retains a positive direction with and without[A], but with greater uncertainty and model variation. Overall, the direction of both key-field effects persists across surrounding information contexts, while their magnitude and precision remain context- and model-dependent. 20 Table 9: Statistical support for the main comparisons. Positive∆in the MiMo–Qwen row indicates a shift toward MiMo V2.5 Pro; the benchmark–reality and Motivation estimates aggregate seven and four models, respectively. The two linguistic-style rows form one eight-test Holm family, while each information-field comparison forms a separate four-model family. All effects use task-level majority outcomes, brackets report repository-clustered 95% CIs, and bold denotes Holm-adjusted p < .05. Panel A. Main-text comparisons ComparisonScope∆ (p)95% CI Repository sign-flip p Benchmark–reality gap Original→ REALSWE-BENCH All tasks−6.6[−8.3,−5.0]< .001 Bug fix−9.6[−13.1,−6.5]< .001 Feature request −3.6[−7.6,−1.0].008 MiMo V2.5 Pro margin over Qwen3.7 Plus Original→ REALSWE-BENCHAll tasks+3.7[+0.7, +7.3].040 Motivation effect [P]→ [PM]Feature request+3.8[+0.2, +8.2].076 Panel B. Linguistic style Comparison DeepSeek V4 Pro DeepSeek V4 Flash MiMo V2.5 Pro MiMo V2.5 Bug: Original→ [PDREA] −1.6 [−4.3, +1.1] Holm p = 1.000 −1.6 [−5.2, +2.5] Holm p = 1.000 +1.0 [−2.3, +5.1] Holm p = 1.000 0.0 [−3.0, +2.2] Holm p = 1.000 Feature: Original→ [PMA] −2.1 [−8.8, +3.6] Holm p = 1.000 +1.6 [−2.6, +5.4] Holm p = 1.000 0.0 [−4.1, +4.9] Holm p = 1.000 −5.8 [−10.7,−0.8] Holm p = .346 Panel C. Information-field ablation Remove [A]: [PDREA]→ [PDRE] −3.1 [−6.5, +0.5] Holm p = .438 +2.1 [−1.8, +6.0] Holm p = .848 −1.0 [−5.6, +2.9] Holm p = .848 −3.1 [−5.6,−0.6] Holm p = .438 Remove [E]: [PDRE]→ [PDR] +2.6 [−1.1, +6.2] Holm p = .801 −2.1 [−5.1, +0.9] Holm p = .801 −0.5 [−4.4, +2.9] Holm p = 1.000 +6.2 [+1.4, +10.3] Holm p = .017 Remove [R]: [PDR]→ [PD] −1.0 [−4.1, +1.9] Holm p = 1.000 −1.0 [−5.6, +3.8] Holm p = 1.000 +1.0 [−2.9, +4.9] Holm p = 1.000 −3.6 [−7.8, +1.1] Holm p = .474 Add [D]: [P]→ [PD] +9.9 [+4.5, +16.0] Holm p = .002 +8.9 [+3.8, +15.4] Holm p = .003 +8.3 [+3.4, +12.8] Holm p = .005 +6.8 [+2.2, +10.9] Holm p = .007 Add [M]: [P]→ [PM] +0.5 [−4.4, +5.6] Holm p = 1.000 +9.5 [+3.6, +16.8] Holm p = .001 +3.2 [−1.2, +8.2] Holm p = .714 +2.1 [−2.0, +7.1] Holm p = 1.000 21 Table 10: Robustness of Desired Behavior[D]and Motivation[M]across surrounding information contexts on bug-fix and feature-request tasks, respectively. Additional context runs are limited to the two DeepSeek models due to computational cost. Descriptive effects are three-run mean resolution- rate differences; inferential effects use task-level majority outcomes with repository-clustered 95% CIs, and all effects are in percentage points. For repeated main-path contrasts, inferential effect estimates and CIs are reused from Table 9, whereas Holm-adjustedp-values follow the 10-[D]and 4-[M] families defined here; bold denotes adjusted p < .05. Matched pairModel DescriptiveInference 3-run ∆ [95% CI]Holm-adjusted p Desired Behavior [D] [P]→ [PD] DeepSeek V4 Pro+8.5+9.9 [+4.5, +16.0].004 DeepSeek V4 Flash+7.5+8.9 [+3.8, +15.4].005 [PA]→ [PDA] DeepSeek V4 Pro+6.4+6.2 [+1.2, +11.5].058 DeepSeek V4 Flash+7.6+8.3 [+3.5, +12.9].012 [PR]→ [PDR] DeepSeek V4 Pro+8.7+9.4 [+5.6, +13.1].002 DeepSeek V4 Flash+7.3+6.2 [+1.2, +10.5].051 [PRE]→ [PDRE] DeepSeek V4 Pro+8.2+7.8 [+3.4, +12.1].012 DeepSeek V4 Flash+10.4+10.4 [+6.5, +14.6].001 [PREA]→ [PDREA] DeepSeek V4 Pro+6.9+8.9 [+5.2, +12.8].001 DeepSeek V4 Flash+7.5+5.7 [+0.7, +10.2].058 Motivation [M] [P]→ [PM] DeepSeek V4 Pro+2.8+0.5 [−4.4, +5.6]1.000 DeepSeek V4 Flash+7.1+9.5 [+3.6, +16.8].001 [PA]→ [PMA] DeepSeek V4 Pro+1.9+3.7 [−2.2, +9.6].430 DeepSeek V4 Flash+2.6+1.6 [−2.2, +6.5]1.000 22 F Validation and Quality Control We validate two high-level claims from §3.4: (i) the reliability of the construction pipeline and the quality of the resulting task families, and (i) the real-user linguistic style used to construct REALSWE-BENCH. Table 11 summarizes these validation blocks and the evidence used in each. Separately, §A.4 audits whether the selection process disproportionately retains easier or smaller tasks or concentrates the selected pool in a few repositories. F.1 Pipeline Validation and Task Quality We evaluate whether each construction stage makes only its intended transformation: i) classification assigns each task to the correct task type, i) cleaning removes template artifacts without deleting user content, i) field assignment places each content unit in the appropriate taxonomy field, and iv) rephrasing changes expression without altering task information or intent. We call a field assignment fully appropriate when every content unit matches its field definition. The complete scoring criteria are provided in Table 12. Agreement with human judgment (Table 13; Figure 12) supports applying the quality-control judge to all candidates; the resulting gate removes 22 critical decomposition failures and no rephrasing failures, leaving 381 task families. We additionally validate the SWE-CHAT field assignments that define the target information composition. Table 11: Human-Validation map. Each row corresponds to a validation block in Sections F.1 and F.2. Validation blockQuestionEvidence Pipeline validation and task quality (F.1) Do the pipeline stages produce their intended outputs without critical failures? Human audit + judge validation Linguistic-style validation (F.2) Do the linguistic annotations reliably characterize real-user style, and does REALSWE-BENCH realize it? Pairwise comparison + human re-annotation Table 12: Operational three-point rubrics. Score 1 denotes a critical failure. The full-pool gate excludes a candidate if any non-empty decomposition field or rephrasing criterion receives score 1. Human annotators and the quality-control judge use the same definitions for these two stages. CriterionScore 3Score 2Score 1 Cleaning: over-deletion No user-authored content is removed. Minor user-authored content is removed without affecting the task. Task-relevant user content is removed. Cleaning: under-deletion All target template artifacts are removed. Minor template artifacts remain without obscuring the request. Substantial template artifacts remain, or cleaning is effectively absent. Information-field assignment Every sentence and content unit matches the definition of its assigned field. Some content would fit another field better. All or nearly all content is assigned to an inappropriate field. Rephrasing: information preservation All task information and technical literals, including code, errors, tracebacks, and paths, are preserved; no task-relevant claim is added. A minor addition or omission does not change the required solution, and technical literals remain intact. Material information is added or removed, or a technical literal is altered. Rephrasing: meaning and intent preservation Scope, causal relations, requirement strength, and implementation objective are unchanged. A minor semantic shift or ambiguity leaves the implementation objective unchanged. Scope, causal relations, requirement strength, or implementation objective is materially changed. 23 Table 13: Pipeline validation and task-quality results. Agreement percentages compare the two human annotators; parenthetical values report Cohen’sκor quadratic-weightedκ(QWK). Judge–human results use human consensus as the reference. StageAnnotator agreementValidation result Data preprocessing Task-type classification88.0% (κ = 0.75)95% correctly classified Template cleaning95–96% (QWK 0.81–0.85)Critical failures: 1% user-content loss; 2% retained template artifacts Information-composition annotation SWE-CHAT field assignment 91.2% (QWK 0.40)93.2% fully appropriate Task-family construction Field decomposition92.7% (QWK 0.62)97% judge–human match (macro-F 1 0.83) Linguistic rephrasing95% information / 94% intent99% judge–human match (macro-F 1 0.75) BugFeature Bug Feature 64 95.5% 3 4.5% 2 6.1% 31 93.9% Task type (n=100 tasks) 123 1 2 3 4 80.0% 1 20.0% 0 0.0% 0 0.0% 19 95.0% 1 5.0% 2 0.6% 9 2.6% 336 96.8% Decomposition (n=372 fields) 23 2 3 0 0.0% 1 100.0% 1 1.0% 98 99.0% Information (n=100 tasks) 23 2 3 1 100.0% 0 0.0% 0 0.0% 99 100.0% Meaning/intent (n=100 tasks) Figure 12: Construction error structure. Rows are human labels and columns are pipeline or judge outputs; cells report counts and row-normalized percentages. Decomposition disagreements remain near the score boundary, and no human score-1 field is assigned judge score 3. Rephrasing disagreements occur only for information preservation between scores 2 and 3. F.2 Linguistic-Style Validation We validate both the linguistic-dimension annotations used to characterize real-user style and the rephrasing used to construct REALSWE-BENCH. F.2.1 Real-User-Style Rephrasing We validate whether the rephrasing used to construct REALSWE-BENCH reflects the real-user linguistic style characterized in §3.2. Pairwise judgments (Table 14) favor the rephrased requests in formality and sentence type, the two dimensions with the clearest SWE-CHAT–SWE-BENCH contrast. The weaker effects in certainty and perspective mirror their less consistent source-level differences, together supporting the intended real-user-style rephrasing. F.2.2 Linguistic-Dimension Annotation We validate whether the linguistic-dimension annotations underlying the benchmark–reality com- parison reflect human judgment. Table 15 shows close agreement with human annotations, while Figures 13 and 14 locate the remaining errors in sparse minority classes. Human re-annotation pre- serves the primary linguistic contrast used to define the real-user style: SWE-CHAT is predominantly casual and imperative, whereas SWE-BENCH is predominantly formal and declarative. 24 Table 14: Blind pairwise comparison of the restructured and rephrased requests used to construct REALSWE-BENCH. For each dimension, the rubric selects the candidate closer to the target category— casual, imperative, confident, or non-first-person—or a tie. Decision columns report the most common judgment and its share. To mitigate position bias, full-pool judge results average the original and swapped candidate orders after mapping both judgments back to their source labels. Human decisions aggregate the two annotators’ 200 source-normalized judgments per dimension on a matched sample of 100 pairs; judge–human agreement averages exact agreement across both annotators and both candidate orders on the same sample. DimensionJudge decisionHuman decision Human agreement (κ) Judge–human agreement FormalityRephrased closer (80.8%) Rephrased closer (82.0%)0.6484.8% Sentence typeRephrased closer (97.4%) Rephrased closer (97.0%)0.9297.5% CertaintyRephrased closer (43.7%) Rephrased closer (61.5%)0.8369.3% PerspectiveTie (79.7%)Tie (69.0%)0.7591.5% Table 15: Human validation of the linguistic-dimension annotations underlying the benchmark–reality comparison. IAA denotes raw inter-annotator agreement, with Cohen’sκin parentheses. Classifier accuracy and macro-F 1 use human consensus as the reference. SWE-CHATSWE-BENCH Dimension IAA (κ) Accuracy (macro-F 1 ) IAA (κ) Accuracy (macro-F 1 ) Formality82% (0.48)88% (0.76)88% (0.15)93% (0.73) Sentence type86% (0.77)89% (0.81)91% (0.37)88% (0.40) Certainty92% (0.79)93% (0.85)90% (0.61)99% (0.97) Perspective94% (0.87)96% (0.65)98% (0.95)98% (0.97) Formal Casual Other Formal Casual Other 6 35.3% 10 58.8% 1 5.9% 0 0.0% 77 98.7% 1 1.3% 0 0.0% 0 0.0% 5 100.0% Formality Declarative Imperative Interrogative Other Declarative Imperative Interrogative Other 13 86.7% 0 0.0% 0 0.0% 2 13.3% 2 3.5% 54 94.7% 0 0.0% 1 1.8% 0 0.0% 1 4.5% 17 77.3% 4 18.2% 1 16.7% 0 0.0% 0 0.0% 5 83.3% Sentence type ConfidentUncertain Other Confident Uncertain Other 75 98.7% 0 0.0% 1 1.3% 2 13.3% 13 86.7% 0 0.0% 4 44.4% 0 0.0% 5 55.6% Certainty First-person Non-first-person Other First-person Non-first-person Other 32 100.0% 0 0.0% 0 0.0% 0 0.0% 64 100.0% 0 0.0% 1 25.0% 3 75.0% 0 0.0% Perspective Figure 13: SWE-CHAT annotation errors. Rows are human consensus and columns are classifier outputs; cells report counts and row-normalized percentages. Errors are concentrated in sparse minority labels, while the dominant casual and imperative classes are preserved. Formal Casual Other Formal Casual Other 90 94.7% 5 5.3% 0 0.0% 2 66.7% 1 33.3% 0 0.0% 0 0.0% 0 0.0% 2 100.0% Formality Declarative Imperative Interrogative Other Declarative Imperative Interrogative Other 85 92.4% 0 0.0% 0 0.0% 7 7.6% 0 0.0% 1 25.0% 0 0.0% 3 75.0% 0 0.0% 0 0.0% 0 0.0% 2 100.0% 0 0.0% 0 0.0% 0 0.0% 2 100.0% Sentence type ConfidentUncertain Other Confident Uncertain Other 89 98.9% 1 1.1% 0 0.0% 0 0.0% 10 100.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0% Certainty First-person Non-first-person Other First-person Non-first-person Other 26 96.3% 1 3.7% 0 0.0% 1 1.4% 72 98.6% 0 0.0% 0 0.0% 0 0.0% 0 0.0% Perspective Figure 14: SWE-BENCH annotation errors in the same layout and scale as Figure 13. The low sentence-type macro-F1 reflects minority-class errors rather than failure to identify the dominant declarative class. 25 G Experimental Details Table 16 reports the exact model snapshot, serving provider, and inference setting for every model used in this work. Models routed through OpenRouter carry theopenrouter/prefix and are listed with their serving provider; the rest were accessed directly. The reasoning column gives the reasoning- effort setting, where default denotes models that expose only an on/off toggle and were run with reasoning on; Claude Haiku 4.5 exposes a thinking-token budget instead of an effort level. Table 16: Models used in this work. RoleModelProvider Model IDReasoning PipelineGPT-5.4– gpt-5.4-2026-03-05low Evaluated models DeepSeek V4 Pro-preview– deepseek/deepseek-v4-prohigh DeepSeek V4 Flash-preview– deepseek/deepseek-v4-flashhigh MiMo V2.5 ProXiaomi openrouter/xiaomi/mimo-v2.5-prodefault MiMo V2.5Xiaomi openrouter/xiaomi/mimo-v2.5default Claude Haiku 4.5– anthropic/claude-haiku-4-5-20251001 budget 16,384 Qwen3.7 PlusAlibaba openrouter/qwen/qwen3.7-plusdefault MiniMax M3MiniMax openrouter/minimax/minimax-m3default LLM-as-a-judgeGPT-5.6 Terra– gpt-5.6-terralow H LLM Prompts This section provides the complete prompts used at every point where this paper relies on an LLM. H.1 Task-Type Classification PROMPT_CLASSIFICATION = """ # Identity You are an annotator. Your task is to classify a GitHub issue into one of three types: "bug", "feature", or "other ". # Task You will receive one problem statement from a GitHub issue. Assign exactly one issue type: - bug: A bug report. The issue describes a problem in existing functionality. - feature: A feature request. The issue requests an addition or improvement to the codebase. - other: The issue does not clearly fit either a bug report or a feature request. # Rules 1. Your task is only to select the issue type. Do not modify , paraphrase , or extract any content from the problem statement. 2. Prefer bug report or feature request when the issue can reasonably be classified as either. Only assign other when neither a bug report nor a feature request applies. """ H.2 Template Artifact Cleaning PROMPT_CLEANING = """ # Identity You are a text cleaner. Your task is to remove GitHub -template artifacts from a GitHub issue ’s problem statement while preserving all user -authored content verbatim. # Task 26 You will receive a problem statement from a GitHub issue. Return it verbatim with all GitHub -template artifacts removed and all user -authored content preserved. The cleaned text will be restructured into predefined fields by a downstream pipeline: - Bug reports: describe_the_bug , expected_behavior , reproduction_code , environment , additional_context. - Feature requests: desired_solution , problem_motivation , additional_context. # Rules 1. The single criterion for removal is whether the content was authored by the user or was automatically inserted by the GitHub issue template. 2. Remove all template artifacts. These include any text that the issue template prompted or pre -filled , such as section labels naming a field the user was asked to complete , hidden instructions guiding the user , and boilerplate text referencing project conventions. 3. Preserve all user -authored content verbatim. This includes the user ’s own prose , code , error messages , references , and any markdown or headers the user added to organize their own writing. 4. When uncertain whether content is user -authored or a template artifact , preserve it . 5. If the problem statement contains no GitHub -template artifacts , return it unchanged . """ H.3 Information Decomposition _TEMPLATE_BUG = """ # Identity You are an annotator. Your task is to classify each line of owner into five predefined fields , preserving every line verbatim. # Task You will receive intro. Classify each line of noun verbatim into these five fields : - describe_the_bug: The core problem or the task to address , along with any output or error messages. - expected_behavior: A clear and concise description of what should have happened instead of the bug. - reproduction_code: Steps or code to reproduce the bug. - environment: Runtime environment details , such as OS , language version , and package versions. - additional_context: Any other context about the problem. # Rules 1. Preserve every line of noun exactly. Do not alter spelling , punctuation , whitespace within a line , or code. 2. Every non -blank line of noun must appear in exactly one field. Do not duplicate a line across fields , and do not drop any line. 3. Lines within a single code block or traceback must be kept together in the same field. When multiple code blocks or tracebacks exist , each can be assigned to a different field based on which field best matches its content. 4. additional_context is the catch -all: content that does not clearly fit the other four fields goes here. 5. If a field has no content , use the JSON value null without quotation marks. Never use the strings "null", "None", "N/A", or any generated placeholder to represent an empty field. """ _TEMPLATE_FEATURE = """ # Identity You are an annotator. Your task is to classify each line of owner into three predefined fields , preserving every line verbatim. # Task You will receive intro. Classify each line of noun verbatim into these three fields: 27 - desired_solution: A clear and concise description of the proposed change or desired behavior. - problem_motivation: A clear and concise description of the current problem or limitation. - additional_context: Any other context about the feature request. # Rules 1. Preserve every line of noun exactly. Do not alter spelling , punctuation , whitespace within a line , or code. 2. Every non -blank line of noun must appear in exactly one field. Do not duplicate a line across fields , and do not drop any line. 3. Lines within a single code block or traceback must be kept together in the same field. When multiple code blocks or tracebacks exist , each can be assigned to a different field based on which field best matches its content. 4. additional_context is the catch -all: content that does not clearly fit the other two fields goes here. 5. If a field has no content , use the JSON value null without quotation marks. Never use the strings "null", "None", "N/A", or any generated placeholder to represent an empty field. """ _TEMPLATE = "bug": _TEMPLATE_BUG , "feature ": _TEMPLATE_FEATURE SUBJECTS = "problem_statement ": "bug": "owner": "a GitHub bug report ’s problem statement", "intro": "one problem statement from a GitHub bug report", "noun": "the problem statement", , "feature ": "owner": "a GitHub feature request ’s problem statement", "intro": "one problem statement from a GitHub feature request", "noun": "the problem statement", , , "requirements ": "bug": "owner": "the bug -fix task ’s requirements", "intro": "the requirements of the bug -fix task", "noun": "the requirements", , "feature ": "owner": "the feature task ’s requirements", "intro": "the requirements of the feature task", "noun": "the requirements", , , H.4 Linguistic Rephrasing PROMPT_REPHRASING = """ # Identity You are a rephraser. Your task is to rephrase each field of a GitHub issue into content that users actually send to an AI coding agent (such as Claude Code , Codex , or Cursor) in the real world. # Input You will receive a structured GitHub issue , which is either a bug report or a feature request. Bug reports are structured with five fields: - describe_the_bug - expected_behavior - reproduction_code - environment - additional_context Feature requests are structured with three fields: - desired_solution 28 - problem_motivation - additional_context # Task Rephrase each field following the instructions below while preserving the original meaning and intent. # Rephrasing Instructions 1. Formality If the vocabulary or expressions are not casual , convert them to casual ones. 2. Sentence type Prefer imperative types. Declarative and interrogative types are acceptable when more natural. 3. Certainty If the vocabulary or expressions are not confident , convert them to confident ones. 4. Perspective Prefer non -first -person perspectives. First -person perspectives are acceptable when more natural. # Rules 1. Preserve code blocks , error messages , tracebacks , version numbers , file paths , and other technical content exactly as -is. 2. Preserve the original meaning and intent. """ H.5 Decomposition Quality Judgment PROMPT_JUDGE_DECOMPOSITION = """ # Role You are an evaluator assessing whether the content assigned to each field of a restructured GitHub issue matches that field ’s definition. # Task You will receive ‘restructured_ps ‘, a JSON object containing the fields of one restructured problem statement. For each field , judge only whether the content placed in that field belongs there according to its definition. ## Field Definitions field_descriptions # Scoring Assign one score to each non -null field. - 3: All sentences or content units in the field match the field definition. - 2: Some sentences or content units in the field do not match the field definition , but the rest do. - 1: All or nearly all sentences or content units in the field do not match the field definition. # Rules - Evaluate field assignment only. Do not judge whether the issue report is factually correct , sufficiently detailed , well written , or useful for solving the task. """ BUG_FIELD_DESCRIPTIONS = """ - describe_the_bug: The core problem or the task to address , along with any output or error messages. - expected_behavior: A clear and concise description of what should have happened instead of the bug. - reproduction_code: Steps or code to reproduce the bug. - environment: Runtime environment details , such as OS , language version , and package versions. - additional_context: Any other context about the problem. """ 29 FEATURE_FIELD_DESCRIPTIONS = """ - desired_solution: A clear and concise description of the proposed change or desired behavior. - problem_motivation: A clear and concise description of the current problem or limitation. - additional_context: Any other context about the feature request. """ H.6 Rephrasing Quality Judgment PROMPT_JUDGE_REPHRASING = """ # Role You are an evaluator assessing whether a GitHub issue , rephrased to resemble real - world requests to AI coding agents such as Claude Code , Codex , or Cursor , preserves previous context. # Task Compare ‘restructured_ps ‘ with ‘rephrased_ps ‘ as complete texts. Score the following two dimensions independently. ## Information Preservation Evaluate whether the substantive information in ‘restructured_ps ‘ remains represented in ‘rephrased_ps ‘. - 3: All information are preserved. - 2: Some informations are missing , but substantial informations to resolve the task are preserved. - 1: Substantial information is missing , so the original task cannot be adequately recovered. ## Meaning and Intent Preservation Evaluate whether ‘rephrased_ps ‘ preserves the meaning , intent , scope , conditions , relationships of ‘restructured_ps ‘. - 3: Meaning and intent are fully preserved. - 2: Some details are less precise or somewhat ambiguous , but ‘rephrased_ps ‘ alone still communicates the same problem and implementation objective. - 1: Meaning or intent is materially changed , or unsupported content is added in a way that could lead to a different implementation or acceptance target. # Rules - Judge each text as a whole , not sentence by sentence. - Allow paraphrasing , reordering. - Do not penalize stylistic changes unless they alter meaning , scope. """ H.7 Linguistic-Dimension Annotation PROMPT_DIMENSION_CLASSIFICATION = """ # Identity You are an annotator. Your task is to classify a user prompt sent to an AI coding agent along four linguistic dimensions. # Input You will receive a user prompt sent to an AI coding agent (such as Claude Code , Codex , or Cursor). The user prompt may take various forms. The following are some examples: - Natural language only. - Natural language with pasted code or error logs. - A short fragment. - A long multi -paragraph passage. # Task Assign exactly one category for each of the four dimensions: 1. formality: - formal: Uses formal vocabulary and expressions. - casual: Uses casual vocabulary and expressions. 30 - other: Does not clearly fit either category. 2. sentence_type: - declarative: A declarative type. - imperative: An imperative type. - interrogative: An interrogative type. - other: Does not clearly fit any single category. 3. certainty: - confident: Expresses certainty. - uncertain: Expresses uncertainty. - other: Does not clearly fit either category. 4. perspective: - first_person: Uses first -person pronouns. - non_first_person: Does not use first -person pronouns. - other: Does not clearly fit either category. # Rules 1. Your task is only to assign categories. Do not modify , paraphrase , or extract any content from the user prompt. 2. Assign exactly one category for each of the four dimensions. The four dimensions are independent of each other. 3. Use other only when the user prompt does not clearly fit any other category in each dimension. 4. If the user prompt contains content not authored by the user , classify based on the natural language authored by the user. # Output Example output: "formality ": "casual", "sentence_type ": "imperative", "certainty ": "confident", "perspective ": "first_person" """ H.8 Pairwise Style Comparison PROMPT_JUDGE_STYLE = """ # Identity You are an annotator comparing two expressions of the same software -engineering request. # Task For each of the four linguistic dimensions below , choose Candidate A, Candidate B, or tie. 1. formality Choose the candidate whose differing wording uses a more conversational and everyday lexical register , rather than formal , institutional , or specification -like language. 2. sentence_type Choose the candidate that is closer to an imperative. 3. certainty Choose the candidate that expresses greater certainty and confidence. 4. perspective Choose the candidate that is closer to a non -first -person perspective. # Rules 1. Judge the four dimensions independently. 2. For every dimension , output exactly A, B, or tie. 3. Evaluate only linguistic expression for the dimension being judged. 4. Do not reward a candidate merely for being longer , more detailed , or more technically complete. 5. Do not evaluate factual correctness , information preservation , task solvability , implementation quality , or solution quality. 31 6. For formality , compare only lexical choices that differ between the candidates. Ignore sentence type , perspective , length , and unchanged technical content. Output tie when the differing wording shows no meaningful lexical -register difference. """ 32

Instant research discovery