Paper deep dive
SWE-Together: Evaluating Coding Agents in Interactive User Sessions
Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, Shengzhi Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 7/5/2026, 3:12:05 AM
Summary
SWE-Together is a new multi-turn, interactive benchmark designed to evaluate coding agents by reconstructing real-world user-agent coding sessions. Unlike static benchmarks, it uses an LLM-based user simulator that provides reactive, state-conditional feedback (clarifications, corrections, new requirements) to simulate realistic human-agent collaboration. The benchmark evaluates agents based on two primary dimensions: final repository correctness (using an agentic rubric judge) and interaction quality (measured via 'User Correction' and 'Intent Coverage'). Experiments with frontier models show that stronger agents achieve higher success rates with fewer required interventions.
Entities (9)
Relation Signals (4)
SWE-Together → constructedfrom → DataClaw
confidence 100% · Four upstream sources contribute to the evaluated suite: DataClow...
SWE-Together → evaluates → OpenCode
confidence 100% · We evaluate seven frontier models on the full SWE-Together benchmark... using a common agent harness, opencode
SWE-Together → uses → LLM-based user simulator
confidence 100% · To replay these interactions across agents, we build a reactive LLM-based user simulator
User Correction → measures → interaction quality
confidence 90% · User Correction, which quantifies the amount of corrective steering provided by the simulated user
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.
Tags
Links
- Source: https://arxiv.org/abs/2606.29957v1
- Canonical: https://arxiv.org/abs/2606.29957v1
Trouble viewing inline? Open PDF directly →
Full Text
48,815 characters extracted from source content.
Expand or collapse full text
SWE-Together: Evaluating Coding Agents in Interactive User Sessions Yifan Wu ∗ , Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, Shengzhi Li ∗ Meta ∗ Project Co-Lead Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user–agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users’ intents and provides feedback when the coding agent’s progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience. Date: June 30, 2026 Correspondence: Yifan Wu at yifannnwu.com and Shengzhi Li at lishengzhi.com Code: https://github.com/Togetherbench/SWE-Together Website: https://togetherbench.com Figure 1 SWE-Together reframes coding-agent evaluation from one-shot SWE-Bench-style tasks into replayable, multi-turn sessions. Top: current mainstream coding benchmarks; Bottom: SWE-Together. Left: pass rates with average user-correction turns overlaid; Right: workflow comparison between existing benchmarks and SWE-Together. 1 arXiv:2606.29957v1 [cs.SE] 29 Jun 2026 1 Introduction Coding agents are increasingly used as software engineering assistants, and benchmarks for these agents have become prominent measures of capability in frontier model releases. Prominent examples include the SWE-Bench family and Terminal-Bench (Jimenez et al., 2024; OpenAI, 2024; Scale AI, 2025; Merrill et al., 2026). More broadly, many widely used coding benchmarks follow a static protocol: they present the full task description at the start and evaluate the agent’s submitted code by running executable tests (Chen et al., 2021; Austin et al., 2021; Jimenez et al., 2024; Zhuo et al., 2025b; Huang et al., 2026). This protocol has driven substantial progress in the development of coding agents. However, as mainstream coding benchmarks approach saturation and become less able to distinguish among frontier models, there is a growing need for evaluations that better reflect the experience of human users working with coding agents in real workflows. The mismatch between benchmarks and practice is twofold: task design and evaluation. First, most benchmarks cast tasks as fixed, single-turn instructions. Real-world coding assistance, by contrast, is inherently interactive: users often reveal intent incrementally, clarify incomplete requests, refine requirements, and correct prior outputs across turns (Zhong et al., 2025; Zhang et al., 2025a). Existing benchmarks therefore miss key conditions of real use, where task-relevant information may be distributed across multiple turns and initial requests may be far less complete than benchmark task descriptions (Laban et al., 2025; Garg et al., 2025). Second, evaluation primarily measures final-task success, but rarely capture how effectively agents incorporate evolving instructions and how much user effort is required. Two agents may receive the same final-task score while imposing very different burdens: one may succeed from a coarse initial request, while another may require a detailed specification, repeated reminders, and extensive corrective feedback. Evaluating coding agents therefore requires moving beyond final-task correctness to account for interaction quality and the human effort needed for success. Building such an evaluation entails a fundamental challenge: preserving realistic user interaction while making tasks verifiable. Real-world conversation logs capture how developers use coding agents, but raw sessions are often not benchmark-ready: they may lack a reproducible repository state, an identifiable user goal, or an observable outcome against which success can be assessed. Moreover, the original conversation cannot be replayed verbatim to a new agent: later user turns must be conditioned on the evaluated agent’s trajectory, while remaining anchored to the original user’s intent; otherwise, the interaction may drift, making final outcomes incomparable. Addressing this challenge therefore requires both verifiable task construction and controlled interaction replay. To address this challenge, we introduce SWE-Together, a benchmark for evaluating coding agents through multi-turn sessions reconstructed from real user–agent conversations. SWE-Together converts selected sessions into sandboxed tasks by retaining sessions with recoverable repository commits, clear user intents, and concrete outcomes such as submitted code changes. Each task includes the restored repository, the user’s initial request as the first-turn instruction, and task-specific artifacts derived from the recorded session, including decomposed user intents and trigger conditions for each feedback turn. During evaluation, an anchored, state-conditional LLM user simulator releases feedback only when its triggering conditions arise in the evaluated agent’s trajectory, preserving the original user’s objectives and intervention order while adapting feedback timing to each agent. This design helps attribute outcome differences to the agents rather than to simulator variation. Final repository states are scored using task-specific rubrics derived from repository inspection and original-session evidence. Beyond final correctness, SWE-Together reports User Correction, which quantifies the amount of corrective steering provided by the simulated user, and Intent Coverage, which measures whether the simulator consistently communicates the underlying user intents across agent runs. Our contributions are summarized as follows: • We introduce SWE-Together, a 109-task benchmark reconstructed from real multi-turn user interactive coding-agent sessions, together with a pipeline that converts recorded sessions into verifiable tasks. •We develop an anchored, state-conditional LLM user simulator that preserves the original user’s intent and intervention order while adapting feedback to each evaluated agent’s evolving trajectory. •We design a joint evaluation protocol that scores final repository correctness against frozen, implementation- agnostic rubrics and characterizes interaction trajectories through User Correction and Intent Coverage. 2 2 SWE-Together SWE-Together transforms recorded multi-turn coding-agent sessions into reproducible interactive software- engineering tasks. Our methodology has three components. First, the task construction pipeline filters and normalizes raw sessions, screens whether their coding objectives can be reproduced locally, and converts viable sessions into sandboxed repository-level tasks with pinned environments, executable checks, and task-specific user-simulation prompts. Second, the user simulator replays the original user’s intent in a trajectory-conditioned manner, intervening only when conditions derived from the recorded session are satisfied. Third, the evaluation framework separately measures the correctness of the agent’s final repository state and the user-simulator behavior during replay. Correctness is assessed by an agentic rubric judge using repository inspection and executable evidence, while user-simulator behavior is characterized through User Correction and Intent Coverage. Together, these components enable controlled evaluation of both an agent’s ability to complete coding tasks with evolving instructions and the corrective steering it elicits from the simulator. 2.1 Session-to-Task Construction We construct executable tasks from raw multi-turn coding sessions drawn from upstream Hugging Face datasets. Four upstream sources contribute to the evaluated suite: DataClaw (DataClaw Community, 2026), Pi-staging (Zechner, 2026), Hyperswitch (Archit11, 2026), and SWE-chat (Baumann et al., 2026), summarized in Table 1. Figure 2 Overview of the session-to-task construction pipeline. The first stage is fully deterministic, the second stage performs viability screening, and the third stage generates reproducible tasks in sandboxes. Table 1 Upstream sources used to construct SWE-Together, all drawn from real-world user sessions with coding agents. “Final tasks” denotes sessions that passed eligibility filtering and viability screening, and were successfully converted into executable tasks for the evaluated 109-task suite. SourceDataset descriptionRaw sessions Final tasks DataClaw (DataClaw Community, 2026)32 community-contributed datasets2,22829 Pi-staging (Zechner, 2026)29 datasets from the Pi staging pipeline 2,39723 Hyperswitch (Archit11, 2026)Traces from a production payments codebase 7849 SWE-chat (Baumann et al., 2026) Sessions spanning multiple agent harnesses 5,85148 Total11,260109 3 Figure 3 Task-construction workflow. The host orchestrator creates an isolated sandbox, supplies the normalized session and authoring prompt, and exports the resulting task package. Inside the sandbox, the task-generation agent screens the candidate, identifies repository verification commands, writes tests, constructs the user-simulation prompt, and audits the generated task. The final suite is a deliberately high-precision subset of the raw sessions: 109/11,260 sessions pass the filters and are converted into executable tasks, a conversion rate of 0.97%. Early filters favor public, sufficiently mature GitHub repositories with multi-turn user interaction and concrete code-changing work; later filters require recoverable changes and outcomes that can be evaluated locally. The construction pipeline has three stages. First, a deterministic rule-based collector filters raw upstream sessions and emits a normalized record for each candidate. Second, an LLM judge determines whether the coding work can be reconstructed as a reproducible and verifiable task. Third, a sandbox orchestrator runs task construction to produce a complete task directory. The overview is shown in Fig. 2. 2.1.1 Step 1: Deterministic Eligibility Filtering The first stage constructs an initial pool of candidate sessions using deterministic filtering. Given upstream coding-agent sessions, the collector removes traces that lack enough interaction, code modification, or repository context to support replay, and emits one normalized record for each remaining candidate. These criteria are deliberately rule-based and require no LLM calls. A candidate must contain multiple genuine user messages, include concrete agent actions or code edits, and provide enough repository signal to identify the working repository. The interaction and edit criteria ensure that each trace contains both multi-turn user feedback and concrete code-changing work, rather than only discussion or read-only exploration. Repository filters favor public, sufficiently mature projects so that downstream sandbox construction is feasible and less dependent on private or unstable codebases. We additionally filter out sessions in which the final change was primarily authored by the human user. This preserves trajectories where the coding agent performed substantive implementation work. 2.1.2 Step 2: Viability Screening The second stage screens whether the substantive coding work in a retained session can be reconstructed as a self-contained, locally executable task. The screener receives a compact session summary: repository metadata, message/tool/edit counts, a tool-use distribution, selected user messages, edited file paths, and truncated shell commands. It returns the session’s primary deliverable and whether that deliverable is reproducible in the local benchmark environment. We reject sessions whose primary deliverable depends on external state, such as pull-request management, issue triage, deployment operations, private credentials, or live-service state. Sessions with incidental external 4 actions, such as a final push or pull-request creation step, remain viable when the code edits are the core outcome. The viability screen does not evaluate correctness. Retained sessions are later reconstructed as original reference patches and evaluated through deterministic verifiers and final repository-state scoring. 2.1.3 Step 3: Task Construction The third stage converts each viable session into an executable benchmark package. For each candidate, a host orchestrator launches an isolated sandbox, provides the normalized session and prompt and harvests the generated task directory. Inside the sandbox, a task-generation agent performs a stricter repository-grounded screen, clones the target repository at a pinned commit, identifies local setup and test commands, and writes the task artifacts. This separation prevents task construction from depending on host-machine state such as cached credentials, local paths, installed toolchains, or an already-applied fix. The resulting package contains the original session record, the initial user instruction, a pinned execution environment, deterministic verifier artifacts, and a task-specific user-simulation prompt. Fig. 3 summarizes the workflow. 2.2 User Simulator Real software-engineering tasks often evolve through user interventions: clarification, correction, new require- ments, or requests to inspect external artifacts. We model this by replaying each reconstructed task as a multi-turn interaction between a coding agent and a user simulator. After each completed agent turn, the replay procedure summarizes the live trajectory and consults the simulator once. The simulator then makes one structured decision: send a user-facing message or remain silent. The simulator action space contains no-op, question, redirect, new-requirement, and check-external. The default action is no-op, which keeps the simulator silent and lets the agent continue without consuming one of the original follow-up messages. The other actions correspond to common user interventions: asking for clarification, redirecting an unproductive trajectory, introducing a follow-up requirement, or asking the agent to inspect an external artifact. Figure 4 Context consulted by the user simulator. Each replay checkpoint combines fixed session anchors, a summary of the evaluated agent’s latest turn, and simulator memory from earlier checkpoints. The simulator emits one structured action: message-bearing actions are injected as user turns, while no-op lets the agent continue. The simulator follows two principles: interventions are trajectory-conditioned rather than scheduled, and 5 anchored to the original session rather than generic. It conditions on recent agent activity, the agent’s latest response, elapsed time, observed repository changes, and its own previous decisions, which helps avoid repeated, premature, or irrelevant messages. At the same time, each task-specific simulator is conditioned on a session analysis reconstructed from the original user–agent interaction (Wu et al., 2026a). This analysis specifies the user’s objective, constraints, and intervention conditions grounded in the original follow-up messages. This avoids two failure modes: fixed replay can be mistimed when the evaluated agent follows a different path, while generic simulation can drift away from the original task. At evaluation time, these anchors define a state-conditioned decision policy: the simulator speaks when the live trajectory warrants feedback and otherwise returns no-op. 2.3 Evaluation Method We evaluate each replay along two dimensions: task correctness and user-simulator behavior. Task correctness measures whether the agent’s final repository state satisfies the coding request, including requirements introduced during the interaction. User-simulator behavior measures the feedback needed to produce that final state. Figure 5 Correctness plus interaction diagnostics. Final correctness scores the submitted repository state, while the interaction diagnostics characterize the replay that produced that state. Intent Coverage measures simulator fidelity to the original user’s scope; User Correction measures agent-elicited steering. 2.3.1 Task Correctness For task correctness, we score behavioral completeness rather than similarity to the original reference patch. This distinction matters because different agents may satisfy the same request with different helper functions, control flow, or integration points. The executable checks produced during task construction are useful but not sufficient for final scoring. Fixed checks can be misaligned with session intent in both directions: narrow checks may enforce incidental implementation details, while broad checks may require behavior that was never requested. This failure mode is not unique to our setting: OpenAI stopped reporting SWE-bench Verified scores after finding that many remaining tasks used tests that rejected functionally correct solutions or required underspecified behavior (OpenAI, 2026), and DeepSWE reports similar verifier-reliability failures in SWE-bench Pro, including false negatives for behaviorally valid patches (Huang et al., 2026). Execution-only scoring can also miss requirements introduced later in the interaction, fail to exercise behavior that depends on realistic repository 6 context, or reject valid alternative implementations. Purely static review has the opposite weakness: relevant evidence may appear in call sites, configuration, generated behavior, or targeted execution results rather than in a local code fragment alone. Our evaluator therefore combines deterministic verifiers with an agentic rubric judge. The deterministic verifier provides executable evidence for each task. Separately, the rubric judge scores task completeness in two phases: Phase 1 runs once per task to derive a weighted task rubric, and Phase 2 applies that same rubric to each candidate repository state. We separate the two phases so that the rubric is fixed offline, before and independently of any candidate, preventing the scoring criteria from being tailored to or biased by the particular solution under evaluation and thereby preserving cross-agent comparability. The Phase 1 rubric may consult the original reference patch to identify the behavior of the recorded solution, but the resulting goals are behavioral and are reused unchanged across agents. For each goal, Phase 2 returns a binarymetdecision with supporting evidence. The final task-correctness score is derived mechanically from those decisions: score = round X g w g I[g met], 2 ! , wheregranges over the task-rubric goals and the weightsw g are normalized to sum to one. Weighting provides partial credit, while reusing the same rubric across agents ensures that all attempts on the same task are evaluated against identical criteria. A host-side validator checks goal coverage, weight normalization, and consistency between the reported decisions and the resulting score. 2.3.2 User Simulation Behavior Correctness alone does not capture the interaction cost of reaching a solution. Two agents may end with similar repository states while requiring very different amounts of simulator feedback. We therefore report user-simulator behavior separately from task correctness. We use two diagnostics. Intent Coverage audits simulator fidelity : whether replayed simulator messages preserve the original user’s intents and remain within scope. User Correction measures the corrective steering elicited by the evaluated agent, counting explicit corrections and lower-weight nudges. This separation keeps simulator fidelity distinct from the agent-facing interaction signal used in the main results. Intent Coverage. Intent Coverage measures how faithfully the simulator preserves the original user intents of the original session. We compute it in two passes. First, once per task, we decompose the original session trajectories into a set of atomic original-session intents, each representing a distinct request made by the original user. Second, for each replayed trial, we match the simulator’s messages against these intents and record both how well each intent is covered and whether each simulator message remains within the original scope. From this matching, we derive weighted intent recallI recall , which measures how completely the simulator re-expresses the original requests, and scope precisionI precision , which measures how consistently its guidance remains within the original user’s scope. We combine them as follows. IntentCoverage = round(0.70I recall + 0.30I precision , 2). We weight recall more heavily because omitting an original intent can directly change the task presented to the evaluated agent. Measuring Intent Coverage separately helps distinguish agent capability from simulator- induced variation in how faithfully the original session is reconstructed. The metric therefore serves as a diagnostic of simulator fidelity and cross-agent comparability: scores should remain broadly stable across model cohorts, while a substantially lower score may indicate that the simulator omitted requirements, drifted beyond the original scope, or struggled to convey the remaining intents because the agent’s responses diverted the interaction from the original trajectory. User Correction. We define “User Correction” to test the hypothesis that a stronger model requires less intervention to reach the same level of performance. To identify corrective interventions, we apply a multi-label 7 tagger to every simulator message. Multi-labeling is necessary because a message may perform several communicative acts simultaneously, such as introducing a new request while correcting an earlier mistake. The taxonomy separates three layers of user behavior. The corrective layer containscorrection, which explicitly asserts that the agent’s work is incorrect, incomplete, or off track, andnudge, which only implies doubt or encourages reconsideration without asserting a defect. The non-corrective ask layer containsrequest, question, andverification, representing new requirements, genuine information-seeking questions, and neutral checks, respectively. Finally,workflow,approval, andcontextcapture mechanical instructions, confirmations, and background information. These latter tags do not contribute to User Correction. For each trial, we compute UserCorrection = N correction + 0.2N nudge . Explicit corrections receive full weight because they directly assert that the agent failed or misunderstood the request. Nudges receive a smaller weight because they capture softer corrective pressure, such as skeptical questions or diagnostic information supplied as a hint. User Correction is first averaged across replicates within each task and then across tasks, ensuring that each task contributes equally to the model-level result. Variation in User Correction across coding agents is primarily agent-dependent and serves as a companion measure to task correctness. Under our hypothesis that stronger models require less corrective intervention to reach comparable performance, User Correction should be negatively correlated with model capability. 3 Experiments and Results 3.1 Main Result Experimental setup. We evaluate seven frontier models on the full SWE-Together benchmark of 109 tasks using a common agent harness,opencode, withk= 2 replicates per task. Each final patch receives a score in [0,1] from the agentic judge against the task’s frozen rubric from Section 2.3.1. We use this judge score, rather than raw test execution, as the primary correctness signal. Evaluation metrics. Let j t,r ∈ [0, 1] denote the judge score for replicate r of task t, and let s t,r = I[j t,r ≥ τ ], ̄ j t = 1 k k X r=1 j t,r , ̄s t = 1 k k X r=1 s t,r , We report four equally task-weighted correctness metrics: pass@1 = 1 N N X t=1 ̄s t ,SSR = 1 N N X t=1 I[ ̄ j t ≥ τ ], pass 2 = 1 N N X t=1 2 Y r=1 s t,r , MeanJudge = 1 N N X t=1 ̄ j t . Here,N= 109. The three threshold-based metrics apply the same success threshold,τ= 0.85, but aggregate thek= 2 runs differently.pass@1is the marginal per-run success rate and estimates the probability that a single fresh run solves the task. The stable solve rate (SSR) first averages the continuous judge scores within each task and then applies the threshold, measuring whether the model is reliable on average while tolerating an occasional weak or near-miss run. In contrast, pass 2 measures joint success and credits a task only when both replicates exceed the threshold, providing the strictest measure of consistency and penalizing run-to-run variance most strongly. Consequently,pass 2 ≤ pass@1.MeanJudgecomplements these binary metrics by reporting the average continuous judge score without thresholding. We additionally report User Correction as the interaction diagnostic most directly tied to agent behavior. It measures the corrective steering elicited by an agent and is first averaged over replicates within each task and then across tasks. We report Intent Coverage separately as a simulator-fidelity diagnostic rather than as a model-ranking metric. We also report output-plus-reasoning tokens and wall-clock time per task. 8 Table 2 Results on the full 109-task SWE-Together benchmark using theopencodeharness andk= 2 replicates. Models are ranked by mean judge score. Oracle denotes the reference-patch baseline. U-Corr measures corrective steering (↓is better), Bold values indicate the best evaluated agent on the correctness, U-Corr, and efficiency metrics. Rank Modelpass@1↑ SSR↑ pass 2 ↑ Mean judge↑ U-Corr↓ Tok./task Min./task ⋆ Reference∼ 78% ∼ 78% ∼ 78%0.90— 1 Claude Opus 4.8 63% 59% 52% 0.801 1.3874.0k23.3 2 GPT-5.558%55% 48%0.7631.59 29.9k 10.7 3 Claude Opus 4.658%58% 46%0.7551.5942.0k23.2 4 GLM-5.255%48% 42%0.7351.5341.7k24.5 5 GLM-5.152%49% 35%0.7291.5441.6k38.8 6 DeepSeek-V4-Pro 48%38% 29%0.6791.7649.8k21.0 7 MiniMax-2.740%34% 26%0.6302.1743.4k36.2 Overall performance. Table 2 shows a broadly consistent ordering across the four correctness metrics. Claude Opus 4.8 achieves the strongest overall performance, leading in pass@1 (63%), SSR (59%),pass 2 (52%), and mean judge score (0.801). It also requires the least corrective steering, with a mean User Correction of 1.38. Nevertheless, its pass@1 remains approximately 15 percentage points below the original reference-patch accuracy, indicating remaining headroom. GPT-5.5 ranks second by mean judge score (0.763), with pass@1 of 58%, SSR of 55%, andpass 2 of 48%; it roughly ties Claude Opus 4.6, with higherpass 2 but lower SSR. Claude Opus 4.6 follows closely in third (mean judge 0.755). GLM-5.2 and GLM-5.1 form the next tier: their SSR values are nearly identical (48% and 49%), but GLM-5.2 achieves higher pass@1 (55% versus 52%) and a substantially higherpass 2 (42% versus 35%), indicating greater performance stability across replicates. DeepSeek-V4-Pro and MiniMax-2.7 rank last, with MiniMax-2.7 obtaining the lowest values on all four correctness metrics and requiring the most corrective steering. Figure 6 Capability (pass@1 left, stable solve rate right) versus User Correction per trial across the seven models, over three task subsets (all 109; the active subset receiving≥1 correction; the hard subset with mean judge<0.85). Capability and correction are strongly inversely related (Pearson−0.92 and−0.84 for pass@1 and stable solve rate, respectively): stronger models need less corrective pushback. Stronger models need less steering. User Correction is the one interaction signal that tracks capability: across the seven models it is strongly inversely correlated with pass@1 (Pearson−0.92), stable solve rate (−0.84), and mean judge score (−0.93). As Figure 6 shows, opus-4.8 reaches the top of the board with the fewest corrective messages per trial (1.38), whereas minimax-2.7—the weakest cohort—requires the most (2.17); the relationship holds on the harder task subsets as well. This operationalises the intuition that a stronger agent 9 Figure 7 Stable solve rate versus cost for the seven opencode cohorts: mean per-task wall-clock minutes (left) and output+reasoning tokens (right). Up-and-left is better (higher capability at lower cost). needs less human pushback to reach the same outcome. Efficiency. Figure 7 plots stable solve rate against per-task wall-clock time and output tokens. The two cost axes are only weakly coupled to capability and to each other: gpt-5.5 is the most efficient cohort on both (29.9k output tokens and 10.7 min per task) while also ranking second on capability, whereas opus-4.8 reaches the top of the board at the highest token cost (74.0k) but moderate wall-clock (23.3 min). Regarding wall-clock latency, GLM-5.1 and MiniMax are the slowest cohorts, at 38.8 and 36.2 minutes, respectively. However, these latency differences may be affected by the serving location or infrastructure of the inference endpoint. Why the reference scores below 100%. The reference baseline attains a mean judge score of 0.90 and a pass rate of≈78% (73/93) at theτ= 0.85 threshold, evaluated on the 93 of 109 tasks that have an extractable reference patch, the remaining 16 have no canonical code diff. Of the 93, 57 (61%) score a perfect 1.0 and 73 (78%) pass; only 20 fall below pass threshold, and their shortfall has three identifiable sources rather than task unresolvability. First, roughly 35% of the unsatisfied goals are process requirements that the rubric inherits from the recorded session—diagnosing the root cause before editing, answering a follow-up question, or explaining the change to the user—which a final patch cannot express and which therefore penalize every submission, reference and agent alike. Second, a few reference patches incompletely capture work that originally spanned multiple commits or sessions, an extraction-noise artifact of the data pipeline. Third, a small residual reflects genuinely imperfect human solutions—e.g., a session that fixes several code paths but leaves the headline bug, which the judge correctly identifies. Since the same frozen rubric is applied to the reference patch and to every evaluated agent, these shared factors largely affect all submissions alike, and the reference row is best read as a like-for-like reference point under our scoring criteria rather than a strict ceiling on resolvability. 3.2 User Simulator Consistency Analysis Intent Coverage remains broadly stable across coding agents. Six of the seven model cohorts obtain overall scores between 0.70 and 0.72, with recall ranging from 0.72 to 0.74 and precision from 0.66 to 0.72. This narrow variation indicates that the simulator generally preserves the original users’ requests and remains within their intended scope, despite differences in the behavior of the underlying coding agents. The resulting evaluations are therefore broadly comparable across model cohorts and are unlikely to be driven by systematic variation in simulator fidelity. GPT-5.5 sits marginally below this band, with an overall score of 0.68 (recall 0.71, precision 0.65). The gap is small and consistent with its strong, efficient task performance: a more capable agent more often resolves or 10 reshapes the interaction on its own, which can leave the simulator fewer natural openings to re-express every original follow-up, rather than indicating reduced simulator fidelity for this cohort. All seven cohorts exhibit consistent user-simulator behavior within a tight range, supporting cross-agent comparability. 3.3 User Simulator Quality Study Protocol. We evaluate whether human annotators can distinguish simulated users from real users. Through a web interface, four annotators make forced two-alternative-choice judgments over paired trajectories, selecting the trajectory they believe was produced by the real user. We use the 52 tasks shared by DeepSeek-V4-Pro, MiniMax-2.7, and Claude Opus 4.6, yielding 156 trajectory pairs and 312 judgments. We report the Turing pass rate, defined as the fraction of judgments in which the simulated trajectory is selected as real. A pass rate of 50% corresponds to chance-level discrimination. Findings. Across all trajectories, the simulator achieves a Turing pass rate of 46% (95% CI [40.5,51.6]%). Because the confidence interval includes 50%, annotators exhibit no statistically reliable ability to distinguish simulated users from real users. These results indicate that, under our evaluation protocol, simulated and real user trajectories are generally indistinguishable to human evaluators. 4 Related Work We distinguish two forms of multi-turn evaluation for coding agents. Agent-environment multi-turn refers to episodes in which an agent iteratively inspects files, runs commands, edits code, and invokes tests while solving a fixed user request. Interactive replay refers to episodes in which the user-facing instruction evolves through feedback,corrections, clarifications, or new requirements. This distinction organizes prior work into three groups: (i) software-engineering benchmarks with realistic agent–environment interaction but fixed user requests; (i) interactive coding benchmarks with simulated users; and (i) real coding-session datasets without task-grounded replay. As summarized in Table 3, SWE-Together combines repository-level agent-environment interaction, interactive user-correction replay, and provenance from real recorded user-agent coding sessions. Table 3 Positioning of SWE-Together against representative coding-agent benchmarks.✓indicates yes,✗indicates no, ▲indicates partial support, and♦indicates mixed or heterogeneous coverage. Agent-env. multi-turn means the agent can iteratively interact with tools, files, terminal, or tests. Interactive replay means evaluation includes sequential user-facing feedback, correction, or clarification turns. Benchmark Repo- level Agent-env. multi-turn Interactive replay Real task source Real user session SWE-bench family (Jimenez et al., 2024; Pan et al., 2024; Liang et al., 2026)✓✗✓✗ Terminal-Bench (Merrill et al., 2026)✗✓✗ MINT / ConvCodeWorld (Wang et al., 2024; Han et al., 2025)✗▲✓✗ CodeAssistBench (Kim et al., 2025)✓✗ RECODE-H / FronTalk (Miao et al., 2025; Wu et al., 2026b)▲✓▲✗ BigCodeArena / CodeChat (Zhuo et al., 2025a; Zhong et al., 2025)♦▲✓ SWE-chat (Baumann et al., 2026)✓✗✓ SWE-Together (ours)✓ Agent-environment multi-turn benchmarks. SWE-bench (Jimenez et al., 2024) and its extensions (Yang et al., 2025; Zhang et al., 2025b; Liang et al., 2026; Pan et al., 2024) are the canonical repository-level software-engineering benchmarks. They ground tasks in real codebases and issues, and agents may take many environment actions before producing a final patch. Terminal-Bench (Merrill et al., 2026) similarly evaluates agents in interactive terminal environments. These benchmarks are multi-turn in the agent–environment sense, but the user request is fixed: the benchmark does not evaluate whether an agent can recover from user corrections or adapt to requirements revealed after intermediate attempts. SWE-Together keeps the repository-level, tool-using setting, but adds an interactive user-correction loop. Interactive coding benchmarks with simulated users. A complementary line of work introduces simulated user feedback into coding evaluation. TiCoder (Lahiri et al., 2022) studies iterative code generation with simulated 11 user queries on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). MINT (Wang et al., 2024) and ConvCodeWorld (Han et al., 2025) extend LLM-driven feedback to standard code-generation tasks. And Pan et al. (2025) converts static benchmarks into interactive evaluations by revealing hidden information through a simulated user. More recent work moves interactive evaluation closer to software engineering: CodeAssistBench (CAB) (Kim et al., 2025) uses GitHub-issue tasks and a satisfaction-driven simulated user. SWEET-RL/ColBench (Zhou et al., 2025), RECODE-H Miao et al. (2025), and FronTalk Wu et al. (2026b) evaluate collaborative refinement in backend, research-code, and front-end settings, and CollabLLM (Wu et al., 2025) trains assistants to act as active collaborators by optimizing multi-turn-aware rewards estimated through a simulated user. These benchmarks show that user feedback is an important evaluation axis, but their interaction loops are generally synthesized from static tasks, issue artifacts, curated feedback policies, or benchmark-generated scenarios (Xu et al., 2025; Zhou et al., 2026; Chen et al., 2026). In contrast, SWE-Together derives tasks from recorded real human–agent coding sessions and grounds the simulated correction loop in the corresponding original session. This design choice is supported by Suh et al. (2026), who quantify the utility of user simulators and find that simulators grounded in real human behavior yield substantially better downstream collaborative assistants than role-playing prompts. Real coding-session data. Recent datasets demonstrate that real user–agent coding interactions can be collected at scale. SWE-chat (Baumann et al., 2026) characterizes real-world coding-agent sessions, while BigCodeArena Zhuo et al. (2025a) and CodeChat Zhong et al. (2025) collect code-centric conversations for preference modeling, analysis, and assistant evaluation. Related benchmarks such as Saving SWE-Bench Garg et al. (2025) and EDIT-Bench Chi et al. (2025) further emphasize realistic user phrasing and in-the-wild edit instructions. However, these works are primarily descriptive, preference-oriented, or single-request evaluation sets: they do not pair each real session with a replayable repository state, deterministic verifier, and live user-correction loop for evaluating new agents. 5 Limitations and Conclusion Limitation. The user simulator cannot interrupt the coding agent during its turns, cannot directly edit files, and relies solely on textual trajectories and tool outputs rather than visual information from the interface. The current design works best when user goals and constraints are clearly defined and focuses on tasks with measurable outcomes, such as submitted patches. Consequently, it provides limited coverage of ambiguous, open-ended tasks and qualitative user behaviors that are difficult to quantify. Conclusion. This work introduces SWE-Together, a benchmark that transforms real multi-turn coding sessions into reproducible software-engineering tasks and evaluates both final task correctness and the user guidance required during interaction. Across 109 tasks and seven frontier models, the results show substantial differences in capability and reliability. User Correction is strongly negatively correlated with performance, supporting the hypothesis that more capable coding agents require less user intervention. In addition, the simulator maintains broadly consistent intent coverage across models and produces trajectories that human annotators cannot reliably distinguish from real-user interactions. Together, these findings demonstrate the importance of evaluating coding agents not only by whether they complete a task, but also by how much corrective steering they elicit. By incorporating realistic multi-turn interactions and user-centered diagnostics, SWE-Together aims to evaluate coding agents in a way that more closely reflects the actual user experience. Acknowledgments We thank Zhiqing Sun and Rui Hou for insightful discussions and feedback. SWE-Together is built entirely on coding-agent sessions that were collected, curated, and openly released by the opensource community. We are deeply grateful to the trajectory providers whose donated data made this benchmark possible: the DataClaw community (DataClaw Community, 2026), the contributors behind the Pi staging pipeline (Zechner, 2026), the maintainers of the Hyperswitch trace collection (Archit11, 2026), and the SWE-chat work (Baumann et al., 2026). 12 References Archit11. claude_traces_hs. https://huggingface.co/datasets/archit11/claude_traces_hs, 2026. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, and Sanmi Koyejo. Swe-chat: Coding agent interactions from real users in the wild, 2026. https://arxiv.org/abs/2604.20779. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Zhaorun Chen, Zhuokai Zhao, Kai Zhang, Bo Liu, Qi Qi, Yifan Wu, Tarun Kalluri, Sara Cao, Yuanhao Xiong, Haibo Tong, Huaxiu Yao, Hengduo Li, Jiacheng Zhu, Xian Li, Dawn Song, Bo Li, Jason Weston, and Dat Huynh. DreamGym: Scaling agent learning via experience synthesis, 2026. ICLR 2026. Wayne Chi, Valerie Chen, Ryan Shar, Aditya Mittal, Jenny Liang, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Ion Stoica, Graham Neubig, Ameet Talwalkar, and Chris Donahue. EDIT-Bench: Evaluating LLM abilities to perform real-world instructed code edits, 2025. DataClaw Community. DataClaw: A donated corpus of real coding-agent sessions.https://huggingface.co/datasets/ alexshengzhili/dataclaw-harbor-candidates, 2026. Spandan Garg, Benjamin Steenhoek, and Yufan Huang. Saving SWE-Bench: A benchmark mutation approach for realistic agent evaluation, 2025. Hojae Han, Seung won Hwang, Rajhans Samdani, and Yuxiong He. ConvCodeWorld: Benchmarking conversational code generation in reproducible feedback environments. In International Conference on Learning Representations (ICLR), 2025. Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. Datacurve blog, May 2026. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. Myeongsoo Kim, Shweta Garg, Baishakhi Ray, Varun Kumar, and Anoop Deoras. CodeAssistBench (CAB): Dataset and benchmarking for multi-turn chat-based code assistance, 2025. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. LLMs get lost in multi-turn conversation, 2025. Shuvendu K. Lahiri, Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, Madanlal Musuvathi, Piali Choudhury, Curtis von Veh, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. Interactive code generation via test-driven user-intent formalization (TiCoder), 2022. Jiarong Liang, Zhiheng Lyu, Zijie Liu, Xiangchao Chen, Ping Nie, Kai Zou, and Wenhu Chen. SWE-Next: Scalable real-world software engineering tasks for agents, 2026. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, Fangxin Wang, Yifan Li, Wooseong Yang, Bowei He, Xinni Zhang, Dianzhi Yu, Hanchen Yang, Hoang H. Nguyen, Yue Zhou, Jie Yang, Jizhou Guo, Wenzhe Fan, Chin-Yuan Yeh, Panpan Meng, Liancheng Fang, Jinhu Qi, Wei-Chieh Huang, Zhengyao Gu, Yuwei Han, Langzhou He, Yuyao Yang, Yinghui Li, Hai-Tao Zheng, Xue Liu, Irwin King, and Philip S. Yu. RECODE-H: A benchmark for research code development with interactive human feedback, 2025. Under review at ICLR 2026. OpenAI. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/, 2024. OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities.https://openai.com/index/ why-we-no-longer-evaluate-swe-bench-verified/, February 2026. 13 Jane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, and Valerie Chen. When benchmarks talk: Re-evaluating code LLMs with interactive feedback, 2025. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym, 2024. Scale AI. SWE-Bench Pro: A new frontier for SWE-Agents. https://scale.com/research/swe_bench_pro, 2025. Joseph Suh, Ayush Raj, Minwoo Kang, and Serina Chang. Quantifying the utility of user simulators for building collaborative LLM assistants, 2026. https://arxiv.org/abs/2605.09808. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In International Conference on Learning Representations (ICLR), 2024. Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. CollabLLM: From passive responders to active collaborators. In International Conference on Machine Learning (ICML), 2025. Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, and James Zou. HumanLM: Simulating users with state alignment beats response imitation, 2026a. Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, and Yeming Wen. FronTalk: Benchmarking front-end development as conversational code generation with multi-modal feedback, 2026b. Under review at ICLR 2026. Zengzhuang Xu, Bingguang Hao, Zechuan Wang, Yuntao Wen, Xinyi Xu, Yang Liu, Long Chen, Dong Wang, Maolin Wang, Tong Zhao, Yicheng Chen, Cunyin Peng, Jinjie Gu, Leilei Gan, Xiangyu Zhao, Chenyi Zhuang, and Shi Gu. FunReason-MT technical report: Advanced data synthesis solution for real-world multi-turn tool-use, 2025. John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025. Mario Zechner. pi-share-hf: Collect, review, and upload redacted pi session files to a hugging face dataset.https: //github.com/badlogic/pi-share-hf, 2026. Binquan Zhang, Li Zhang, Haoyuan Zhang, Fang Liu, Song Wang, Bo Shen, An Fu, and Lin Shi. Decoding human-LLM collaboration in coding: An empirical study of multi-turn conversations in the wild, 2025a. Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Jianfeng Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Yudong Zhang. SWE-bench goes live! In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025b. Suzhen Zhong, Ying Zou, and Bram Adams. Developer-LLM conversations: An empirical study of interactions and generated code quality, 2025. Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks, 2025. Yuhang Zhou, Lizhu Zhang, Yifan Wu, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao, and Hong Yan. Synthetic sandbox for training machine learning engineering agents, 2026. Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, and Leandro von Werra. BigCodeArena: Unveiling more reliable human preferences in code generation via execution, 2025a. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, and Indraneil Paul. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. In International Conference on Learning Representations (ICLR), 2025b. 14