Paper deep dive
Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?
Xiaonan Xu, Wenjing Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/3/2026, 2:16:07 AM
Summary
This paper introduces BSG-VA (buggy-state/candidate-state/gold-fix validation analysis), a method to evaluate the evidential quality of validation events in Large Language Model (LLM) repair agents. By replaying agent-generated tests on the original buggy code (B), the candidate patch (S), and the developer's gold fix (G), the authors classify validation events into roles such as 'gold-aligned bug-discriminating' or 'regression-only'. In a study of 3,730 events across 110 tasks, 46% of positive events lacked bug-discriminating information. The study further demonstrates that providing agents with 'bug-contrast feedback' (the B-replay outcome) significantly reduces evidence-inadequate closure and increases bug-discriminating evidence, particularly when using the gpt-5.6-sol model.
Entities (8)
Relation Signals (6)
BSG-VA → classifies → validation events
confidence 95% · BSG-VA captures each validation command... and replays the command... The captured outcome and the replay results assign every event an evidence role
Bug-Contrast Feedback → increases → bug-discriminating evidence
confidence 92% · raises bug-discriminating evidence by 7.4 points
Bug-Contrast Feedback → reduces → evidence-inadequate closure
confidence 92% · Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder
BSG-VA → uses → GPT 5.6 Sol
confidence 90% · We apply BSG-VA at scale in a controlled experiment using gpt-5.6-sol
BSG-VA → appliesto → SWE-bench Verified
confidence 88% · We apply BSG-VA at scale in a controlled experiment using gpt-5.6-sol on 110 tasks drawn equally from SWE-bench Verified
GPT 5.6 Sol → benefitsfrom → Bug-Contrast Feedback
confidence 85% · the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bug-discriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentage-point smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. Keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment.
Tags
Links
- Source: https://arxiv.org/abs/2607.28871v1
- Canonical: https://arxiv.org/abs/2607.28871v1
Trouble viewing inline? Open PDF directly →
Full Text
75,311 characters extracted from source content.
Expand or collapse full text
[1] [1] of Computing, Institute of Technology, , , 30332, 2] of Computer Science, of Colorado Boulder, , , 80309, Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug? xiaonanxu5@gmail.com * [ Abstract When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bug-discriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p=0.0029p=0.0029) and raises bug-discriminating evidence by 7.4 points (p=0.011p=0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentage-point smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment 1 Introduction Repair agents on SWE-bench-style benchmarks now resolve a substantial and growing share of real-world GitHub issues. A typical successful trajectory involves dozens of tool calls: reading files, editing code, running tests, observing outputs, and iterating. Among these actions, running a test and seeing it pass occupies a privileged position: it is the closest thing the agent has to empirical confirmation that its changes work. Benchmark evaluation reinforces the reading, since the final verdict is whether the submitted patch passes a held-out test suite. But a passing test can mean different things. Consider an agent assigned a defect in a date-formatting routine. The agent modifies the routine, then writes a test that imports the module and asserts that the output is a string. The test passes. It would also have passed on the original buggy code, because the bug was not about return types but about locale handling. The agent has confirmed that its patch does not break the import or change the output type. It has learned nothing about whether the locale bug is fixed. If this is the only positive evidence the agent collects before submitting, the submission rests on validation that is real but irrelevant to the assigned defect. How common is this pattern? No existing method answers the question, because mid-trajectory validation events have not been systematically evaluated for evidential content. Prior work on agent-generated tests has treated them as standalone artefacts: the SWT-Bench family evaluates dedicated pipelines that take an issue description and produce a test designed to fail on buggy code and pass on the fix [29, 1, 19]. These are dedicated systems, separate from the repair process. Trajectory analyses of frontier models [10] have characterised the frequency and syntactic form of in-trajectory test events, showing that agents write tests often but not evaluating what those tests prove about the assigned bug. The gap between “the agent ran a test” and “the test targeted the defect” remains unmeasured. We introduce BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) to close this gap. The method intercepts every validation command the agent executes, snapshots the working tree at execution time, and extracts a test-only patch that separates the validation logic from any concurrent production edits. It then replays the same command on three code states: the original buggy version (B), the captured candidate (S), and the developer gold fix (G). The captured outcome and the replay results place each event in one of seven evidence roles. An event where the test fails on B, passes on S, and passes on G is gold-aligned bug-discriminating: it targets a property specific to the defect, in a way that would also accept the developer’s own fix. An event that passes on B as well as S is at best regression-only: it confirms the patch has not introduced a new failure, but says nothing about the bug. Between these poles sit candidate-specific events (B-fail, S-pass, G-fail), misleading events, and diagnostic negatives; two further categories set aside events whose outcomes are unstable or not evaluable. The taxonomy is exhaustive, mutually exclusive, and defined entirely by recorded, observable outcomes. We apply BSG-VA at scale in a controlled experiment using gpt-5.6-sol (hereafter sol) on 110 tasks drawn equally from SWE-bench Verified and SWE-rebench. The measurement covers 3,730 retained post-edit validation events across 643 rollouts (events before the agent’s first production edit are excluded, as discussed in Section 3). Of 2,548 positive comparable events, 46.0% are regression-only or misleading. At the rollout level, 23.8% of baseline runs exhibit what we call evidence-inadequate closure: the agent submitted a patch, collected positive validation results, and none of those results discriminated the reported bug from a regression-free alternative. The measurement enables a direct intervention. Because BSG-VA can evaluate an event’s B-replay outcome in real time (median 11.0 seconds per rollout), we test whether feeding that outcome back to the agent changes the closure pattern. In the three-arm experiment, bug-contrast feedback (BCF) takes the agent’s passing validation event, replays it on B, and reports the result in a system message. If the test also passes on B, the agent learns its evidence is non-discriminating. The comparison arms are a structure-matched generic reminder that prompts the agent to reconsider its evidence without providing B-replay information, and a baseline arm into which no messages are injected. BCF reduces evidence-inadequate closure by 7.8 percentage points relative to the reminder (p=0.0029p=0.0029, 95% CI [−-12.9, −-2.7]) and raises bug-discriminating evidence by 7.4 points (p=0.011p=0.011). Repair success shows no detectable change. The active-control decomposition shows the generic reminder alone accounts for roughly a third of the total BCF-versus-baseline improvement, a share that comes from prompting the agent to attend to evidence quality rather than from the B-replay content itself. Under a plan-execute-verify scaffold, baseline evidence-inadequate closure falls to 8.3%, a floor effect that leaves little room for either component. On gpt-5.6-terra (hereafter terra) in the unconstrained loop, the generic reminder produces a 7.5-point reduction on its own, comparable to the full BCF effect on sol, while the B-replay content adds nothing detectable. The prompting effect recurs on the second model; the B-replay content adds measurable value only with gpt-5.6-sol in the unconstrained loop. The study is organised around four objectives. RQ1 (descriptive): what share of positive validation evidence produced mid-trajectory discriminates the assigned bug? RQ2 (associational): does evidence quality carry predictive signal for official resolution (the benchmark’s own pass/fail verdict) beyond task-level covariates? The confirmatory objective tests whether BCF reduces evidence-inadequate closure relative to a structure-matched generic reminder, with bug-discriminating evidence as a gated secondary outcome. Two exploratory replications [38] then ask how these patterns change across a different scaffold and a second model. This paper contributes an event-level measurement method and evidence-role taxonomy that assign each validation event an evidence role with directly checkable content; a quantified prevalence estimate showing that nearly half of the positive comparable validation evidence in a 3,730-event study does not discriminate the assigned bug; and a fully crossed three-arm experiment that separates the prompting effect of a generic reminder, which recurs on a second model, from the additional increment of B-replay content, which does not. 2 Background and related work 2.1 Repository-level program repair with LLM agents Automated program repair (APR) has evolved from template-based and search-based patch generation [23, 42] toward neural and large-language-model approaches [46, 16]. The current generation combines a large language model with an agentic tool-use loop: the model reads an issue description, navigates the repository, edits files, and runs commands, iterating until it produces a candidate patch or exhausts a budget. SWE-agent [49] established the architecture; Agentless [45] showed that a fixed pipeline of localisation, repair, and patch validation without persistent agent state can be competitive; CodeR [8], Moatless [32], and AutoCodeRover [50] introduced variations on retrieval, planning, and tool design. SWE-bench [17] and its curated subset SWE-bench Verified [31] provide the standard evaluation protocol: the agent receives an issue, works within a repository checkout, and submits a patch that is evaluated against a held-out developer test suite. Across all these architectures, agents routinely execute validation commands during the trajectory. The evaluation protocol, however, examines only the final submitted patch. The mid-trajectory validation activity is logged but otherwise ignored by the benchmark harness. 2.2 Deliberately generated tests for patch validation A separate body of work generates tests deliberately, as a post-hoc step to validate candidate patches. SWT-Bench [29] defines the fail-to-pass criterion: a generated test must fail on the original buggy code and pass on the developer fix. Filtering agent patches through such tests raises precision substantially [29]. The protocol has become a competitive benchmark in its own right, with systems including Otter, AssertFlip, Issue2Test, Echo, and EvoOtter optimising fail-to-pass success rates [1, 19, 30, 12, 2], and with repair agents tasked to co-generate fixes and reproduction tests inside the repair loop [11]. LIBRO [18] generates issue-reproducing tests from bug reports without access to the developer fix at generation time. A valid fail-to-pass test is still not a complete oracle: it can cover a single manifestation of the issue and steer repair toward a partial patch [13]. Assessment of generated suites has its own tooling, from mutation-based measures of discriminative power [40] to lifecycle evidence infrastructure that tracks execution, coverage, and flakiness [22]. The distinction between this body of work and the present study is one of context and agency. Deliberate test generation is an external pipeline step, executed after the agent’s trajectory is complete, with the explicit goal of producing a discriminating test. BSG-VA examines the agent’s own spontaneous validation behaviour during the trajectory, where the agent decides what to test, when, and how, without any external instruction to produce a fail-to-pass test. The two perspectives are complementary: the former asks whether discriminating tests can be generated on demand, while the latter asks what the tests that agents already generate actually prove. 2.3 Trajectory analysis and test-value assessment Z. Chen . [10] analyse six models on SWE-bench Verified and report that test-writing frequency is similar between successful and failed trajectories; that most validation commands are print-statement explorations rather than assertion-bearing tests; and that prompting models to write more or fewer tests produces no detectable change in repair outcomes. The finding that test-writing is abundant but not predictive of success is consistent with the hypothesis that much of the validation activity lacks discriminating power, but the study does not measure discriminating power directly. Lin . [24] analyse the cost-effectiveness of code execution in LLM-based repair at scale and report that failed commercial-agent runs often pass their own self-validation while failing official evaluation. That mismatch is the starting point here: where their analysis works at the level of run frequency, cost, and coarse agreement between the agent’s own test results and the official evaluation, BSG-VA reconstructs the exact candidate state of each self-selected command and replays it to establish which passing checks could not have detected the defect. Broader diagnostics work instruments trajectories at a structural level: lucky-pass detection and production trajectory review [34, 33], trajectory-structure and verification-skip diagnostics [37, 5], traceability analyses of reproduction and regression testing across repair agents [7], and interactive trajectory visualisation [35]. Cross-framework comparison warns that the same aggregate trajectory signal can carry different semantics in different frameworks [25]. BSG-VA is narrower: it attaches an executable, issue-specific meaning to individual validation events rather than characterising trajectory structure. Overfitting to the validation signal has a long history in APR: generated patches can overfit weak test suites [39], and test generation was proposed early to expose overfitted patches [47]. Konstantinou . [20] approach the agent-era version of the problem, showing that tests generated after exposure to an incorrect implementation detect fewer faults than independently generated tests, a mechanism by which non-discriminating suites arise. Controlled evidence that incorrect code context reduces generated-test fault detection supports the same account [15]. Hora Robbes [14] document a related fragility in which agent commits mock more heavily than non-agent commits, risking tests that compile and run but exercise little of the relevant behaviour. Y. Ma . [26] show, in a controlled code-as-spec study, that agents can satisfy every visible check without making the requested artefact load-bearing. These findings identify specific mechanisms by which validation evidence can be non-discriminating, but none provides a systematic taxonomy of evidence roles or a prevalence estimate across a controlled population. BSG-VA extends this line by operationalising the measurement. Rather than classifying tests by static syntactic signals, as in large-scale oracle-signal audits of agent-authored test code [4], or by comparing agent tests with developer tests as monolithic suites, BSG-VA captures each event at its exact state, isolates the test-only component, and replays it on the buggy, candidate, and gold states. The captured outcome and the replay pattern assign each individual event an evidence role with directly checkable content, yielding an event-level taxonomy and prevalence estimate. 2.4 Feedback and steering in agentic repair Existing work on steering agent behaviour during repair has focused on tool design, retrieval augmentation, and planning structure [49, 50, 32]. At a conceptual level, automated verification signals have been framed as imperfect proxies constrained by scalability, faithfulness, and robustness [41], and process-discipline benchmarks score whether agents verify and recover at all [27]. Closest in spirit to an evidence-quality intervention, EviACT gates repair actions behind evidence-driven retrieval, compilation, and target-test guardrails [28]. Few studies have investigated online feedback about the quality of the agent’s own validation evidence as a steering signal. The approach tested here, in which the agent receives the outcome of replaying its own test on the original buggy state, is related in spirit to self-debugging techniques [9] that feed execution output back into the model, but differs in that the information provided is a counterfactual comparison (the test on a state the agent never visited) rather than an observation of the current execution. 3 The BSG-VA measurement method agenttrajectoryvalidation eventcommand + working treetest-onlypatchB: buggy baseS: candidateG: gold fixevidencerolesame command,pinned container Figure 1: BSG-VA: each self-selected validation event is captured at its exact working-tree state, reduced to a test-only patch, and replayed deterministically on the buggy base (B), the captured candidate (S), and the developer gold fix (G); the captured outcome and the replay pattern assign an evidence role 3.1 Event capture BSG-VA instruments the agent’s tool loop so that every shell command the agent issues after its first production edit is intercepted and classified (Figure 1). Classification is online and identical across all experimental arms. Each command is labelled as one of four types: a project test run (an invocation of the repository’s own test runner on its own test files), a scripted explicit oracle (an agent-written script whose exit code encodes a pass/fail verdict), an observational probe (output to be inspected by the model without a programmatic oracle), or a non-validation action. Non-validation actions are logged but excluded from the measurement. The classifier was audited against a sample of 111 events labelled manually by an author; it agreed on 92.8% of them, the 8 disagreements were adjudicated by the second author, and all resulting corrections were adopted into the production classifier. For each validation event the harness records the command string, the standard output and error streams, the exit status, and the working-tree state at execution time. The working-tree state is decomposed into two components: a production patch containing the agent’s changes to the repository’s source and configuration files, and a test-only patch containing everything the agent added or modified to support the validation itself (new test files, test helpers, fixture data). The decomposition matches changed paths against the repository’s existing test directory conventions and the set of files the agent created during the trajectory. 3.2 Test-only patch extraction Reducing the captured state to a test-only patch is the step that makes cross-state replay meaningful. If the full working tree were replayed on B, any test depending on the agent’s production changes would fail for trivial reasons (missing functions, changed imports), and the replay outcome would reflect dependency breakage rather than evidential content. The test-only patch retains only the validation artefacts: the test files, scripts, and fixtures the agent created or modified, applied on top of whatever code state the replay targets. An event whose test-only patch is empty indicates that the agent ran a pre-existing repository test without adding test code; these events are retained because the choice of which existing test to run is itself a validation decision. The extraction handles several boundary cases. When the agent’s test imports a function that exists only in the agent’s production patch, the replay on B will fail with an import error rather than a test-logic failure. BSG-VA flags such events as depending on a new production symbol. When the agent creates a temporary file that its test reads, the file is included in the test-only patch. When the agent modifies an existing test file, the extraction preserves the modification against the repository’s baseline version of that file. These cases are defined in advance in the event codebook; the classification is deterministic given the working-tree snapshot. 3.3 Cross-state replay Each retained event is replayed on three code states, each constructed in a fresh container from the same base image used during the agent’s original run. State B is the original buggy code with the test-only patch applied. B represents the question: would this validation event have detected the reported defect? If the test passes on B, the event carries no information that distinguishes buggy from candidate code. State S is the captured candidate code. Replaying on S checks consistency: in the common case the outcome matches what the agent observed during the trajectory, and Section 3.4 defines how the remaining cases are classified. State G is the developer gold fix: the buggy code with the benchmark’s gold patch applied, plus the same test-only patch. G answers the question: is the event’s discriminating power tied to the specific candidate, or does it generalise to the correct fix? An event that fails on B and passes on S but also fails on G discriminates the bug from this particular candidate, but would reject the developer’s own solution. Every event is replayed once per state. Stability is assessed by comparing the S-replay with the captured outcome: disagreement marks the event flaky, and flaky events are excluded from all constructs. This rule favours precision over recall: only stable observations support role assignment. 3.4 Evidence-role taxonomy Two kinds of outcome attach to each event. The captured outcome is the result the agent observed during the trajectory. The replay outcomes are what the offline replays produce on B, S, and G. Roles are assigned in a fixed order that makes the seven categories mutually exclusive and exhaustive: the captured outcome anchors the assignment, the B and G replays supply the bug contrast, and the S-replay enters through the stability comparison of Section 3.3. DIAGNOSTIC_NEGATIVE. The captured outcome is not positive. The event gave the agent no positive result on the candidate and carries no positive evidence, whatever its replays show. FLAKY. The captured outcome is positive but the S-replay disagrees with it. The observation is not reproducible; flaky events are excluded from all constructs. NOT_COMPARABLE. The captured outcome is positive and stable, but the B-replay or the G-replay is not evaluable. Reasons include import errors from tests that depend on symbols introduced by the agent’s production edits, missing temporary artefacts, container setup failures, and timeout. Every remaining event has a stable positive captured outcome with evaluable B and G replays and receives one of four roles. GOLD_ALIGNED_BUG_DISCRIMINATING. B fails, G passes. The event detects a property present in the buggy code and absent in both the candidate and the gold fix. This is the strongest form of validation evidence: it targets the defect and generalises beyond the specific candidate. CANDIDATE_SPECIFIC. B fails, G fails. The event distinguishes the bug from the candidate, but its discriminating power depends on implementation details unique to the agent’s patch. The gold fix does not satisfy this test. REGRESSION_ONLY. B passes, G passes. The event confirms only that the candidate has not introduced a failure relative to the original code. It says nothing about whether the bug is fixed. MISLEADING. B passes, G fails. The check is one the buggy code satisfies and the developer gold fix does not: the passing result contradicts the developer’s fix, so reading it as support for the candidate misleads. An event is positive comparable when its captured outcome was positive and its assigned role is one of the four roles from GOLD_ALIGNED_BUG_DISCRIMINATING to MISLEADING; under the assignment order above, these are exactly the events that reach the four outcome roles. These are the events the agent could have counted as supporting evidence, evaluated for what they in fact establish. Two rollout-level constructs aggregate the event roles. A rollout provides bug-discriminating evidence when it contains at least one gold-aligned bug-discriminating or candidate-specific event, the two roles whose checks fail on the buggy state. Evidence-inadequate closure (EIC) is true when a rollout submitted a patch, produced at least one positive comparable event, and contains no bug-discriminating event: every positive comparable result is regression-only or misleading. 3.5 Use of AI assistants An LLM assistant (Claude, Fable 5 model, Anthropic) was used in four capacities: configuring the container-based experiment environment (Docker images, harness wiring, replay infrastructure); implementing and running the harness and orchestration code that launched experiment sessions and collected data, under designs specified by the authors (the repair agents under study are separate OpenAI models, not this assistant); writing analysis code; and improving the readability and language of text written by the authors. All experimental parameters, analysis plans, and decision rules were specified by the authors before the corresponding model requests. The analysis code was verified by the authors through code review, unit-level checks against manually computed reference values, and reproduction of every reported statistic from the released data. The authors reviewed all LLM-assisted text, revised it against the style and accuracy standards of the study, and take full responsibility for the final content. 4 Study design 4.1 Task population We drew tasks equally from two sources: SWE-bench Verified, a curated subset of SWE-bench with confirmed human solutions [17, 31], and SWE-rebench, a dataset of GitHub issues filed after the model’s training cutoff, with lower contamination risk [3]. The two sources define two strata, one of curated verified tasks and one of recent tasks with lower contamination risk, weighted equally in all primary analyses. Each task specifies a repository, a commit, an issue description, and a container image. Official resolution is judged by each benchmark’s evaluator on the submitted production patch, with agent-written test artefacts stripped before evaluation. 4.2 Three-arm design Every task is run under all three arms, with 2 repetitions per task and arm: a fully crossed, within-task design. Because each task serves as its own control, no between-arm randomisation of tasks is required, and every arm contrast is estimated within task by the estimator of Section 4.3. Baseline. The agent works with the default scaffold: an unconstrained tool-use loop in which the model freely interleaves file reads, edits, and command execution until it submits (hereafter the unconstrained loop). No messages are injected. Static Reminder. After each eligible positive validation event, a system message prompts the agent to reconsider whether its evidence targets the reported defect. The message carries no B-replay information. Its structure, length, and directive strength are matched to the BCF message, so this arm serves as an attention control that isolates the effect of the B-replay content from the effect of prompting the agent to attend to evidence quality. Table 1 reproduces the text. Bug-Contrast Feedback (BCF). After each eligible positive validation event, the harness replays the event’s test-only patch on B in real time and injects the outcome as a system message. If the test passes on B, the message states that the check does not distinguish the candidate from the original buggy code. If the test fails on B, the message confirms that the check detects a difference. Events where the B-replay is not evaluable receive a message stating that the comparison was not possible. Table 1 reproduces all message variants. All arms share one model configuration: gpt-5.6-sol in standard reasoning mode with high effort, 6,000 maximum output tokens per response, and no repository network access. Each rollout is capped at 200 responses and 200 tool calls. Eligibility for feedback is defined identically in all arms: the event must be a validation event with a positive captured outcome, and it must not be a duplicate of an event already triggered in the same rollout. In the baseline arm, eligible events are identified but no message is sent; this ensures that trigger-rate statistics are comparable across arms. Table 1: Intervention messages. BCF delivers exactly one of its three variants according to the deterministic B-replay outcome00footnotetext: Messages are reproduced verbatim as delivered to the agent (hence the American spelling). Condition Message Static Reminder This check passed in the current candidate. A passing check may cover only part of the reported issue. Before concluding, consider whether it is sensitive to the reported behavior and whether regressions remain. BCF, B passes This check also passes on the original buggy version. It provides regression evidence but does not distinguish the two states. Before concluding, consider whether your validation is sensitive to the reported behavior and whether regressions remain. BCF, B fails This check fails on the original buggy version and passes in the current candidate. It distinguishes the two states, but one check does not establish complete correctness. Before concluding, consider remaining issue behavior and regressions. BCF, not comparable The original-version replay did not reach a comparable behavioral check, so the current pass is unverified evidence for the reported issue. Before concluding, consider a check that is directly sensitive to the reported behavior. 4.3 Confirmatory experiment The confirmatory experiment covers 110 tasks (55 per stratum) with 2 repetitions of each task under each arm, giving 660 planned cells. 16 candidate tasks failed an automated preflight environment check at startup and were replaced by the next candidate from the same stratum before any model request, giving a final set of 110 tasks across 64 repositories. Table 2 summarises the design. The analysis plan, task list, and decision rules were specified before any confirmatory model request. The primary outcome is EIC (BCF versus Static Reminder). The secondary outcome, tested in a fixed sequence gated on the primary [44], is bug-discriminating evidence (BCF versus Static Reminder). The estimator is a task-level paired difference: for each task with at least one observation in both arms, we compute the within-task rate difference, average within each stratum, and take the unweighted mean of the two stratum averages (the equal-source estimate, weighting the two strata equally). Inference uses a two-stratum Welch t-test with Satterthwaite degrees of freedom [43, 36]. The smallest effect size of interest (SESOI) [21] was set before execution to 10 percentage points: an effect of that size would change which of a repair agent’s closures an integrator would accept. The SESOI sized the experiment; it is not a threshold for dismissing smaller observed effects. A further secondary outcome, closure without positive machine-verifiable evidence, monitors whether the feedback suppresses validation activity. The analysis plan also includes a secondary associational analysis (RQ2) examining the relationship between evidence quality and repair success through logistic models with repository-grouped cross-validation, computed from the released data with the plan’s seed and procedure. Table 2: Confirmatory experiment population and design Tasks 110 (55 per stratum) Repositories 64 Arms Baseline, Static Reminder, BCF Repetitions per task and arm 2 Planned rollout cells 660 Observed cells 643 Cells lost to failures (by arm) 17 (6/5/6 by arm) Model gpt-5.6-sol, standard reasoning, high effort Scaffold unconstrained tool-use loop; no timeout, no automatic retry Primary contrast BCF −- Static Reminder Primary outcome EIC (favourable direction negative) Key secondary outcome bug-discriminating evidence (favourable direction positive) 4.4 Exploratory replications: varying the scaffold and the model Two exploratory replications probe generality on a shared 20-task subset selected by a deterministic outcome-blind rule from the confirmatory task set. The scaffold replication was specified together with the confirmatory design; the second-model replication was designed after the scaffold results were available and specified before any request to the second model. The scaffold replication replaces the unconstrained loop with a plan-execute-verify scaffold that requires the agent to submit a plan before executing and gates the final submission on a verification phase. The model remains gpt-5.6-sol. Three arms, 2 repetitions per task, 120 cells. The second-model replication replaces gpt-5.6-sol with gpt-5.6-terra in the unconstrained loop. Three arms, 2 repetitions, 120 cells. Estimates from both studies are reported with confidence intervals and descriptive p-values and carry no confirmatory weight. 4.5 Power and sample size We calibrated sample size to the prespecified SESOI of 10 percentage points rather than to the pilot point estimates, and the estimates below are reported against it rather than filtered by it. No published threshold was available to take the value from, because the construct is introduced here; Section 4.3 gives the design judgement it rests on. Two pilots preceded the experiment; their data served only to develop the measurement and intervention pipeline and to parameterise the power model below, and pilot observations enter no confirmatory estimate. Under a data-generating process using the pilot’s stratum-level base rates and within-task dependence, 110 tasks with 2 repetitions yield 96.2% power for EIC and 93.2% for bug-discriminating evidence at the SESOI, and 91.2% joint power. A supplementary five-percentage-point scenario, added before execution, yields 28.8% joint power, consistent with the expectation that an effect below the SESOI would likely go undetected. 5 Results 5.1 Evidence-role taxonomy at scale Table 3 and Figure 2 report the distribution of evidence roles across 3,730 retained events from the confirmatory experiment. Of these, 2,548 events are positive comparable: the captured outcome was positive and the event holds one of the four comparable roles. Within this set the roles divide as follows: 1,007 gold-aligned bug-discriminating, 370 candidate-specific, 1,141 regression-only, and 30 misleading. Regression-only and misleading events account for 46.0% of positive comparable events. Among the 1,377 events where the check fails on B and passes on the candidate, 26.9% are candidate-specific: they fail on the developer gold fix G. At the rollout level, 72.9% of baseline rollouts contain at least one bug-discriminating event, while 23.8% exhibit evidence-inadequate closure. Both patterns coexist: 48.1% of baseline rollouts that submit a patch contain both discriminating and non-discriminating positive evidence. On the shared 20-task subset of the second-model replication, baseline EIC is 40.5% on gpt-5.6-terra versus 18.9% on gpt-5.6-sol. Figure 2: Evidence roles of the 3,730 retained validation events in the confirmatory experiment. Dark bars are positive comparable roles; hatching marks positive roles that do not discriminate the reported bug Table 3: Evidence roles of the 3,730 retained validation events in the confirmatory experiment00footnotetext: Shares in the two indented rows are relative to their parent row, marked a of positive comparable events and b of B-fail/S-pass events. Evidence role Events Share of events (%) GOLD_ALIGNED_BUG_DISCRIMINATING 1,007 27.0 CANDIDATE_SPECIFIC 370 9.9 REGRESSION_ONLY 1,141 30.6 MISLEADING 30 0.8 DIAGNOSTIC_NEGATIVE 797 21.4 NOT_COMPARABLE 370 9.9 FLAKY 15 0.4 All retained events 3,730 100.0 Positive comparable events 2,548 68.3 of which regression-only or misleading 1,171 46.0a B-fail/S-pass events 1,377 36.9 of which candidate-specific 370 26.9b 5.2 Evidence quality and repair success Over the 643 observed rollouts in 64 repositories (ten repository-grouped folds), the covariates-only model attains an out-of-fold Brier score of 0.231 against 0.203 for the covariates-plus-evidence model, a reduction of 0.028 (repository-cluster bootstrap 95% CI [0.003, 0.053]); out-of-fold AUROC rises from 0.521 to 0.694, and the augmented model recalibrates with intercept 0.22 and slope 0.63. These evidence terms carry associational predictive signal for official resolution beyond the covariates. The analysis is associational and predictive; it does not establish that evidence causes official resolution. 5.3 Confirmatory experiment Of 660 planned cells, 643 completed and 17 were lost to infrastructure failures, balanced across arms. EIC (BCF minus Static Reminder) is −-7.8 percentage points (95% CI [−-12.9, −-2.7], p=0.0029p=0.0029), based on 109 paired tasks. The difference in bug-discriminating evidence is +7.4 percentage points (95% CI [+1.7, +13.0], p=0.011p=0.011). Both outcomes favour BCF in the prespecified fixed sequence. Both point estimates fall below the prespecified 10-percentage-point SESOI; the confidence intervals include the SESOI but do not exclude effects well below it, so the data do not resolve whether the true effects reach that threshold. Official resolution (BCF minus Static Reminder) is +0.5 percentage points: BCF does not detectably change repair success. Closure without positive machine-verifiable evidence shifts by +0.4 points: BCF does not suppress validation activity. Sign tests, leave-one-repository-out estimation, a repository-cluster wild bootstrap [6], a complete-pairs restriction, and worst-case missing-cell bounds all preserve the primary direction; Appendix 8 reports them together with per-arm cell accounting and trigger exposure (Table 7). 5.4 Active-control decomposition Table 4 decomposes the BCF effect into two components. The generic reminder reduces EIC by 3.2 percentage points relative to baseline and raises bug-discriminating evidence by 4.1 points. BCF adds a further 7.8 points on EIC and 7.4 points on bug-discriminating evidence beyond the reminder. The total BCF-versus-baseline contrast is −-10.5 points on EIC and +10.9 on bug-discriminating evidence. Table 4: Arm-level rates and three-arm decomposition of the confirmatory effects (task-level equal-source contrasts; Welch p values outside the confirmatory family are descriptive) Rollout-level rate (%) Baseline Static Reminder BCF EIC 23.8 21.4 13.6 Bug-discriminating evidence 72.9 75.8 83.2 Official resolution 65.9 66.0 66.8 Contrast Difference (p) 95% CI (p) p Static −- Baseline, EIC −-3.22 [−-7.58, 1.13] 0.15 Static −- Baseline, bug-discriminating evidence 4.12 [−-0.94, 9.17] 0.11 BCF −- Static, EIC −-7.79 [−-12.87, −-2.72] 0.0029 BCF −- Static, bug-discriminating evidence 7.36 [1.73, 12.98] 0.011 BCF −- Baseline, EIC −-10.45 [−-15.78, −-5.13] 0.00018 BCF −- Baseline, bug-discriminating evidence 10.91 [4.82, 17.00] 0.00058 Secondary outcomes, BCF −- Static Closure without positive machine-verifiable evidence 0.44 [−-1.94, 2.82] 0.72 Official resolution 0.48 [−-3.73, 4.69] 0.82 5.5 Runtime and token overhead Table 5 reports per-rollout resource means by arm. The replay computation itself is inexpensive: BCF adds a median of 11.0 seconds of B-replay time per rollout, 3.3% of wall time at the median. The BCF-versus-Static difference in per-rollout token consumption is 99 thousand tokens, not statistically distinguishable from zero. Table 5: Runtime and token overhead of the interventions in the confirmatory experiment (per-rollout means over observed cells) Per-rollout mean Baseline Static Reminder BCF Model responses 29.8 29.1 31.4 Total tokens (10310^3) 956 946 1068 Output tokens (10310^3) 10.7 10.5 11.5 Wall time (s) 390 381 440 5.6 Exploratory replications Table 6 and Figure 3 report both exploratory replications on the shared 20-task subset; Appendix 8 details cell loss and trigger exposure. The scaffold replication reduces baseline evidence-inadequate closure to 8.3%, compared to 18.9% for the same tasks under the unconstrained loop, leaving little room for either component. On gpt-5.6-terra in the unconstrained loop, the generic reminder reduces evidence-inadequate closure by 7.5 points relative to baseline, matching the full BCF effect on sol, while the B-replay content adds nothing detectable on top; feedback reached 97.4% of intervention rollouts, so the null is not explained by too few intervention triggers. Across the two studies, the reminder-only arm shows the more consistent favourable pattern; an incremental BCF advantage is established only with sol in the unconstrained loop. Figure 3: Task-level equal-source differences with 95% CIs across settings. Filled markers show the primary contrast family; open markers show descriptive comparisons. The dashed line separates the confirmatory experiment from the exploratory replications Table 6: Exploratory replications on the shared 20-task subset (BCF −- Static Reminder; all quantities are descriptive)00footnotetext: Settings: the confirmatory experiment and the second-model replication run the unconstrained tool-use loop; the scaffold replication runs the plan-execute-verify scaffold. Models: gpt-5.6-sol for the confirmatory experiment and the scaffold replication; gpt-5.6-terra for the second-model replication. The confirmatory column restricts the confirmatory sample to the 19 of these 20 tasks with complete pairs. Interaction estimates compare each replication with the full confirmatory experiment and are descriptive. Confirmatory (same 20 tasks) Scaffold replication Second-model replication (terra) Observed / planned cells – 108/120 113/120 EIC difference (p) −-7.78 −-3.13 2.50 EIC 95% CI (p) – [−-10.51, 4.26] [−-16.08, 21.08] Bug-discriminating evidence difference (p) 7.78 −-1.88 −-2.50 Bug-discriminating evidence 95% CI (p) – [−-11.56, 7.81] [−-21.74, 16.74] Baseline EIC (%) – 8.3 40.5 Baseline bug-discriminating evidence (%) – 83.3 51.4 Interaction, EIC (p; p) – 5.00; 0.52 10.83; 0.24 Interaction, bug-discriminating evidence (p; p) – −-10.00; 0.33 −-10.56; 0.33 Cells lost to failures (by arm) – 12 (4/4/4) 7 (3/2/2) 6 Discussion 6.1 What the taxonomy reveals The central finding is a prevalence estimate: 46.0% of the positive validation evidence that agents produce mid-trajectory carries no information about the reported defect. This holds in a population where 72.9% of rollouts also contain at least one discriminating event. Agents frequently produce both kinds of evidence in the same run. The tool loop, however, exposes no signal that distinguishes the two kinds: nothing indicates whether a passing check would also have passed on B. A finer-grained pattern sits within the discriminating events. Among events where the check fails on B and passes on the candidate (the B-fail/S-pass pattern), 26.9% also fail on the developer gold fix G. Their discriminating power is tied to implementation details of the agent’s patch rather than to the defect itself: used as acceptance tests downstream, they would reject correct alternative fixes. This distinction is invisible without the G-replay that BSG-VA provides, and it bears on the growing practice of using agent-generated tests as patch validators [29]. On the shared subset of the second-model replication, baseline EIC is 40.5% on gpt-5.6-terra versus 18.9% on gpt-5.6-sol: the phenomenon is at least as pronounced on gpt-5.6-terra. 6.2 Awareness versus information We designed the three-arm experiment to distinguish two hypotheses about why BCF works. The first is that agents simply need to be reminded to evaluate their evidence. The second is that the specific B-replay content matters. The generic reminder, matched in structure and timing to BCF but carrying no replay information, serves as the attention control that separates these contributions. The results favour a mixed account. The reminder alone reduces EIC by 3.2 percentage points relative to baseline: a nonspecific prompting effect, captured by the attention-control arm. The B-replay content adds a further 7.8 points beyond it. On sol in the unconstrained loop, both components contribute. On terra, the prompting effect alone accounts for the full improvement. Under the plan-execute-verify scaffold, baseline EIC is already at 8.3%, and neither component has room to show an effect. The practical takeaway is that the cheaper intervention has the broader reach. A generic validation prompt, delivered at the right moment in the tool loop, produces a favourable shift in both models we tested and requires no replay infrastructure. The B-replay content adds incremental value with gpt-5.6-sol in the unconstrained loop. Whether that increment justifies the infrastructure cost depends on the deployment context. The measurement method is relevant either way: it provides the means to evaluate evidence quality regardless of whether a feedback loop is deployed. 6.3 Threats to validity Construct validity. Evidence roles are assigned against the developer gold fix as reference standard, which introduces noise on tasks where the gold fix is incomplete or where multiple valid fixes exist. The EIC construct depends on the validation-event definition set before data collection; alternative definitions of what counts as a validation event would shift the prevalence estimates. The measurement begins at the agent’s first production edit, so diagnostic commands issued during initial bug exploration fall outside the captured population. If agents produce discriminating checks before editing code, the reported non-discriminating share would overstate the trajectory-wide rate. Internal validity. Cells lost to infrastructure failures are balanced across arms in all three experiments, and the confirmatory worst-case bound preserves the sign of the effect; the second-model bound does not. External validity. Both models come from a single provider and model family, and the task population draws on two Python-ecosystem benchmarks. The exploratory replications cover a shared subset with few repetitions per arm, yielding wide confidence intervals; their results are descriptive. 7 Conclusion We introduce BSG-VA, a method that measures the evidential value of the validation activity repair agents perform mid-trajectory, by replaying each command on the buggy, candidate, and gold-fix code states. Applied at scale, the method reveals that 46.0% of positive comparable validation events carry no bug-discriminating information, and that 23.8% of baseline rollouts close on the basis of such evidence alone. Bug-contrast feedback, which returns the B-replay outcome to the agent in real time, reduces evidence-inadequate closure by 7.8 percentage points, without measurable cost to repair success. That estimate falls below the prespecified 10-percentage-point SESOI, so the direction of the effect is established while its practical magnitude is not. The active-control decomposition attributes roughly a third of the total improvement to directing the agent’s attention to evidence quality, and the exploratory replications show the same reminder-only pattern on the second model; the richer B-replay content adds value in one configuration, so a deployment without replay infrastructure still has a measured, cheaper option in the generic reminder. What holds across every configuration we tested is the measurement itself: BSG-VA and the evidence-role taxonomy assign each passing check in any replayable trajectory an evidence role with directly checkable content, making the adequacy of an agent’s validation evidence a property that can be audited before a patch is trusted. Statements and Declarations Funding. The authors received no funding for this work. Competing interests. The authors have no competing interests to declare. Ethics approval and consent to participate. Not applicable. This study involved no human participants and no animal subjects; all data are machine-generated execution records. Consent for publication. Not applicable. Data availability. The event-level, rollout-level, and task-level validation data supporting this study are openly available on Zenodo at https://doi.org/10.5281/zenodo.21642576 [48]. The release carries the three JSONL files behind every reported statistic. Replay environments are pinned by public container image references. The protocol documents, field definitions and comparability rules, and the raw trajectory logs are available from the corresponding author on reasonable request. Code availability. The BSG-VA capture, replay, and analysis code used to produce every reported statistic is not publicly released; it is available from the corresponding author on reasonable request. Author contributions. Conceptualization: Xiaonan Xu. Methodology: Xiaonan Xu. Formal analysis: Xiaonan Xu. Software: Wenjing Wu. Data curation: Wenjing Wu. Writing, original draft: Xiaonan Xu. Writing, review and editing: Xiaonan Xu and Wenjing Wu. Both authors read and approved the final manuscript. Use of AI tools. Section 3.5 details the role of an LLM assistant (Claude, Fable 5 model, Anthropic) in environment configuration, experiment implementation, analysis code, and language polishing. The authors take responsibility for all content. appendices 8 Robustness analyses and exposure details 8.1 Confirmatory robustness Table 7 collects the robustness checks for the primary contrast. The exact sign test on task-level differences is 19:5 for EIC (p=0.0066p=0.0066) and 21:7 for bug-discriminating evidence (p=0.013p=0.013). Leave-one-repository-out estimation is favourable in 63/63 repositories for EIC and 63/63 for bug-discriminating evidence. The repository-cluster wild bootstrap (99,999 draws) yields p=0.00020p=0.00020 for EIC and p=0.0018p=0.0018 for bug-discriminating evidence. Restricting to the 100 tasks with complete pairs in both intervention arms gives −-8.0 points on EIC. The 17 cells lost to infrastructure failures split 6/5/6 across baseline, Static Reminder, and BCF; under worst-case imputation of all of them, the EIC difference attenuates to −-5.0 points and the bug-discriminating-evidence difference to +4.5, preserving direction in both cases. Table 7: Confirmatory contrast (BCF versus Static Reminder, 109 paired tasks, equal-source weighting) and robustness analyses00footnotetext: Fixed-sequence family: EIC tested first at two-sided α=0.05α=0.05; bug-discriminating evidence tested only after EIC rejection in its favourable direction. Additional analyses are reported as robustness checks. EIC (p) Bug-discriminating evidence (p) Task-level difference −-7.79 7.36 Welch 95% CI [−-12.87, −-2.72] [1.73, 12.98] Welch two-sided p 0.0029 0.011 Repository wild bootstrap p 0.00020 0.0018 Wild bootstrap 95% CI [−-12.16, −-3.43] [2.55, 12.13] Exact sign test (favourable:unfavourable) 19:5 21:7 Exact sign test p 0.0066 0.013 Leave-one-repository-out favourable 63/63 63/63 Complete-pairs difference (100 tasks) −-8.00 7.45 Worst-case missingness bound −-5.00 4.55 8.2 Trigger exposure Feedback reached 97.2% of Static Reminder rollouts and 96.7% of BCF rollouts in the confirmatory experiment. In the second-model replication the trigger rate is 97.4% in both intervention arms. 8.3 Replication cell loss The scaffold replication lost 12 of 120 cells (4/4/4 by arm); 6 of these are one task whose rollouts the scaffold’s submission gate aborted, affecting all arms equally (two cells per arm). The second-model replication lost 7 cells (3/2/2 by arm) to infrastructure failures. The second-model worst-case missingness bound can reverse the sign of the estimate. References Ahmed . [ 2025] APACrefauthorsAhmed, T., Ganhotra, J., Pan, R., Shinnar, A., Sinha, S. Hirzel, M. 2025. : Generating Tests from Issues to Validate SWE Patches Otter: Generating Tests from Issues to Validate SWE Patches. Conference on Machine Learning (ICML). International conference on machine learning (ICML). APACrefURL https://arxiv.org/abs/2502.05368 Ahmed . [ 2026] APACrefauthorsAhmed, T., Ganhotra, J., Shinnar, A. Hirzel, M. 2026. : Evolutionary Reproduction Test Generator. EvoOtter: Evolutionary Reproduction Test Generator. APACrefURL https://arxiv.org/abs/2607.02854 preprint Badertdinov . [ 2025] APACrefauthorsBadertdinov, I., Golubev, A., Nekrashevich, M., Shevtsov, A., Karasik, S., Andriushchenko, A. , B. 2025. -rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. APACrefURL https://arxiv.org/abs/2505.20411 Banik . [ 2026] APACrefauthorsBanik, D., Chowdhury, K. Shamim, S.I. 2026. Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code. All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code. APACrefURL https://arxiv.org/abs/2606.18168 AITest 2026 accepted Bouzenia Pradel [ 2025] _TRAJAPACrefauthorsBouzenia, I. Pradel, M. 2025. Software Engineering Agents: A Study of Thought-Action-Result Trajectories Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories. 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) 2025 40th IEEE/ACM international conference on automated software engineering (ASE) ( 2846–2857). . Cameron . [ 2008] 2008APACrefauthorsCameron, A.C., Gelbach, J.B. Miller, D.L. 2008. -Based Improvements for Inference with Clustered Errors Bootstrap-Based Improvements for Inference with Clustered Errors. of Economics and Statistics903414–427, APACrefDOI https://doi.org/10.1162/rest.90.3.414 Ceka . [ 2026] APACrefauthorsCeka, I., Mitchell, H., Pujar, S., Buratti, L., Ramji, S., Yang, J. , B. 2026. Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study. Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study. APACrefURL https://arxiv.org/abs/2506.08311 2026 accepted D. Chen . [ 2024] APACrefauthorsChen, D., Lin, S., Zeng, M., Zan, D., Wang, J ., Cheshkov, A. , Q. 2024. : Issue Resolving with Multi-Agent and Task Graphs. CodeR: Issue Resolving with Multi-Agent and Task Graphs. APACrefURL https://arxiv.org/abs/2406.01304 X. Chen . [ 2023] 2023SELFDEBUGAPACrefauthorsChen, X., Lin, M., Schärli, N. Zhou, D. 2023. Large Language Models to Self-Debug. Teaching Large Language Models to Self-Debug. APACrefURL https://arxiv.org/abs/2304.05128 Z. Chen . [ 2026] APACrefauthorsChen, Z., Sun, Z., Shi, Y., Peng, C., Gu, X., Lo, D. Jiang, L. 2026. the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents. Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents. APACrefURL https://arxiv.org/abs/2602.07900 v2 preprint Cheng . [ 2026] _BRTAPACrefauthorsCheng, R., Tufano, M., Cambronero, J., Wei, R., Shi, S., Uy, G. čić, F. 2026. Cogeneration of Bug Reproduction Test in Agentic Program Repair. Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair. APACrefURL https://arxiv.org/abs/2601.19066 preprint Fei . [ 2026] APACrefauthorsFei, Z., Pan, Y., Sarro, F., Ge, J., Liu, M., Ng, V. Ye, H. 2026. : Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation. Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation. APACrefURL https://arxiv.org/abs/2603.07326 preprint Guo . [ 2026] APACrefauthorsGuo, Y., Liu, Y., Zhang, J.M., Ma, Y., Lou, Y. Chen, Z. 2026. -Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests. SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests. APACrefURL https://arxiv.org/abs/2607.00990 preprint Hora Robbes [ 2026] APACrefauthorsHora, A. Robbes, R. 2026. Coding Agents Generating Over-Mocked Tests? An Empirical Study. Are Coding Agents Generating Over-Mocked Tests? An Empirical Study. APACrefURL https://arxiv.org/abs/2602.00409 2026 Huang . [ 2026] _CODEAPACrefauthorsHuang, D., Zhang, J.M., Harman, M., Du, M. Cui, H. 2026. the Influence of Incorrect Code on Test Generation. Measuring the Influence of Incorrect Code on Test Generation. APACrefURL https://conf.researchr.org/details/icse-2026/icse-2026-research-track/15/Measuring-the-Influence-of-Incorrect-Code-on-Test-Generation 2026 Research Track Jiang . [ 2023] 2023APACrefauthorsJiang, N., Liu, K., Lutellier, T. Tan, L. 2023. of Code Language Models on Automated Program Repair Impact of Code Language Models on Automated Program Repair. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) 2023 IEEE/ACM 45th international conference on software engineering (ICSE) ( 1430–1442). Jimenez . [ 2024] APACrefauthorsJimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. Narasimhan, K. 2024. -bench: Can Language Models Resolve Real-World GitHub Issues? SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Conference on Learning Representations (ICLR). International conference on learning representations (ICLR). APACrefURL https://arxiv.org/abs/2310.06770 Kang . [ 2023] APACrefauthorsKang, S., Yoon, J. Yoo, S. 2023. Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) 2023 IEEE/ACM 45th international conference on software engineering (ICSE) ( 2312–2323). Khatib . [ 2025] APACrefauthorsKhatib, L., Mathews, N.S. Nagappan, M. 2025. : Reproducing Bugs via Inversion of LLM-Generated Passing Tests. AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests. APACrefURL https://arxiv.org/abs/2507.17542 2026 Research Track Konstantinou . [ 2026] _BEFORE_TESTAPACrefauthorsKonstantinou, M., Tambon, F. Papadakis, M. 2026. the risk of coding before testing: An empirical study on LLM-based test generation workflow. On the risk of coding before testing: An empirical study on LLM-based test generation workflow. APACrefURL https://arxiv.org/abs/2607.05139 preprint Lakens . [ 2018] 2018APACrefauthorsLakens, D., Scheel, A.M. Isager, P.M. 2018. Testing for Psychological Research: A Tutorial Equivalence Testing for Psychological Research: A Tutorial. in Methods and Practices in Psychological Science12259–269, APACrefDOI https://doi.org/10.1177/2515245918770963 Leary . [ 2026] APACrefauthorsLeary, H., Hanuska, L. Brown, C. 2026. : Evidence Infrastructure for Foundation-Model-Assisted Test Generation. TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation. APACrefURL https://arxiv.org/abs/2606.10211 2026 arXiv Track Le Goues . [ 2012] 2012APACrefauthorsLe Goues, C., Dewey-Vogt, M., Forrest, S. Weimer, W. 2012. systematic study of automated program repair: Fixing 55 out of 105 bugs for $ 8 each A systematic study of automated program repair: Fixing 55 out of 105 bugs for $ 8 each. 2012 34th International Conference on Software Engineering (ICSE) 2012 34th international conference on software engineering (ICSE) ( 3–13). Lin . [ 2026] APACrefauthorsLin, Z., Zhu, J., Zhou, M., Wang, X., Sun, Z., Yang, R. , L. 2026. Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair. To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair. APACrefURL https://arxiv.org/abs/2606.26978 2026 accepted W. Ma . [ 2026] _SIGNALAPACrefauthorsMa, W., Chen, Z., Gu, J., Li, T., Liu, S. Jiang, L. 2026. Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents. Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents. APACrefURL https://arxiv.org/abs/2605.18332 preprint Y. Ma . [ 2026] 2TAPACrefauthorsMa, Y., Kereopa-Yorke, B. Schultz, B. 2026. to the Test: Coding Agents Deliver What You Check, Not What You Requested. Building to the Test: Coding Agents Deliver What You Check, Not What You Requested. APACrefURL https://arxiv.org/abs/2606.28430 preprint / Microsoft Research Madiraju Madiraju [ 2026] APACrefauthorsMadiraju, M.B. Madiraju, M.S.P. 2026. : Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents. RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents. APACrefURL https://arxiv.org/abs/2606.22678 preprint Meng . [ 2026] APACrefauthorsMeng, Q., Zhang, X., Ren, Z. Visser, J. 2026. : An Evidence-to-Action Framework for Agentic Program Repair. EviACT: An Evidence-to-Action Framework for Agentic Program Repair. APACrefURL https://arxiv.org/abs/2605.27238 preprint Mündler . [ 2024] APACrefauthorsMündler, N., Müller, M., He, J. Vechev, M. 2024. -Bench: Testing and Validating Real-World Bug-Fixes with Code Agents SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. in Neural Information Processing Systems (NeurIPS). Advances in neural information processing systems (NeurIPS). APACrefURL https://openreview.net/forum?id=9Y8zUO11EQ Nashid . [ 2026] 2TESTAPACrefauthorsNashid, N., Bouzenia, I., Pradel, M. Mesbah, A. 2026. 2Test: Generating Reproducing Test Cases from Issue Reports. Issue2Test: Generating Reproducing Test Cases from Issue Reports. APACrefURL https://conf.researchr.org/details/icse-2026/icse-2026-research-track/87/Issue2Test-Generating-Reproducing-Test-Cases-from-Issue-Reports 2026 Research Track OpenAI [ 2024] _VERIFIEDAPACrefauthorsOpenAI 2024. SWE-bench Verified. Introducing SWE-bench Verified. APACrefURL https://openai.com/index/introducing-swe-bench-verified/ -validated subset of SWE-bench Örwall [ 2024] APACrefauthorsÖrwall, A. 2024. Tools. Moatless Tools. APACrefURL https://github.com/aorwall/moatless-tools repository Podivilov . [ 2026] _PRODAPACrefauthorsPodivilov, A., Lomshakov, V., Savin, S., Startsev, M., Pozharskiy, R., Parshin, M. Nikolenko, S. 2026. : Production-Assessed Trajectory Reviews for Coding Agent Evaluation. AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation. APACrefURL https://arxiv.org/abs/2607.06624 preprint Sahoo . [ 2026] _LUCKYAPACrefauthorsSahoo, P., Mittal, G., Li, X., Ma, S., Steenhoek, B., Lin, P. Hu, Y. 2026. : Revealing the Lucky Pass Problem in SWE-Agent Evaluation. AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation. APACrefURL https://arxiv.org/abs/2605.12925 v3 preprint Sajadi . [ 2026] APACrefauthorsSajadi, A., Nguyen, T., Huynh, K., Parra, E. Chatterjee, P. 2026. : Interactive Visualization of Agentic Program Repair Trajectories. TraceView: Interactive Visualization of Agentic Program Repair Trajectories. APACrefURL https://arxiv.org/abs/2606.22110 preprint Satterthwaite [ 1946] 1946APACrefauthorsSatterthwaite, F.E. 1946. Approximate Distribution of Estimates of Variance Components An Approximate Distribution of Estimates of Variance Components. Bulletin26110–114, APACrefDOI https://doi.org/10.2307/3002019 Shu . [ 2026] APACrefauthorsShu, R., Chong, C.Y., Zhou, X., Peng, Y., Wu, Z., Han, X. , Y. 2026. Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents. What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents. APACrefURL https://arxiv.org/abs/2607.06184 preprint Shull . [ 2008] 2008APACrefauthorsShull, F.J., Carver, J.C., Vegas, S. Juristo, N. 2008. role of replications in Empirical Software Engineering The role of replications in Empirical Software Engineering. Software Engineering132211–218, APACrefDOI https://doi.org/10.1007/s10664-008-9060-1 Smith . [ 2015] 2015APACrefauthorsSmith, E.K., Barr, E.T., Le Goues, C. Brun, Y. 2015. the cure worse than the disease? overfitting in automated program repair Is the cure worse than the disease? overfitting in automated program repair. of the 2015 10th Joint Meeting on Foundations of Software Engineering Proceedings of the 2015 10th joint meeting on foundations of software engineering ( 532–543). Sun . [ 2026] _MUTATIONAPACrefauthorsSun, Y., Zhao, Y., Wang, Y., Du, Y., Ma, Z., Wang, J. , Z. 2026. -Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? APACrefURL https://arxiv.org/abs/2605.22175 of ACL 2026 Wang . [ 2026] _HORIZONAPACrefauthorsWang, B., Zhang, C., Liu, D., Zhang, J., Chen, J., Li, M. , Z. 2026. Verification Horizon: No Silver Bullet for Coding Agent Rewards. The Verification Horizon: No Silver Bullet for Coding Agent Rewards. APACrefURL https://arxiv.org/abs/2606.26300 preprint Weimer . [ 2009] 2009APACrefauthorsWeimer, W., Nguyen, T., Le Goues, C. Forrest, S. 2009. finding patches using genetic programming Automatically finding patches using genetic programming. 2009 IEEE 31st International Conference on Software Engineering 2009 IEEE 31st international conference on software engineering ( 364–374). Welch [ 1947] 1947APACrefauthorsWelch, B.L. 1947. Generalization of ‘Student’s’ Problem when Several Different Population Variances are Involved The Generalization of ‘Student’s’ Problem when Several Different Population Variances are Involved. 341–228–35, APACrefDOI https://doi.org/10.1093/biomet/34.1-2.28 Westfall Krishen [ 2001] 2001APACrefauthorsWestfall, P.H. Krishen, A. 2001. weighted, fixed sequence and gatekeeper multiple testing procedures Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. of Statistical Planning and Inference99125–40, APACrefDOI https://doi.org/10.1016/s0378-3758(01)00077-5 Xia . [ 2025] APACrefauthorsXia, C.S., Deng, Y., Dunn, S. Zhang, L. 2025. LLM-Based Software Engineering Agents Demystifying LLM-Based Software Engineering Agents. of the ACM on Software Engineering2FSE801–824, APACrefDOI https://doi.org/10.1145/3715754 Xia . [ 2023] 2023APACrefauthorsXia, C.S., Wei, Y. Zhang, L. 2023. Program Repair in the Era of Large Pre-trained Language Models Automated Program Repair in the Era of Large Pre-trained Language Models. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) 2023 IEEE/ACM 45th international conference on software engineering (ICSE) ( 1482–1494). Xin Reiss [ 2017] APACrefauthorsXin, Q. Reiss, S.P. 2017. test-suite-overfitted patches through test case generation Identifying test-suite-overfitted patches through test case generation. of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis ( 226–236). Xu Wu [ 2026] 2026validationAPACrefauthorsXu, X. Wu, W. 2026 07. evidence in LLM repair agents: event-level, rollout-level, and task-level data. Validation evidence in LLM repair agents: event-level, rollout-level, and task-level data. , Zenodo. APACrefURL https://doi.org/10.5281/zenodo.21642576 Yang . [ 2024] APACrefauthorsYang, J., Jimenez, C.E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K. Press, O. 2024. -agent: Agent-Computer Interfaces Enable Automated Software Engineering SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. in Neural Information Processing Systems (NeurIPS). Advances in neural information processing systems (NeurIPS). APACrefURL https://arxiv.org/abs/2405.15793 Zhang . [ 2024] APACrefauthorsZhang, Y., Ruan, H., Fan, Z. Roychoudhury, A. 2024. : Autonomous Program Improvement AutoCodeRover: Autonomous Program Improvement. of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis Proceedings of the 33rd ACM SIGSOFT international symposium on software testing and analysis ( 1592–1604).