Paper deep dive
Catching Developers in the Flow: Low-Latency Agentic Program Repair at Google Scale
Celal Ziftci, Spencer Greene, Ray Liu, Livio Dalloro, Lorenzo Dini
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/7/2026, 9:04:04 AM
Summary
The paper introduces AutoRepair, an AI agent deployed at Google to automatically repair pre-submit test failures in continuous integration systems. Unlike existing post-submit APR tools, AutoRepair operates in the pre-submit outer-loop workflow with strict low-latency constraints to assist developers in real-time. It utilizes a ReAct-style generate-and-validate loop with an internal LLM (Gemini 2.5 Pro), pre-execution filters to skip non-viable changes, and post-execution filters to ensure fix relevance. Manual evaluation showed 67.18% accuracy on 195 test failures, and production deployment resulted in fixes being applied to 28,554 out of 295,508 changes.
Entities (10)
Relation Signals (9)
AutoRepair → developedby → Google
confidence 99% · we introduce AutoRepair, an AI agent deployed at Google
AutoRepair → hasaccuracy → 67.18%
confidence 95% · demonstrated 67.18% accuracy in suggesting correct fixes
AutoRepair → integratedwith → Cider
confidence 95% · Integrated into Google’s internal developer tools, Critique and Cider
AutoRepair → integratedwith → Critique
confidence 95% · Integrated into Google’s internal developer tools, Critique and Cider
AutoRepair → listensto → TAP
confidence 95% · AutoRepair listens to TAP notifications on test failures
AutoRepair → usesmodel → Gemini 2.5 Pro
confidence 95% · LLM = Internal version of Gemini 2.5 Pro
AutoRepair → usesmethodology → ReAct
confidence 90% · utilizes a ReAct-style generate-and-validate loop
AutoRepair → comparedto → AutoCodeRover
confidence 85% · State-of-the-art tools such as ... AutoCodeRover
AutoRepair → comparedto → SWE-Agent
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Manual repair of program failures is time-consuming and disruptive for software developers, particularly during the pre-submit phase where test failures occur within continuous integration systems. While Automated Program Repair has seen significant advancement through Large Language Models, existing state-of-the-art techniques primarily focus on post-submit workflows, operating offline without the low-latency requirements necessary to assist developers in real-time within their flow before they switch context. In this paper, we introduce FlowAgent, an AI agent deployed at Google to automatically repair test failures in the pre-submit outer-loop workflow inside continuous integration systems. Integrated into Google's internal developer tools, Critique and Cider,FlowAgent utilizes a ReAct-style generate-and-validate loop, as well as rigorous pre-execution and post-execution abstention filters to ensure high-quality suggestions under strict latency constraints. Based on our case studies, FlowAgent is highly effective. First, a manual evaluation conducted on 195 real-world test failures demonstrated 67.18% accuracy in suggesting correct fixes. Following its Google-wide deployment, FlowAgent suggested fixes on 295,508changes, of which developers previewed 65,069 and applied 28,554. Developer feedback from interviews indicate that the agent is useful in suggesting correct fixes, integration of autonomous repair agents into industrial software engineering workflows is received well, while interesting challenges and opportunities still remain.
Tags
Links
- Source: https://arxiv.org/abs/2610.07289v1
- Canonical: https://arxiv.org/abs/2610.07289v1
Trouble viewing inline? Open PDF directly →
Full Text
74,332 characters extracted from source content.
Expand or collapse full text
Catching Developers in the Flow: Low-Latency Agentic Program Repair at Google Scale CCS: Software and its engineering Software testing and debuggingCCS: Computing methodologies Intelligent agentsCCS: Software and its engineering Maintaining softwareCCS: Software and its engineering Automatic programming Celal Ziftci Affiliation: Google, New York, USA email: ziftci@google.com , Spencer Greene Affiliation: Google, New York, USA email: greenespencer@google.com , Ray Liu Affiliation: Google, New York, USA email: rayliu@google.com , Livio Dalloro Affiliation: Google, New York, USA email: dalloro@google.com and Lorenzo Dini Affiliation: Google, New York, USA email: dini@google.com Abstract. Manual repair of program failures is time-consuming and disruptive for software developers, particularly during the pre-submit phase where test failures occur within continuous integration systems. While Automated Program Repair has seen significant advancement through Large Language Models, existing state-of-the-art techniques primarily focus on post-submit workflows, operating offline without the low-latency requirements necessary to assist developers in real-time within their flow before they switch context. In this paper, we introduce AutoRepair, an AI agent deployed at Google to automatically repair test failures in the pre-submit outer-loop workflow inside continuous integration systems. Integrated into Google’s internal developer tools, Critique and Cider, AutoRepair utilizes a ReAct-style generate-and-validate loop, as well as rigorous pre-execution and post-execution abstention filters to ensure high-quality suggestions under strict latency constraints. Based on our case studies, AutoRepair is highly effective. First, a manual evaluation conducted on TotalEvalExamples real-world test failures demonstrated TotalEvalAccuracy accuracy in suggesting correct fixes. Following its Google-wide deployment, AutoRepair suggested fixes on TotalChangesPostedTo changes, of which developers previewed TotalChangesPreviewed and applied TotalChangesApplied. Developer feedback from interviews indicate that the agent is useful in suggesting correct fixes, integration of autonomous repair agents into industrial software engineering workflows is received well, while interesting challenges and opportunities still remain. Keywords: Testing, Program, Repair, LLM, AI, Agent, Maintenance, Production Preprint Notice This is a preprint of a paper accepted at the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026). 1. Introduction Software developers frequently modify codebases to add new features, fix bugs, or improve system security and performance. To ensure these modifications do not introduce regressions into the wider codebase, they must be rigorously tested. When code modifications result in test failures, developers must engage in program repair. This process is time-consuming and disruptive, particularly in the outer-loop pre-submit phase, where test failures occur outside the developers’ integrated development environments (IDE) within the continuous integration systems, like Google’s TAP (Memon et al., 2017; Micco, 2012), to identify global regressions before a change is submitted. Upon notification of a failure, developers typically investigate the failure and implement fixes. Our analysis reveals that the median time between a failure notification and a developer’s subsequent code edit at Google is FixAttemptLatencyP50. This provides a critical window where automated repair assistance could meaningfully improve developer velocity and reduce frustration. To alleviate the burden of manual debugging, the field of Automated Program Repair (APR) has received significant attention. Early APR solutions relied on manually described mutation rules, symbolic constraints, and static analysis. More recently, the advent of Large Language Models (LLMs) has driven a shift toward machine learning and autonomous agent-based APR. State-of-the-art tools such as SWE-Agent (Yang et al., 2024), AutoCodeRover (Zhang et al., 2024), Google’s Passerine (Rondon et al., 2025) and Meta’s Engineering Agent (Maddila et al., 2025) utilize ReAct-style (Yao et al., 2023) loops to autonomously navigate codebases, utilize tools, and generate fixes. However, these approaches primarily focus on the post-submit workflow, designed to repair programs in an offline capacity, operating without latency constraints required to assist developers in real-time before they switch context out of their flow. In this paper, we introduce AutoRepair, an AI agent deployed at Google to automatically debug and repair test failures directly within the pre-submit outer-loop workflow. Successful fixes, alongside an AI-generated summary of the root cause and the fix, are surfaced in Google’s internal development tools, allowing developers to preview and apply fixes inside their habitual workflows. To the best of our knowledge, unlike post-submit APR tools, AutoRepair is the first APR system in the pre-submit outer-loop workflow built to operate under critical latency requirements to catch developers in their active change modification flow. We provide a comprehensive empirical evaluation of AutoRepair with, to the best of our knowledge, the largest industrial dataset reported to date, starting first with a manual evaluation, then continuing over a multi-month period in production. We discuss AutoRepair’s success in obtaining valid fixes for thousands of failures, and present detailed telemetry and qualitative insights derived from adoption and user interviews, highlighting developer preferences, tooling adoption behaviors, and the varying levels of trust developers place in AI-generated code repairs. 2. Manual Program Repair Developers modify code for many reasons, e.g. to fix bugs, to add new features, to improve performance or security. Once code is modified, in one or more files, developers create a change, a mutation to the code repository, reviewed and submitted atomically. 2.1. Program Failure Stages Program failures may involve build failures where the code does not compile, or test failures where the program builds correctly but there are failing tests. Our work focuses on test failures only. In a change, when the modified code has accompanying tests that fail, either the program or the tests themselves may need to be updated. Such repairs may happen in two separate development stages: (1) Pre-submit: These repairs happen before the change is submitted, typically as part of the change itself, without impacting other developers. (2) Post-submit: These repairs happen after breaking changes have been submitted to the repository, in turn impacting all developers intending to modify the code, as there are now failing tests regardless of any new changes. Our work focuses on the pre-submit workflow, before faulty code or tests are submitted to the repository. During pre-submit, automated testing typically follows two distinct workflows depending on the stage of the change life-cycle: (1) Inner-loop: This is the code modification process within the integrated development environment (IDE). In this phase, either the developer or an AI coding agent, like Antigravity (Google Inc., 2026a), runs unit tests related to the modified code, providing early feedback before a change is even created. The tests executed in this workflow are typically either adjacent or in close proximity to the edited code, e.g. in the same or the parent directory. (2) Outer-loop: This is the stage where the local changes are complete and a change has been created. This phase focuses on global correctness, finding and executing all dependent tests in the entire monorepo to ensure that the change does not introduce regressions into the wider codebase across Google. Our work focuses on the outer loop, where various tools at Google are used by developers during their program repair workflows, discussed below. 2.2. Outer-Loop Pre-Submit Developer Tools There are several developer tools used at Google during the outer-loop pre-submit workflow to prepare a change for submission. Cider: This is Google’s internal IDE (Nguyen et al., 2025). Developers typically start code changes in Cider, utilize more tooling to help iterate on code changes, and move between Cider and other tooling as they finalize their changes. Critique: This is the internal web-based code review system at Google (Sadowski et al., 2018). After a developer changes code in Cider, they create a change with the modifications. Before sending the change to other developers for review, automated tools and regression tests run on the changed version of the code to help the owner with findings and suggested fixes by showing notifications on relevant parts of the code, or the entire change itself. TAP: The Testing Automation Platform (TAP) is the continuous integration system at Google that runs unit-tests (Memon et al., 2017; Micco, 2012). For a change, TAP finds all tests that depend on the change in the monorepo and runs them to identify potential regressions before the change can be submitted. An example TAP test failure finding is shown in Critique in Figure 1. Figure 1. Test failure finding surfaced in Critique (Sadowski et al., 2018), Google’s web-based internal code review system.Critique finding for failed tests Developers typically start coding in Cider, as they get ready to finish their modifications, they create a change and run analyses in Critique, at which time TAP may uncover failing tests due to the changes. Once notified of failures, developers then investigate and attempt to fix the failing tests by modifying their change or the tests themselves in Cider, manually or using assistive tooling. We conducted an analysis of code changes submitted across Google within a 30-day period during the month of September-2025 to understand developer behavior on failing tests. We investigated fix attempt latency, the wall clock time between TAP reporting test failures on a version of the change and a developer making any additional edits and moving the change to the following version, summarized in Figure 2. Cider suggested fix for failed tests Figure 2. Fix attempt latency distribution.Cider suggested fix for failed tests Fix attempt latency is the opportunity window to help developers with test failures, either by providing root cause analysis information or suggesting a potential fix, before they attempt to keep working on the change to fix the tests themselves, and it is FixAttemptLatencyP25 at p25, and FixAttemptLatencyP50 at p50. 3. Automated Program Repair With AutoRepair With the recent advances in AI, autonomous AI coding agents have shown great success on automated program repair (APR). To help developers in program repair, we propose AutoRepair, an autonomous agent integrated into Google’s developer tooling to automatically debug and repair pre-submit outer-loop test failures. Although AutoRepair and the tools it is integrated with are internal to Google, the logic and flow of AutoRepair can be replicated using standard developer tooling available in the industry. Figure 3. System overview of automatically repairing programs with AutoRepair. design The high level architecture of AutoRepair is shown in Figure 3. After a change is created and test execution is triggered in Critique, AutoRepair listens to TAP notifications on test failures. Upon failure, without developer involvement, it analyzes the change, checks if it passes supported filters, runs an AI-agent to fix the code, checks if the fix passes safety filters, and finally publishes its fix to Critique for developers to preview and apply either in Critique or Cider. 3.1. Pre-Execution Filters AutoRepair ignores certain changes with failure by using pre-execution filters to avoid nonviable or unnecessary fix attempts, and developer confusion due to unrelated or low quality fix suggestions. These filters, listed below, check characteristics of the change or the failing tests, and are created based on institutional knowledge, manual evaluations and observations on a set of failures, and developer feedback in production. Build failure Failure is due to build, not tests. Already failing Tests already failing in the repository are skipped. Flaky Failing known flaky tests (Fowler, 2011; Micco, 2016; Ziftci and Cavalcanti, 2020), those that pass and fail at the same code version without changes, are skipped. Automated Failing tests on changes created by automated tools are skipped. Stale Failing tests on changes that the developer already modified are skipped. Unsupported files Failing tests that involve certain file types, e.g. images and binary files, are skipped. Too many files Failing tests on changes with more than 100100 files are skipped. Too many lines Failing tests on changes with more than 10001000 lines are skipped If a test failure on a change passes all filters, AutoRepair is triggered. 3.2. Agentic Repair Once a test failure passes the pre-execution filters, the AutoRepair hybrid system with predefined and fully agentic steps running in a sandbox, shown in Figure 3, is triggered with the setup parameters: • LLM = Internal version of Gemini 2.5 Pro (Comanici et al., 2025) fine-tuned on internal Google code and data (Maniatis and Tarlow, 2023; Frömmgen et al., 2024), • temperature=0.1temperature=0.1 so that agent actions are mostly deterministic for easy debugging and reproducibility, • top=0.95top_p=0.95 so that there is room for creativity while keeping out extremely unlikely outcomes. First, the orchestrator initiates the agent by reading test logs and reading change information, including its description, the list of modified files and the modifications on them in diff (Wikipedia contributors, 2026; Hunt and McIlroy, 1976) format. These pieces of information are then passed to a ReAct-style (Yao et al., 2023) generate-and-validate loop using the prompt shown in Listing 4, where it can iteratively refine its fix based on build and test feedback, with: • Access to tools, including reading/editing/deleting files, searching code, building code, reading build status, reading build logs, executing tests, reading test status and reading test logs, • A maximum of 100 tool calls, to avoid excessive runtime and tool calling, • A maximum runtime of 30 minutes, in line with fix attempt latency discussed in Section 2.2 to avoid running for too long and to catch developers in their coding flow. If the loop finishes successfully and produces a fix diff, we conduct a final validation step, either by confirming that the agent executed the tests in its last validation cycle, or by executing the tests and checking the result again. This additional validation is necessary, as LLMs are known to hallucinate, and without final deterministic validation, there is a risk of showing a potentially invalid fix to developers. If the verification is successful, we summarize the failure root cause and the fix by making a single call to the LLM and passing the transcript of the entire trajectory of the generate-and-validate loop above, with the prompt shown in Listing 4. Finally, the suggested fix and its summary is passed to post-execution filters. The following change has failing pre-submit tests. The change information is provided below: === Change description === change_description === Change diff === change_diff Your task is to analyze the provided error logs from failing tests (paying close attention to +/- diffs indicating discrepancies), and the content of the relevant test file(s). Based on this analysis, you must output only a list of specific modifications required to fix the failing tests, along with their target file names. Your Analysis Steps (Internal Process): - Identify Test Failures & Discrepancies: Parse the error logs to pinpoint failing tests, error messages, stack traces. Crucially, analyze any + (expected) / - (actual) diffs within the logs, as these often directly indicate necessary corrections in test expectations or data. - Contextualize with Code: Analyze the test files (present in the error logs) to understand the structure, setup, assertions, and logic of the failing test(s) and related code. - Determine Required Modifications: Synthesize all information to identify the exact changes needed. This might involve updating expected values in tests based on log diffs. Do not add any comments or whitespaces. Your Output: - Your output must consist exclusively of the modification instructions and target files in diff format. - Do not include any explanations, analysis summary, greetings, apologies, or conversational text. Fix the following tests that produced errors: === Test name: test_name === test_error (a) The system prompt template used in the ReAct agent loop.Agent ReAct loop prompt You are an expert software engineer. You are given a transcript of an AI agent’s thoughts and actions to repair a test failure. Please provide a concise summary of what caused the test failure and the steps taken by the agent to repair the failure. Try your best to summarize the steps taken by the agent to fix the test. Focus on the specific actions taken and the thought process behind them. Do not simply copy the transcript of repair steps. When possible it’s best to include artifacts like file names, method headers, variables, etc. The resulting synopsis should be at most 5 sentences and suitable for a user interface, but ideally it can be one or two. If appropriate, presenting the synopsis in bullet points can be helpful. It is imperative that the summary follows HTML formatting, so make sure to use HTML as your only formatting style. Do not use any other formatting. The summary should be as compact as possible, this means the HTML should be written in a single line. === TRANSCRIPT === agent_transcript Agent summarization prompt (b) The prompt template to summarize the failure root cause and the suggested fix. Listing 4. Prompts used by AutoRepair. 3.3. Post-Execution Filters AutoRepair uses post-execution filters, listed below, to avoid posting fix suggestions to developers if the fix will not be useful. Stale If the developer modified the change while the agent was processing the failing version, the agent’s fix is obsolete. Submitted If the developer already submitted the change, the agent’s fix is obsolete. Once the fix passes these filters, it is ready to be displayed to developers. 3.4. Showing Fixes to Developers Developers have varying preferences on the environment where they tend to preview and apply fixes. Some developers prefer using Critique, as the fixes are posted there first, while others prefer going back to the Cider IDE, as they are more used to making changes there. To maximize developer adoption, AutoRepair is integrated into both Critique and Cider. AutoRepair fixes are displayed to developers in Critique as part of the TAP test failure notification, shown in Figure 1, instead of an additional finding, to help developers associate the fix to the failure, and to avoid overwhelming them with too many findings posted on their change, as there are hundreds of analyses executed for each change and several findings may be displayed. Shown in Figure 5, the fix contains a summary of the root cause of the failure for developers to first understand if the diagnosis is correct, before viewing and deciding to apply the suggested fix. Additionally, we provide a link to preview the fix in the Cider IDE, and a button to preview and apply it inside Critique. Figure 5. AutoRepair finding surfaced in Critique, with a root cause summary, and a link to Cider. AutoRepair finding for failed tests Developers can click the button to preview the fix in Critique, as shown in Figure 6, and apply it in place. Figure 6. AutoRepair suggested fix displayed in Critique.Cider suggested fix for failed tests For more complex fixes that span several lines or files, or if they want to perform further manual edits, developers may choose to preview and apply the fix in the Cider IDE, as shown in Figure 7. Figure 7. AutoRepair suggested fix displayed in Cider.Cider suggested fix for failed tests 4. Evaluation and Discussion In this section, we discuss how we evaluate our system, describe our case studies, and report the results of our interviews on the usability of our system. 4.1. Manual Evaluation To evaluate the effectiveness of AutoRepair, we first conducted a case study where we ran AutoRepair on TotalEvalExamples randomly selected TAP test failures from TotalEvalTeams distinct teams across Google, summarized in Table 1. We asked TotalEvalDevelopers expert developers with at least five years of software development experience each to assess the suggested fixes, similar to shown in Figure 6, and report whether the fix was appropriate for the intent of the change and the failure. We then held a meeting to go over their reports and align on any disagreements, to obtain final agreement on the assessment for each fix. Based on the experts’ reporting, AutoRepair was successful in suggesting correct fixes in TotalEvalExamplesGood of the failures, resulting in a TotalEvalAccuracy success rate. For the remaining 64 cases, the fixes were either incorrect or not exactly correct, discussed below with examples. Table 1. Statistics about the manual evaluations. # teams that owned the TAP failures evaluated TotalEvalTeams # developers that conducted evaluations TotalEvalDevelopers # failures evaluated TotalEvalExamples # accurate fixes on the failures evaluated TotalEvalExamplesGood (TotalEvalAccuracy) 4.2. Production Adoption & User Behavior After the manual evaluation, we launched AutoRepair to automatically run on all changes with TAP test failures, TotalChanges in total, and display suggested fixes across Google’s code repository, starting AutoRepairLaunchDate continuing to date, with detailed statistics listed in Table 2. Table 2. Statistics about AutoRepair executions and usage across Critique and Cider. # changes with failing tests TotalChanges # changes dropped by pre-execution filters TotalChangesDroppedPreExecution # changes AutoRepair attempted to fix TotalChangesAttempted (sym:funnelTotalChangesAttemptedPct) # changes AutoRepair obtained a fix TotalChangesAttemptedSuccessful (sym:funnelTotalChangesAttemptedSuccessfulPct) # changes dropped by post-execution filters TotalChangesDroppedPostExecution # changes AutoRepair posted a fix to TotalChangesPostedTo (sym:funnelTotalChangesPostedToPct) # changes developers previewed a fix TotalChangesPreviewed (sym:funnelTotalChangesPreviewedPct) # changes developers applied a fix TotalChangesApplied (sym:funnelTotalChangesAppliedPct) # changes with a fix previewed in Critique TotalChangesPreviewedCritique # changes with a fix applied in Critique TotalChangesAppliedCritique (CritiqueAppliedOverPreviewed) # changes with a fix previewed in Cider TotalChangesPreviewedCider # changes with a fix applied in Cider TotalChangesAppliedCider (CiderAppliedOverPreviewed) 4.2.1. System Trigger Our system triggered on TotalChanges changes with failing tests. TotalChangesDroppedPreExecution were skipped by pre-execution filters. A significant subset of the changes get skipped at this stage, as AutoRepair targets human-authored, viable fix attempts. This points to future improvement opportunities to attempt fixing a wider set of code changes, including non-human authored, and larger changes. 4.2.2. Fix Attempts Summarized in Table 3, AutoRepair attempted to fix TotalChangesAttempted changes authored by TotalAuthors distinct developers, there were a mean of ChangesMeanFiles and median of ChangesMedianFiles files per change, there were ChangesTotalFileExtensions distinct file extensions across all changes, there were a mean of ChangesMeanTests and median of ChangesMedianTests failing tests per change. Table 3. Statistics about code changes AutoRepair attempted to fix. # changes AutoRepair attempted to fix TotalChangesAttempted # distinct authors of those changes TotalAuthors mean # files per change ChangesMeanFiles median # files per change ChangesMedianFiles # distinct file extensions (e.g. .py) across changes ChangesTotalFileExtensions mean # failing tests per change ChangesMeanTests median # failing tests per change ChangesMedianTests mean # tool calls per AutoRepair execution ChangesMeanToolCalls mean # tokens used per AutoRepair execution ChangesMeanTokens total # tokens used by AutoRepair to date ChangesTotalTokens AutoRepair made a mean of ChangesMeanToolCalls tool calls, used a mean of ChangesMeanTokens tokens in each of its executions, and a total of ChangesTotalTokens tokens to date. Shown in Figure 8, from TAP failure notification to obtaining a successful fix, it took AutoRepair AutoRepairLatencyP50 in the median (p50), and AutoRepairLatencyP90 in the 90th percentile (p90) wall clock time, within the median fix attempt latency of FixAttemptLatencyP50 discussed in Section 2.2. AutoRepair successfully generated a suggested fix for TotalChangesAttemptedSuccessful changes. Unsuccessful attempts are due to the limits placed on the agent execution to avoid running for too long, as users may get notified of the failing tests and attempt to fix them manually themselves. Based on our internal experiments, we predict that a significant subset of the failed attempts at this stage may successfully produce fixes with improved language models released after our study. Figure 8. AutoRepair execution latency. AutoRepair execution latency 4.2.3. Successful but Dropped Fix Suggestions TotalChangesDroppedPostExecution successful fixes were dropped by post-execution filters, as the developers modified or submitted their changes already, and the agent’s fixes were obsolete. Some of the fixes may still be relevant even though the developer modified the change, as they may have modified unrelated files. However, such fixes may confuse developers, and potentially cause conflicts with the developer’s changes. As a result, they are dropped entirely by post-execution filters. Even though AutoRepair is optimized for low latency, a non-negligible share of successful fixes are still dropped by the post-execution filters, pointing to a need for further latency improvements to catch developers in their change modification flows. 4.2.4. Suggested Fixes After post-execution filters, TotalChangesPostedTo fixes were posted to Critique. Across Critique and Cider, TotalChangesPreviewed fixes were previewed and TotalChangesApplied were accepted by developers. Based on our manual analysis of a subset of previewed but not applied fixes of failing tests that have later been manually fixed by developers, we observe that there are several reasons why developers do not apply fixes, with examples shown in Figure 9. (a) @Ignore added to test (b) Assertion swallowed by try/catch block (c) Fix is close but not exact, should assert "== 3", not "> 2" Figure 9. Examples of suggested fixes not applied by users.Examples of suggested fixes not applied by users. First, even though the fix helps the tests pass, sometimes it is irrelevant or wrong, as shown in Figure 9 and 9. Such fixes should not be suggested to developers, as they undermine trust in the long run, and they should instead be verified with an additional step in our agent architecture, left as future work based on our study. Second, there are cases where the fix is correct, but not exactly what the developer wants, as shown in Figure 9. In these cases, developers most likely understand where to fix the test as they preview the suggested fix, but then proceed to fix it themselves. Then, there are cases where AutoRepair suggested reverting the modifications in the change, essentially undoing developer’s modification intent. Such suggestions should not be suggested to developers and have also been noted as future improvements to be addressed in the agent. Finally, there are cases when the fix was correct, but developers still typed it manually themselves. We predict that those developers did not realize they could apply the fix in Critique, and are using an editor other than Cider, where the link posted in our findings is not applicable, or various other reasons, some of which are discussed in Section 4.3. 4.2.5. Developer Preferences on Applying Fixes In Critique, TotalChangesPreviewedCritique fixes were previewed and TotalChangesAppliedCritique fixes were applied, while in Cider, TotalChangesPreviewedCider fixes were previewed and TotalChangesAppliedCider applied. Developers previewed some suggested fixes both in Critique and in Cider. Based on user interviews discussed in Section 4.3, developers sometimes preview a fix in Critique, but then move to Cider to apply it to make further edits on it afterwards. Additionally, AutoRepair may suggest multiple fixes for a single change, as developers work through their change to satisfaction, and tests may fail on several versions of the change. This is the reason that the sum of the number of changes with a fix applied in Critique and Cider is greater than the number of changes developers applied a fix in Table 2. The appliedpreviewed appliedpreviewed rate was CritiqueAppliedOverPreviewed in Critique, while it was CiderAppliedOverPreviewed in Cider. Developers seem to prefer viewing and applying fixes in Cider, compared to Critique. We propose that this is due to several factors, including developers’ tendency to make further modifications on the suggested fixes, and developers being used to viewing diffs in the IDE, as supported by user interviews discussed in the next section. 4.3. User Interviews We conducted 45 minute in-person interviews with TotalInterviewedDevelopers developers from TotalInterviewedDevelopers different groups that have been working at Google for at least 6 months, that previewed and/or applied suggested fixes either in Critique or in Cider, with excerpts discussed below. In this section, we summarize key learnings based on the participants’ responses. 4.3.1. Perceptions on AutoRepair First, we asked participants about their perception on automatically receiving fix suggestions from AutoRepair without triggering it themselves. Participants DeveloperF and DeveloperG reported enjoying the integration of AutoRepair into existing development workflows they are accustomed to. Then, we asked participants their prediction on how AutoRepair works, and they all reported that it is most likely an AI-based product, as the fix suggestions it provides are not likely to come from a non-AI tool, and its name suggests it uses AI. As a follow up, we asked them if this creates bias in their expectations of AutoRepair’s suggested fixes. As DeveloperA summarized it, participants reported that it does not matter where the suggestion comes from as long as it is relevant and accurate. [DeveloperG]: [...] I go to some other tasks in the mean time and then this new thing I’ve never seen before called AutoRepair just fixes the test for me! [DeveloperF]: This [workflow] is really great. It’s in-line with a lot of Critique workflows, so it’s following the same pattern to not have to think what type of review it is because the approach is always the same. [DeveloperA]: I don’t care what, who, or where the suggestion comes from, just that it is relevant and accurate. 4.3.2. Root Cause Summaries in Findings About the root cause summaries AutoRepair findings provide, participants DeveloperD, DeveloperE and DeveloperF reported easily understanding the failure reason by reading the root cause analysis summary in the finding. [DeveloperD]: It explains the issue pretty clearly. Thanks for building this! Can definitely see myself using this going forward :) [DeveloperE]: It pulled out the conclusion and put it at the top of the message, [...] know relatively fast glance what the error was; if this didn’t exist, it would have taken longer to dig through what the actual error was. [DeveloperF]: This information is helpful; usually have to go into test logs, but now it’s putting all the information in front, saves time; wouldn’t have to go into logs, filter by error, don’t need to scan all of that, have summarized info; can skip step of finding the issue. 4.3.3. Previewed and Applied Fixes Several participants, DeveloperA, DeveloperG, DeveloperH, reported finding AutoRepair fixes very useful, especially when the fix is on repetitive and tedious changes such as using mocks, and on refactoring related changes. [DeveloperG]: Did you guys implement AutoRepair on Critique? This thing is amazing! It fixed [change]. I updated some code, then the test failed cause it was using a bunch of mocks. I hate updating these types of tests. [DeveloperA]: The experience of AutoRepair fixing complex tests (especially those involving many mocks) was one of my biggest wow moments for LLM agents. [DeveloperH]: Thanks for developing and maintaining a tool which I find very promising and useful for developer velocity! Even my very first experience makes me excited about offloading the grungy task of looking at multiple documentation and past solved bugs to find easy parameter changes to fix tests. 4.3.4. Previewed but Not Applied Fixes Based on telemetry, we observed that, in some cases, developers previewed fixes but did not apply them, and we asked them about the reasons behind these. Participants DeveloperA, DeveloperB and DeveloperC reported that AutoRepair sometimes suggested irrelevant or wrong fixes. [DeveloperA]: The fix was inaccurate, I didn’t want to sort the list. [DeveloperB]: [...] it suggested to add something that wasn’t needed, so it wasn’t fixing anything. [DeveloperC]: It was inaccurate, I didn’t think the backslash should be removed. Participants DeveloperB and DeveloperC reported that AutoRepair findings sometimes arrive too late, by which point they have initiated their own manual fix. If the finding had come on time, they would have found it useful. [DeveloperB]: To be honest, I was frustrated by the broken [...] all day. I’ve manually debugged the problem, and it was cumbersome to find the root cause across so many different layers. I can confirm that your agent’s output is correct, and it would have saved me a lot of time if I had seen it before manually digging through the logs. [DeveloperC]: The [finding] did successfully identify the issue! Or at least, one of the issues. It definitely would have helped. Several developers also reported that they sometimes preview the suggested fix, but instead of applying it in Critique or Cider, they type the fix in the IDE themselves manually, as the proposed fix does not exactly conform to their coding style, and they prefer a slightly different edit, similar to shown in Figure 9. Participant DeveloperF reported that they applied the fix manually to save time and avoid issues that may stem from the Critique workflow conflicting with changes they wanted to make in their IDE. [DeveloperF]: I have a second monitor, or third monitor depending on where I am working [...], but my Cider is already open there. So instead of going to click Show Fix, apply the fix, [...], I just move my mouse over [...] and amend my code, in case I get into a [conflict] or something weird. Finally, participant DeveloperI reported that even if they fix the failures themselves, having an agent independently provide feedback is useful, as the agent has its own context on the intent of the change and the resulting failure. [DeveloperI]: I like having a separate agent in Critique to also chime in, as its context is fresh. Even though there are clear instructions in its system prompt, AutoRepair sometimes suggests reverting the change or commenting out the failing tests, as shown in Figure 9. Several developers reported getting frustrated about such fix suggestions, and some made humorous statements about AutoRepair both during our interviews and in internal forums. As this is a critical user trust problem, we are currently working on addressing it by adding an additional step into AutoRepair in Figure 3 to verify that the suggested fix is not removing tests or reverting the developer’s change, before sending the suggestion to the developer. 4.3.5. Usability Feedback During the interviews, several participants asked for a feature to have an estimate of the remaining time until AutoRepair finishes, so that they can decide whether to wait for its fix suggestion or to fix the failures themselves. [DeveloperD]: [...] if there’s an indicator that it’s running, I could say, "okay, it’s trying to fix some things", I could maybe go do something else and come back to this. Although we could not add the feature immediately, as it is not supported in Critique, we observe that developers are adapting to working with AI agents to take decisions on their waiting behaviors (Maister, 1985; Watkins et al., 2011). We are currently working on addressing this by adding additional mechanisms to notify users that AutoRepair is working on the failures on their change. 5. Threats To Validity In this section, we discuss the threats to the validity of our work and case studies. Manual evaluation: Our manual evaluation was conducted by three experienced developers. However, these developers did not own the production or test code of the failing tests, and may have made mistakes in their reporting of the accuracy of AutoRepair fixes. Additionally, the failures used in this evaluation were randomly chosen across Google, and may not be representative of all failures. Filters: The pre-execution filters are implemented based on early feedback from a small group of AutoRepair users. As a significant set of changes are skipped due to these filters, our evaluation results may not generalize to the changes with such characteristics. Developer bias: AutoRepair suggests fixes for all eligible code changes across Google. However, developers have varying positive/negative judgments on the use of AI and AI assistance in their development journeys. As we do not control for such variables in our evaluation of the previews and applies of the fixes, this may have introduced a selection bias, and biased the accuracy and validity of our results. Additionally, the interviews were conducted with code change authors that volunteered to give feedback. This constitutes a self-selected group, users who did not participate in our surveys and interviews might have different perspectives that could alter the qualitative insights gained from those sessions. Developer infrastructure: Our case study was conducted within the specific context of Google’s development environment and infrastructure. Although all internal tools have corresponding products in the industry, our findings may not be generalizable to other organizational settings. LLM: Our study uses an internal version of Gemini 2.5 Pro. Although popular large LLMs have been shown to perform well on tasks such as coding, the generalizability of our results to other LLMs may be limited. Additionally, AutoRepair relies on a specific prompt template for Gemini. Small changes in the prompt wording, structure, or ordering of instructions could potentially lead to different or degraded LLM performance. The robustness of the prompt engineering to minor variations or future LLM updates is a potential threat to the generalizability of our findings. Tests: As they are implemented by developers, the quality and detail of tests across different groups and systems at Google may vary significantly, and this may have influenced the accuracy of AutoRepair on different tests and the results of our case studies. 6. Related Work Automated program repair (APR) (Le Goues et al., 2019) has received significant attention in literature, starting with non-machine learning based approaches, then machine learning based approaches, later moving on to LLM based techniques, and most recently LLM agent based methods. Non-machine learning based APR: Several techniques approach APR as a search problem with manually described mutation rules and repair patterns (Le Goues et al., 2011; Le et al., 2016; Liu et al., 2019), or semi-automatic transformation rules derived from human implemented patches (Kim et al., 2013; Bader et al., 2019; Bavishi et al., 2019). Some approaches use symbolic constraints to extract fixes (Nguyen et al., 2013; Ke et al., 2015; Xuan et al., 2016; Mechtaev et al., 2016), while others use static analysis to identify and then repair bugs (Van Tonder and Le Goues, 2018; Liu et al., 2023; Jain et al., 2023), and some methods replace faulty code with similar but correct code from the same codebase (Yang et al., 2022). Beyond functional repairs, several approaches target different kinds of issues, including syntax errors (Wang et al., 2018; Gupta et al., 2019; Sakkas et al., 2022), performance issues (Yu and Pradel, 2018), security vulnerabilities (Harer et al., 2018), type errors (Chow et al., 2024), and build breakages (Tarlow et al., 2020). Machine learning based APR: Early work uses machine learning techniques to rank and choose fixes from among candidates (Long and Rinard, 2016). Recent work proposes using machine learning to generate fixes, including neural machine translation models to map faulty code to fixed code (Gupta et al., 2017; Tufano et al., 2019; Lutellier et al., 2020; Chen et al., 2019), tree transformation prediction models (Li et al., 2020; Zhu et al., 2021), neural models for more specific bug types (Vasic et al., 2019), and approaches that train models for specific repairs (Ye et al., 2022b; Ye et al., 2022a). SelfAPR (Ye et al., 2022a) trains a model using training samples obtained by perturbing a previous version of the program being repaired, enforcing the neural model to capture project specific knowledge. LLM-based APR: Recently, instead of training task-specific models, use of general purpose LLMs has been more common. AlphaRepair (Xia and Zhang, 2022) is a zero-shot technique that framed APR as an in-filling task, without requiring any additional training data. FitRepair (Xia et al., 2023) improves on AlphaRepair by incorporating the plastic surgery hypothesis (Barr et al., 2014), suggesting that the code needed to fix a bug typically exists within the same codebase already. Several techniques use a one-shot prompt technique, where a single prompt is sent to the model with problem descriptions (Jiang et al., 2023; Xia et al., 2023) or error messages (Joshi et al., 2023), and the model responds with the proposed code fix. More recent techniques use iterative approaches, where the LLM is prompted several times and the feedback obtained from the previous repair attempts are passed to the subsequent iterations (Xia and Zhang, 2024; Kang et al., 2023; Ye and Monperrus, 2024; Hidvégi et al., 2024). ChatRepair (Xia and Zhang, 2024) is a dialogue based technique that works through a conversation with the engineer as it incrementally collects more information, proposes potential patches and generates better patches over time to locate and repair the bug. ITER (Ye and Monperrus, 2024) improves partial patches until they become plausible and correct. Agentless (Xia et al., 2025) employs generated tests and a pre-defined three phase repair pipeline comprising of localization, repair and validation, achieving state-of-the-art performance at a remarkably low cost, outperforming previous open-source techniques. Agent-based APR: Most recently, new agentic approaches have emerged, where LLMs autonomously plan and perform tasks necessary to repair the program making use of tools provided to them. FixAgent (Lee et al., 2025) uses specialized LLM agents ("Tester" and "Debugger") through prompt chaining for end-to-end fault localization and repair. LLM4FL (Rafi et al., 2024) uses two LLM agents to iteratively analyze code to rank suspicious methods. SWE-Agent (Yang et al., 2024) implements a ReAct-style (Yao et al., 2023) open-ended loop with access to a set of APIs as tools to provide terminal access. RepairAgent (Bouzenia et al., 2024) builds upon prior work on augmenting LLMs with tools to generate and execute code (Gao et al., 2023) invoked via APIs (Schick et al., 2023), (Patil et al., 2023) and presents a state machine constrained agent, disallowing certain tool use sequences, unlike SWE-Agent. AutoCodeRover (Zhang et al., 2024) is an agent leveraging program metadata including class and method definitions. SpecRover (Ruan et al., 2024) builds upon AutoCodeRover to introduce a natural-language specification for expected behavior and incorporates an agent that reviews potential patches to guide the repair agent. CodeR (Chen et al., 2024) proposes a different architecture for APR, with a "manager" agent that coordinates sub-tasks described as a task graph. MarsCode Agent (Liu et al., 2024) proposes a dynamic, iterative APR approach with a generate-and-validate pipeline using multiple sub-agents. OpenHands (Wang et al., 2025) (previously named OpenDevin) is an open-source agent platform inspired by the Devin software engineering agent (Cognition, 2024), and provides an agentic foundation to build solutions for software engineering and other domains. Google’s Passerine (Rondon et al., 2025) is a ReAct-style (Yao et al., 2023) agent employing tools including a test-suite-based metric calculator for verification, is deployed to production use, and is reported to be three times more successful on fixing machine generated bugs. Passerine has recently been improved with techniques to filter out patches unlikely to be accepted by humans (Cambronero et al., 2025). Meta’s Engineering Agent (Maddila et al., 2025) also implements a ReAct-style (Yao et al., 2023) loop to automatically fix failing unit-tests using an iterative loop to provide feedback on potential patches using validation and verification tools. Extending this work, Meta’s MetaMateCR (Maddila et al., 2026) provides AI-assisted fixes for reviewer comments. Our work is an APR technique combining LLM-based and agent-based approaches discussed above. However, these prior work are focused on post-submit APR, i.e. repairs are performed offline in existing code without stringent low latency requirements. To the best of our knowledge, our work is the first one that triggers APR automatically upon failures in the pre-submit developer workflow, with latency requirements critical to the developer experience, and that reports outcomes and learnings in industrial scale for the pre-submit workflow respectively. Benchmarks for APR: The APR community created several public and private benchmarks to assess the quality of APR solutions. HumanEvalFix (Muennighoff et al., 2023) covers translated benchmarks for three software engineering downstream tasks, including program repair. SWE-Bench (Jimenez et al., 2024b) is a benchmark sourced from GitHub issues across 12 popular open source projects, containing 2294 Python bugs and repairs. It also offers a smaller version, SWE-Bench-Lite (Jimenez et al., 2024a), with 300 bug/fix pairs. SWE-Bench Verified (Jimenez et al., 2024b) is a human annotated subset of SWE-Bench deemed "solvable" and is divided into “easy” and a “hard” subsets. SWE-Bench has emerged as the standard evaluation benchmark for many APR techniques. DebugEval (Yang et al., 2025b) defines several software engineering tasks, including code repair, to evaluate LLMs’ capabilities across languages. DebugBench (Majdoub and Ben Charrada, 2024) is a large dataset of LeetCode (LeetCode, 2026) coding problems with artificially injected bugs to compare LLMs on APR. Recently, to address the complexity of creating benchmarks and facilitate comprehensive evaluations of APR methods, SWE-smith (Yang et al., 2025a) has been proposed as a pipeline to generate software engineering tasks at scale, to support the automatic synthesis of code changes that break existing tests, and to drive the development of more robust and scalable APR solutions. Another well-known benchmark is SWE-Lancer (Miserendino et al., 2025), which consists of over 1400 freelance software engineering tasks. Most recently, Google’s Passerine (Rondon et al., 2025) uses an internal benchmark similar to SWE-Bench, while Meta’s Engineering Agent (Maddila et al., 2025) uses an internal machine generated tasks benchmark. Our work has an internal benchmark comprised of real-world program failures at Google, discussed in Section 4. We are not able to publicly release our benchmark dataset due to the constraints discussed in Section Data Availability Statement. 7. Conclusion In this paper, we presented AutoRepair, an autonomous AI coding agent deployed at Google to automatically debug and repair test failures during the pre-submit outer-loop workflow. By integrating directly with Google’s internal tools, Critique and Cider, AutoRepair provides suggested fixes before developers switch context from their change modification flow. To ensure high-quality suggestions with low latency, the system employs pre-execution and post-execution filters, as well as an iterative generate-and-validate loop powered by an LLM. Our evaluation demonstrates the efficacy and practical viability of this approach at an industrial scale. In a manual evaluation by TotalEvalDevelopers expert developers, AutoRepair successfully suggested accurate fixes for TotalEvalAccuracy of evaluated test failures. Following its production deployment on AutoRepairLaunchDate, the system suggested TotalChangesPostedTo fixes, with developers accepting and applying TotalChangesApplied of the suggestions. Qualitative feedback from user interviews provides insights into adoption and user behaviors. Developers reported time savings and reduced frustration when diagnosing and fixing complex test failures. Challenges also remain, such as optimizing execution latency to catch developers before they context-switch, and mitigating unhelpful fix suggestions like commenting out tests. Our findings underscore the potential of integrating LLM-based autonomous agents into latency critical industrial developer workflows for APR. 8. Future Work We observed several improvement opportunities during our case studies of AutoRepair. First, we plan to improve AutoRepair’s test deletion and code revert behaviors to avoid developer frustration. Second, we plan to improve AutoRepair’s latency and success to provide more fixes using more recent language models. Additionally, we plan to add a feature to report to developers when AutoRepair is working on fixing the failing tests on their changes, so that they can wait for it to finish. Finally, we plan to extend AutoRepair to AI-authored changes as well, as there is a general shift in the industry towards more AI-generated code changes. Data Availability Statement The data that support the findings of this study are proprietary to Google and cannot be made publicly available. Due to strict confidentiality policies, commercial sensitivity, and non-disclosure agreements, we are unable to provide a public reproduction package or dataset. Acknowledgements. We thank Gaurav Johari for his work on surveying the APR domain at Google, Adrian Berding, Ahmed Omran, Allen Edwards, Assaf Raman, Bernhard Konrad, Chris Lewis, Daotong(Daniel) Dai, Gaurav Johari, James Wilson, Jeyavaishnavi Muralikumar, John Penix, Mariana Stariolo, Nic Cornejo, Peter Spragins, Rob Dryke, Rose Rodrigues, Sara Toth, Sruthi Santhosh Nair, Will Findley, Yang Liu, Yurun Shen for their work on AutoRepair, and Elaine Thai, Laurie Pham, Maria Arguello, Ryan McGarry for their help on conducting user interviews on AutoRepair usage. Google Gemini (Google Inc., 2026c) was utilized to generate sections of this work, specifically several tables in Latex (Lamport, 1994) format, and several graphs inside Google Colab (Google Inc., 2026b). References Bader et al. (2019) J. Bader, A. Scott, M. Pradel, and S. Chandra Getafix: learning to fix bugs automatically. Proceedings of the ACM on Programming Languages 3 (OOPSLA), p. 1–27. External Links: Document Cited by: §6. Barr et al. (2014) E. T. Barr, Y. Brun, P. Devanbu, M. Harman, and F. Sarro The plastic surgery hypothesis. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, p. 306–317. External Links: Document Cited by: §6. Bavishi et al. (2019) R. Bavishi, H. Yoshida, and M. R. Prasad Phoenix: automated data-driven synthesis of repairs for static analysis violations. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, p. 613–624. External Links: Document Cited by: §6. Bouzenia et al. (2024) I. Bouzenia, P. Devanbu, and M. Pradel RepairAgent: an autonomous, LLM-based agent for program repair. In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), External Links: Document Cited by: §6. Cambronero et al. (2025) J. Cambronero, M. Tufano, S. Shi, R. Wei, G. Uy, R. Cheng, C. Liu, S. Pan, S. Chandra, and P. Rondon Abstain and validate: a dual-llm policy for reducing noise in agentic program repair. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’26, p. 118–129. External Links: Document Cited by: §6. Chen et al. (2024) D. Chen, S. Lin, M. Zeng, D. Zan, J. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev, et al. CodeR: issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304. External Links: Document Cited by: §6. Chen et al. (2019) Z. Chen, S. Kommrusch, M. Tufano, L. Pouchet, D. Poshyvanyk, and M. Monperrus Sequencer: sequence-to-sequence learning for end-to-end program repair. IEEE Transactions on Software Engineering 47 (9), p. 1943–1959. External Links: Document Cited by: §6. Chow et al. (2024) Y. W. Chow, L. Di Grazia, and M. Pradel PyTy: repairing static type errors in Python. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, p. 1–13. External Links: Document Cited by: §6. Cognition (2024) Cognition Introducing devin, the first ai software engineer. External Links: Link Cited by: §6. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. External Links: Document Cited by: 1st item. Fowler (2011) M. FowlerEradicating non-determinism in tests(Website) Note: [Online; Accessed 2026-03-25] External Links: Link Cited by: §3.1. Frömmgen et al. (2024) A. Frömmgen, J. Austin, P. Choy, N. Ghelani, L. Kharatyan, G. Surita, E. Khrapko, P. Lamblin, P. Manzagol, M. Revaj, et al. Resolving code review comments with machine learning. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, p. 204–215. External Links: Document Cited by: 1st item. Gao et al. (2023) L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig PAL: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, p. 10764–10799. External Links: Link, Document Cited by: §6. Google Inc. (2026a) Google Inc. Google antigravity. Note: https://antigravity.googleOnline; Accessed 2026-03-25 Cited by: item (1). Google Inc. (2026b) Google Inc. Google colab. Note: https://colab.research.google.comOnline; Accessed 2026-03-25 Cited by: Acknowledgements. Google Inc. (2026c) Google Inc. Google gemini: Google’s ai assistant. Note: https://gemini.google.comOnline; Accessed 2026-03-25 Cited by: Acknowledgements. Gupta et al. (2019) R. Gupta, A. Kanade, and S. Shevade Deep reinforcement learning for syntactic error repair in student programs. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, p. 930–937. External Links: Document Cited by: §6. Gupta et al. (2017) R. Gupta, S. Pal, A. Kanade, and S. Shevade Deepfix: fixing common c language errors by deep learning. In Proceedings of the aaai conference on artificial intelligence, Vol. 31. External Links: Document Cited by: §6. Harer et al. (2018) J. Harer, O. Ozdemir, T. Lazovich, C. Reale, R. Russell, L. Kim, et al. Learning to repair software vulnerabilities with generative adversarial networks. Advances in neural information processing systems 31. External Links: Document Cited by: §6. Hidvégi et al. (2024) D. Hidvégi, K. Etemadi, S. Bobadilla, and M. Monperrus Cigar: cost-efficient program repair with llms. arXiv preprint arXiv:2402.06598. External Links: Document Cited by: §6. Hunt and McIlroy (1976) J. W. Hunt and M. D. McIlroy An algorithm for differential file comparison. Technical report Technical Report Computing Science Technical Report 41, Bell Laboratories, Murray Hill, NJ. External Links: Link Cited by: §3.2. Jain et al. (2023) N. Jain, S. Gandhi, A. Sonwane, A. Kanade, N. Natarajan, S. Parthasarathy, S. Rajamani, and R. Sharma Staticfixer: from static analysis to static repair. arXiv preprint arXiv:2307.12465. External Links: Document Cited by: §6. Jiang et al. (2023) N. Jiang, K. Liu, T. Lutellier, and L. Tan Impact of code language models on automated program repair. In Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), p. 1430–1442. External Links: Document Cited by: §6. Jimenez et al. (2024a) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan SWE-bench lite. Note: [Online; Accessed 2026-03-25] External Links: Link Cited by: §6. Jimenez et al. (2024b) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, External Links: Document Cited by: §6. Joshi et al. (2023) H. Joshi, J. P. C. Sánchez, S. Gulwani, V. Le, G. Verbruggen, and I. Radicek Repair is nearly generation: multilingual program repair with LLMs. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 5131–5140. External Links: Document Cited by: §6. Kang et al. (2023) S. Kang, B. Chen, S. Yoo, and J. Lou Explainable automated debugging via large language model-driven scientific debugging. Empirical Software Engineering. External Links: Document Cited by: §6. Ke et al. (2015) Y. Ke, K. T. Stolee, C. Le Goues, and Y. Brun Repairing programs with semantic code search (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 295–306. External Links: Document Cited by: §6. Kim et al. (2013) D. Kim, J. Nam, J. Song, and S. Kim Automatic patch generation learned from human-written patches. In 2013 35th international conference on software engineering (ICSE), p. 802–811. External Links: Document Cited by: §6. Lamport (1994) L. Lamport LaTeX: a document preparation system. 2nd edition, Addison-Wesley. Cited by: Acknowledgements. Le Goues et al. (2011) C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer Genprog: a generic method for automatic software repair. Ieee transactions on software engineering 38 (1), p. 54–72. External Links: Document Cited by: §6. Le Goues et al. (2019) C. Le Goues, M. Pradel, and A. Roychoudhury Automated program repair. Communications of the ACM 62 (12), p. 56–65. External Links: Document Cited by: §6. Le et al. (2016) X. B. D. Le, D. Lo, and C. Le Goues History driven program repair. In 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), Vol. 1, p. 213–224. External Links: Document Cited by: §6. Lee et al. (2025) C. Lee, C. S. Xia, L. Yang, J. Huang, Z. Zhu, L. Zhang, and M. R. Lyu Unidebugger: hierarchical multi-agent framework for unified software debugging. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 18248–18277. External Links: Document Cited by: §6. LeetCode (2026) LeetCode LeetCode - leetcode - the world’s leading online programming learning platform. Note: [Online; Accessed 2026-03-25] External Links: Link Cited by: §6. Li et al. (2020) Y. Li, S. Wang, and T. N. Nguyen Dlfix: context-based code transformation learning for automated program repair. In Proceedings of the ACM/IEEE 42nd international conference on software engineering, p. 602–614. External Links: Document Cited by: §6. Liu et al. (2019) K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé TBar: revisiting template-based automated program repair. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, p. 31–42. External Links: Document Cited by: §6. Liu et al. (2024) Y. Liu, P. Gao, X. Wang, J. Liu, Y. Shi, Z. Zhang, and C. Peng Marscode agent: ai-native automated bug fixing. arXiv preprint arXiv:2409.00899. External Links: Document Cited by: §6. Liu et al. (2023) Y. Liu, S. Mechtaev, P. Subotić, and A. Roychoudhury Program repair guided by datalog-defined static analysis. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, p. 1216–1228. External Links: Document Cited by: §6. Long and Rinard (2016) F. Long and M. Rinard Automatic patch generation by learning correct code. In Proceedings of the 43rd annual ACM SIGPLAN-SIGACT symposium on principles of programming languages, p. 298–312. External Links: Document Cited by: §6. Lutellier et al. (2020) T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan Coconut: combining context-aware neural translation models using ensemble for program repair. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, p. 101–114. External Links: Document Cited by: §6. Maddila et al. (2026) C. Maddila, N. Ghorbani, J. Saindon, P. Thakkar, V. Murali, R. Abreu, J. Shen, B. Zhou, N. Nagappan, and P. C. Rigby AI-assisted fixes to code review comments at scale. In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering, p. 439–449. External Links: ISBN 9798400726361, Link Cited by: §6. Maddila et al. (2025) C. Maddila, A. Tait, C. Chang, D. Cheng, N. Ahmad, V. Murali, M. Roch, A. Avondet, A. Meltzer, V. Montalvao, et al. Agentic program repair from test failures at scale: a neuro-symbolic approach with static analysis and test execution feedback. IEEE Transactions on Software Engineering. External Links: Document Cited by: §1, §6, §6. Maister (1985) D. H. Maister The psychology of waiting lines. In The Service Encounter: Managing Employee/Customer Interaction in Service Businesses, J. A. Czepiel, M. R. Solomon, and C. F. Surprenant (Eds.), p. 113–123. External Links: ISBN 9780669082739 Cited by: §4.3.5. Majdoub and Ben Charrada (2024) Y. Majdoub and E. Ben Charrada Debugging with open-source large language models: an evaluation. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, p. 510–516. External Links: Document Cited by: §6. Maniatis and Tarlow (2023) P. Maniatis and D. TarlowLarge sequence models for software development activities(Website) Note: [Online; Accessed 2026-03-25] External Links: Link Cited by: 1st item. Mechtaev et al. (2016) S. Mechtaev, J. Yi, and A. Roychoudhury Angelix: scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the 38th international conference on software engineering, p. 691–701. External Links: Document Cited by: §6. Memon et al. (2017) A. Memon, Z. Gao, B. Nguyen, S. Dhanda, E. Nickell, R. Siemborski, and J. Micco Taming google-scale continuous testing. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), p. 233–242. External Links: Document Cited by: §1, §2.2. Micco (2012) J. Micco Tools for continuous integration at Google scale. Google Tech Talk, Google Inc. Note: https://w.youtube.com/watch?v=KH2_sB1A6lAOnline; Accessed 2026-03-25 Cited by: §1, §2.2. Micco (2016) J. MiccoFlaky tests at google and how we mitigate them(Website) Note: [Online; Accessed 2026-03-25] External Links: Link Cited by: §3.1. Miserendino et al. (2025) S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke SWE-lancer: can frontier LLMs earn $1 million from real-world freelance software engineering?. arXiv preprint arXiv:2502.12115. External Links: Document Cited by: §6. Muennighoff et al. (2023) N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang, L. Von Werra, and S. Longpre Octopack: instruction tuning code large language models. In NeurIPS 2023 workshop on instruction tuning and instruction following, External Links: Document Cited by: §6. Nguyen et al. (2013) H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra Semfix: program repair via semantic analysis. In 2013 35th International Conference on Software Engineering (ICSE), p. 772–781. External Links: Document Cited by: §6. Nguyen et al. (2025) V. Nguyen, G. Herzog, J. Cambronero, M. Revaj, A. Kini, A. Frömmgen, and M. Tabachnyk Smart paste: automatically fixing copy/paste for google developers. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), External Links: Document Cited by: §2.2. Patil et al. (2023) S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez Gorilla: large language model connected with massive APIs. Advances in Neural Information Processing Systems. External Links: Document Cited by: §6. Rafi et al. (2024) M. N. Rafi, D. J. Kim, T. Chen, and S. Wang A multi-agent approach to fault localization via graph-based retrieval and reflexion. arXiv preprint arXiv:2409.13642. External Links: Document Cited by: §6. Rondon et al. (2025) P. Rondon, R. Wei, J. Cambronero, J. Cito, A. Sun, S. Sanyam, M. Tufano, and S. Chandra Evaluating agent-based program repair at Google. In Proceedings of the 47th International Conference on Software Engineering: Software Engineering in Practice Track, External Links: Document Cited by: §1, §6, §6. Ruan et al. (2024) H. Ruan, Y. Zhang, and A. Roychoudhury SpecRover: code intent extraction via LLMs. In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), External Links: Document Cited by: §6. Sadowski et al. (2018) C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli Modern code review: a case study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, p. 181–190. External Links: Document Cited by: Figure 1, Figure 1, §2.2. Sakkas et al. (2022) G. Sakkas, M. Endres, P. J. Guo, W. Weimer, and R. Jhala Seq2parse: neurosymbolic parse error repair. Proceedings of the ACM on Programming Languages 6 (OOPSLA2), p. 1180–1206. External Links: Document Cited by: §6. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. Advances in neural information processing systems 36, p. 68539–68551. External Links: Document Cited by: §6. Tarlow et al. (2020) D. Tarlow, S. Moitra, A. Rice, Z. Chen, P. Manzagol, C. Sutton, and E. Aftandilian Learning to fix build errors with graph2diff neural networks. In Proceedings of the IEEE/ACM 42nd international conference on software engineering workshops, p. 19–20. External Links: Document Cited by: §6. Tufano et al. (2019) M. Tufano, J. Pantiuchina, C. Watson, G. Bavota, and D. Poshyvanyk On learning meaningful code changes via neural machine translation. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), p. 25–36. External Links: Document Cited by: §6. Van Tonder and Le Goues (2018) R. Van Tonder and C. Le Goues Static automated program repair for heap properties. In Proceedings of the 40th International Conference on Software Engineering, p. 151–162. External Links: Document Cited by: §6. Vasic et al. (2019) M. Vasic, A. Kanade, P. Maniatis, D. Bieber, and R. Singh Neural program repair by jointly learning to localize and repair. CoRR abs/1904.01720. External Links: Link, 1904.01720, Document Cited by: §6. Wang et al. (2018) K. Wang, R. Singh, and Z. Su Search, align, and repair: data-driven feedback generation for introductory programming exercises. In Proceedings of the 39th ACM SIGPLAN conference on programming language design and implementation, p. 481–495. External Links: Document Cited by: §6. Wang et al. (2025) X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. OpenHands: an open platform for AI software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025, p. 65882–65919. External Links: Document Cited by: §6. Watkins et al. (2011) K. E. Watkins, B. Ferris, A. Borning, G. S. Rutherford, and D. Layton Where is my bus? impact of mobile real-time information on the perceived and actual wait time of transit riders. Transportation Research Part A: Policy and Practice 45 (8), p. 839–848. External Links: Document Cited by: §4.3.5. Wikipedia contributors (2026) Wikipedia contributors Diff - wikipedia. Note: [Online; Accessed 2026-03-25] External Links: Link Cited by: §3.2. Xia et al. (2025) C. S. Xia, Y. Deng, S. Dunn, and L. Zhang Demystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering 2 (FSE), p. 801–824. External Links: Document Cited by: §6. Xia et al. (2023) C. S. Xia, Y. Wei, and L. Zhang Automated program repair in the era of large pre-trained language models. In Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), p. 1482–1494. External Links: Document Cited by: §6, §6. Xia and Zhang (2022) C. S. Xia and L. Zhang Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, p. 959–971. External Links: Document Cited by: §6. Xia and Zhang (2024) C. S. Xia and L. Zhang Automated program repair via conversation: fixing 162 out of 337 bugs for $0.42 each using ChatGPT. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 819–831. External Links: Document Cited by: §6. Xuan et al. (2016) J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus Nopol: automatic repair of conditional statement bugs in java programs. IEEE Transactions on Software Engineering 43 (1), p. 34–55. External Links: Document Cited by: §6. Yang et al. (2022) D. Yang, X. Mao, L. Chen, X. Xu, Y. Lei, D. Lo, and J. He Transplantfix: graph differencing-based code transplantation for automated program repair. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, p. 1–13. External Links: Document Cited by: §6. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-Agent: agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37, p. 50528–50652. External Links: Document Cited by: §1, §6. Yang et al. (2025a) J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang SWE-Smith: scaling data for software engineering agents. Advances in Neural Information Processing Systems. External Links: Document Cited by: §6. Yang et al. (2025b) W. Yang, H. Wang, Z. Liu, X. Li, Y. Yan, S. Wang, Y. Gu, M. Yu, Z. Liu, and G. Yu COAST: enhancing the code debugging ability of llms through communicative agent based data synthesis. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 2570–2585. External Links: Document Cited by: §6. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, External Links: Link, Document Cited by: §1, §3.2, §6, §6, §6. Ye et al. (2022a) H. Ye, M. Martinez, X. Luo, T. Zhang, and M. Monperrus SelfAPR: self-supervised program repair with test execution diagnostics. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 92:1–92:13. External Links: Document Cited by: §6. Ye et al. (2022b) H. Ye, M. Martinez, and M. Monperrus Neural program repair with execution-based backpropagation. In Proceedings of the 44th international conference on software engineering, p. 1506–1518. External Links: Document Cited by: §6. Ye and Monperrus (2024) H. Ye and M. Monperrus Iter: iterative neural repair for multi-location patches. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), External Links: Document Cited by: §6. Yu and Pradel (2018) T. Yu and M. Pradel Pinpointing and repairing performance bottlenecks in concurrent programs. Empirical Software Engineering 23 (5), p. 3034–3071. External Links: Document Cited by: §6. Zhang et al. (2024) Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury Autocoderover: autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 1592–1604. External Links: Document Cited by: §1, §6. Zhu et al. (2021) Q. Zhu, Z. Sun, Y. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang A syntax-guided edit decoder for neural program repair. In Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, p. 341–353. External Links: Document Cited by: §6. Ziftci and Cavalcanti (2020) C. Ziftci and D. Cavalcanti De-flake your tests: automatically locating root causes of flaky tests in code at Google. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), p. 736–745. External Links: Document Cited by: §3.1.