Paper deep dive

Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes

Utku Boran Torun, Veli Karakaya, Eray Tüzün

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 86

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/10/2026, 2:45:13 AM

Summary

This study proposes and evaluates an automated, execution-based pipeline for verifying no-code bug fixes generated by Large Language Models (LLMs). Using a benchmark of 322 fixes across 12 configurations and 9 reproducible bug reports, the authors test three agentic executors (OpenCUA-72B, Claude Sonnet 5, and Meta Muse Glimmer) in a real browser environment. Results show that only 17.6% of issues could be set up, and fix resolution rates varied significantly (14.6% to 49.7%) depending on the executor. The choice of executor had a larger impact on outcomes than the LLM configuration itself, with executors matching human consensus only 66.1% to 88.1% of the time. The study concludes that LLM-generated no-code fixes are unreliable and require automated verification before reaching users.

Entities (10)

No-Code Fix · concept · 98%Automated Verification Pipeline · methodology · 95%Brave Browser · softwareproduct · 95%Claude Sonnet 5 · softwareagent · 92%Meta Muse Glimmer · softwareagent · 92%OpenCUA-72B · softwareagent · 92%Claude Opus 4.6 · softwaremodel · 90%External System & Dependency · bug-category · 90%Faulty Configuration · bug-category · 90%Wrong Version · bug-category · 90%

Relation Signals (8)

Automated Verification Pipeline → uses → OpenCUA-72B

confidence 95% · We repeat the pipeline with three executors: two Computer-Use Agents, OpenCUA-72B...

Automated Verification Pipeline → uses → Claude Sonnet 5

confidence 95% · We repeat the pipeline with three executors: ... Claude Sonnet 5...

Automated Verification Pipeline → uses → Meta Muse Glimmer

confidence 95% · ...and one multimodal agentic LLM, Meta's Muse Glimmer.

Executor Choice → impacts → Resolution Rate

confidence 92% · Changing only the executor shifted a configuration's resolution rate by 38.8% on average

Brave Browser → sourceof → Faulty Configuration

confidence 90% · benchmark of invalid Brave browser bug reports... Faulty Configuration... (19)

Brave Browser → sourceof → Wrong Version

confidence 90% · benchmark of invalid Brave browser bug reports... Wrong Version... (12)

Brave Browser → sourceof → External System & Dependency

confidence 90% · benchmark of invalid Brave browser bug reports... External System & Dependency... (20)

Claude Opus 4.6 → achievesbestresolution → Automated Verification Pipeline

confidence 85% · the strongest configuration, Claude Opus 4.6 in the Vanilla pipeline, resolved up to 74.1% of its fixes under Claude Sonnet 5.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:A no-code fix resolves an invalid bug report by directing the user to change a setting, update to a version where the problem is already fixed, or adjust their workflow. Manually verifying whether a proposed no-code fix resolves the reported bug takes considerable developer time. This study proposes an automated, execution-based pipeline for evaluating the capability of large language models (LLMs) to generate no-code fixes in a real browser environment. We evaluate 322 no-code fixes generated by the 12 configurations released with the benchmark of a previous study, covering bug reports categorized as Faulty Configuration, Wrong Version, or External System & Dependency. An executor agent applies each fix by following its natural-language instructions, and an issue-specific checker determines whether the reported bug persists. We repeat the pipeline with three executors: two Computer-Use Agents, OpenCUA-72B and Claude Sonnet 5, and one multimodal agentic LLM, Meta's Muse Glimmer. Only 17.6% of the candidate issues could be set up and passed both sanity gates. Across the 322 fixes, 14.6% to 49.7% resolved the bug depending on the executor, and the strongest configuration, Claude Opus 4.6 in the Vanilla pipeline, resolved up to 74.1% of its fixes under Claude Sonnet 5. Changing only the executor shifted a configuration's resolution rate by 38.8% on average, and the three executors reached the same verdict on only 46.9% of the fixes. Compared with human execution, the executors matched the human consensus for 66.1% to 88.1% of the sampled fixes. Even under the best executor, fewer than half of the LLM-generated no-code fixes resolve the reported bug, so such fixes need verification before they reach users. Execution-based verification can provide this, but the measured capability depends strongly on the executor, which evaluations must report and control.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

82,634 characters extracted from source content.

Expand or collapse full text

Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes CCS: Software and its engineering Empirical software validationCCS: Software and its engineering Software testing and debuggingCCS: Computing methodologies Natural language generation Utku Boran Torun Note: Both authors contributed equally to this research. email: boran.torun@bilkent.edu.tr Affiliation: Bilkent University, Ankara, Turkey , Veli Karakaya email: veli.karakaya@ug.bilkent.edu.tr Affiliation: Bilkent University, Ankara, Turkey and Eray Tüzün email: eraytuzun@cs.bilkent.edu.tr Affiliation: Bilkent University, Ankara, Turkey © none Abstract. Context. A no-code fix resolves an invalid bug report by directing the user to change a setting, update to a version where the problem is already fixed, or adjust their workflow, rather than by modifying the source code. Bug reports frequently admit such fixes, but manually verifying whether a proposed no-code fix genuinely resolves the reported bug consumes considerable developer time that could otherwise go toward core development work. Objective. This study proposes and evaluates an automated pipeline for the capability of large language models (LLMs) to generate no-code fixes, replacing manual verification with execution-based evaluation in a real browser environment. Method. We evaluate 322 no-code fixes generated by the 12 LLM-based configurations released with the benchmark of a previous study. From this benchmark, we select the bug reports categorized as Faulty Configuration, Wrong Version, or External System & Dependency. For each report, evaluation proceeds through an execution pipeline in which an executor agent applies each fix by following its natural-language instructions in a real browser environment, and an issue-specific checker determines whether the reported bug persists. We repeat the full pipeline with three executors from different providers: two Computer-Use Agents (CUAs), OpenCUA-72B and Anthropic’s Claude Sonnet 5 with computer use, and one multimodal agentic LLM, Meta’s Muse Glimmer, to characterize how much the choice of executor affects the outcome. Results. 17.6% of the candidate issues could be set up in the environment and passed both sanity gates, and the remaining issues could not be reproduced. Across the 322 fixes 14.6% to 49.7% resolved the bug depending on the executor, and the strongest configuration, Claude Opus 4.6 in the Vanilla pipeline, resolved up to 74.1% of its fixes under the executor Claude Sonnet 5. Changing only the executor shifted a configuration’s resolution rate by 38.8% on average, and the three executors reached the same verdict on only 46.9% of the fixes. Compared with human execution, the executors matched the outcome on which two human executors agreed for 66.1% to 88.1% of the sampled fixes. Conclusion. Even under the best executor, fewer than half of the LLM-generated no-code fixes resolve the reported bug, so such fixes need verification before they reach users. Execution-based verification can provide this without manual inspection, but the measured capability depends strongly on the executor, which evaluations must therefore report and control. Keywords: Large Language Models, No-Code Bug Fixes, Computer-Use Agents, Automated Fix Verification, Bug Reports, Automated Evaluation 1. Introduction Large language models (LLMs) have rapidly become part of everyday software engineering practice. They now assist developers across the software development lifecycle, powering code generation (Jiang et al., 2026; Zhang et al., 2023; Jiang et al., 2024), automated code review (Pornprasit and Tantithamthavorn, 2024; Cihan et al., 2025; Yu et al., 2024), test generation (Schäfer et al., 2023; Celik and Mahmoud, 2025; Zhang et al., 2024), and bug triage (Du et al., 2024; Aracena et al., 2024; Colavito et al., 2025). As these systems move from research prototypes into tools developers rely on daily (Stack Overflow, 2025) software engineering tasks, they are increasingly applied not only to writing and analyzing code, but to the surrounding work of managing the issues that software projects accumulate (Jimenez et al., 2024). One of the most time-consuming parts of that work is triaging bug reports, and in particular dealing with reports that turn out not to require a code change at all (Gon et al., 2026). A bug report is invalid when it does not describe a defect in the source code: the reported behavior instead stems from user misconfiguration, an outdated version, a conflicting extension, or a misunderstanding of intended behavior, or the report is not considered as a bug report at all, such as a feature request or a question. Such reports make up a substantial share of what projects receive (Fan et al., 2018; Herzig et al., 2013): a manual inspection of more than 7,000 reports across five open-source projects found that 33.8% of those filed as bugs were misclassified (Herzig et al., 2013). Rather than with a patch, many invalid reports are resolved by directing the reporter to adjust a setting, update to a version where the problem is already fixed, or disable a conflicting extension. Following Gön et al. (Gon et al., 2026), we call such a resolution a no-code fix. Triage already consumes a large fraction of maintenance effort, and invalid reports add to that burden without leading to a code change: in an industrial case study, Laiq et al. (Laiq et al., 2022) found that 14 to 17% of submitted reports were invalid, yet these took roughly 16 to 18 calendar days to resolve on average, about as long as valid reports despite requiring no code change. A maintainer must still read the report, reproduce or investigate the problem, determine that no code fix is needed, and explain the resolution to the reporter (Xuan et al., 2014; Gon et al., 2026). Gön et al. (Gon et al., 2026) introduced a benchmark of invalid Brave11 1 https://brave.com/ browser bug reports, subclassify them by root cause, and generate no-code fixes for them with several LLM configurations, the same benchmark and fixes we evaluate in this paper. Evaluating such fixes is hard. The only reference is the maintainer’s resolution, a free-form comment written for one reporter in one situation, so two fixes can be worded differently while doing the same thing, or worded almost identically while one points to a setting that does not exist. Their study therefore relies on text similarity and an LLM judge, the usual fallbacks when an output cannot be checked by running it (Torun et al., 2026b). Their BERTScore (Zhang et al., 2019) values fall within a narrow band of 0.81 to 0.82 across all models and configurations, and the LLM judge, while more discriminative, inherits the documented biases of LLM judges (Shi et al., 2025). Neither shows whether a fix would work if a user followed it (Section 5.2 revisits this on the fixes we execute). This is the gap we address. Execution offers an alternative. A code patch either passes the test suite or not, and Zhang et al. (Zhang et al., 2026b) have extended this principle to code review by turning review comments into executable tests. A no-code fix produces no code change, but its instructions can still be carried out: changing a setting or disabling an extension is an action in a real environment, after which the reported symptom either persists or disappears. Our goal is therefore to evaluate LLM-generated no-code fixes by whether they actually resolve the reported problem, not by whether they resemble a fix that did. Recent Computer-Use Agents (CUAs) make this process practical to automate, since they interpret natural-language instructions and act on a real interface, which is exactly what applying a no-code fix requires. Building on this observation, we design an automated, execution-based pipeline for evaluating no-code fixes. For each bug report, the pipeline first applies the maintainer’s fix and then rolls it back, which yields a verified fixed state and, from it, a verified buggy state; it then applies a candidate fix to the buggy state and checks whether the bug persists afterwards. Every candidate fix is applied by an executor agent directly from its natural-language text and verified by a deterministic, issue-specific checker; the bug-specific parts of the pipeline are confined to setting up each issue’s environment and its checker. We run it with three executors (Claude Sonnet 5 (Anthropic, 2026b), OpenCUA-72B (Wang et al., 2025), Meta Muse Glimmer (Meta Superintelligence Labs, 2026)) from different providers and validate the automated verdicts against humans performing the same procedure by hand. Concretely, we investigate three research questions: • RQ1: How effectively do agentic executors resolve LLM-generated no-code fixes? We measure the fix-generation capability of 12 LLM-based configurations by executing their fixes end-to-end, reporting resolution and applicability rates overall and broken down by bug subclass. • RQ2: To what extent does the choice of executor affect the measured fix capability? We run the same fixes through three executors from different providers to separate variance introduced by the executor from variance in the underlying fix quality, and to test whether conclusions about LLM performance are stable across executor choice. • RQ3: How accurately do agentic executors execute LLM-generated fixes compared to human execution? On a sampled subset of fixes, we compare each executor’s outcome against a human performing the same procedure by hand, establishing how far the automated verdicts can be trusted. This paper makes the following contributions: • We introduce a rollback-based, execution-driven pipeline for evaluating LLM-generated no-code fixes. Two sanity gates confirm that the bug is absent once the maintainer’s fix is applied and present once it is rolled back, so every candidate fix runs on a verified buggy state and is judged by whether it removes the bug. • We instantiate the pipeline with agentic executors that apply each natural-language fix directly, verified by issue-specific state check, and we compare three executors, two CUAs and one multimodal agentic LLM, to quantify the effect of executor choice. • We evaluate 322 individual no-code fixes (12 LLM-based configurations × three generation runs × nine reproducible bug reports, of which two generations are missing from the source benchmark), reporting per-configuration resolution and applicability rates at the level of individual fixes and their variation across bug subclasses. • We compare the automated verdicts with human execution of the same fixes, indicating how far each executor’s verdicts can be trusted and in which direction they diverge. 2. Related Work 2.1. Invalid Bug Reports: Classification and No-Code Fix Generation Not all issues submitted to a bug tracker call for a code change. Herzig et al. (Herzig et al., 2013) manually examined more than 7,000 issue reports across five open-source projects and found that 33.8% were misclassified: they did not point to a defect but requested a new feature, a documentation update, or an internal refactoring, introducing measurable bias into downstream bug-prediction studies. Distinguishing such reports from genuine defects has since become a task in its own right, studied extensively before the arrival of LLMs. Antoniol et al. (Antoniol et al., 2008) and Zanetti et al. (Zanetti et al., 2013) used text-based and social-network features, respectively, to separate genuine bugs from other change requests; Fan et al. (Fan et al., 2018), Otoom et al. (Otoom et al., 2019), and Pandey et al. (Pandey et al., 2017) trained SVM and random-forest classifiers for the same task, with Fan et al. identifying textual content and submitter experience as the strongest predictors. Meng and Visser (Meng and Visser, 2023) fine-tuned BERT to classify and explain bug report validity, and Laiq et al. (Laiq et al., 2025) compared classical and transformer-based models on a unified benchmark, finding fine-tuned RoBERTa to perform best. Du et al. (Du et al., 2024) propose LLM-BRC, a large-language-model–based framework for bug report classification; Aracena et al. (Aracena et al., 2024) and Colavito et al. (Colavito et al., 2025) likewise apply and benchmark LLMs for automated issue labeling. Closer to our setting, Dinç and Tüzün (Dinç and Tüzün, 2025) apply a range of LLM and machine-learning methods specifically to the valid/invalid distinction, characterizing where LLMs succeed and fail at this classification. Once a report is identified as invalid, resolving it typically requires no patch, only an explanation of a configuration change, a version update, or a workaround. This is what we call a no-code fix. Gön et al. (Gon et al., 2026) introduce a benchmark of invalid Brave browser issues, subclassify them by root cause, and generate no-code fixes for them using 12 LLM-based configurations (Section 3.2). We build directly on this benchmark (Section 3.1). Torun et al. (Torun et al., 2026a) situate this task within a broader vision for artificial intelligence (AI)-assisted bug tracking, in which incoming reports are automatically triaged as valid or invalid, invalid reports are resolved by agent-suggested no-code fixes, and valid ones proceed to patch generation and human review. Our work can be read as an empirical test of whether the no-code-fix step of that vision is achievable with current LLMs. Where prior work (Gon et al., 2026) in this space evaluates generated fixes by comparing them to a reference resolution, we evaluate by execution: applying each fix to a live environment and checking whether the reported bug persists. 2.2. Agent-Based Bug Reproduction and Fix Verification A separate line of work uses LLMs and agents to interact with running software. Kang et al. (Kang et al., 2024) prompt LLMs to generate bug-reproducing tests directly from natural-language bug reports and evaluate 15 models on the Defects4J benchmark, reproducing roughly one third of the included bugs; their technique, like later agentic approaches (Cheng et al., 2025; Guo et al., 2026), operates at the level of unit tests rather than a live GUI. Feng and Chen (Feng and Chen, 2024) instead target a live GUI directly: their AdbGPT system prompts an LLM to extract steps-to-reproduce from an Android bug report and replays them as GUI events on the device without any bug-specific scripting, closer in spirit to how our pipeline executes steps-to-reproduce in a browser. Closer to our environment, Yapağcı et al. (Yapağcı et al., 2025) introduce BugCraft, which uses an LLM to synthesize steps-to-reproduce from a Minecraft crash report and a vision-based agent to execute them in the game, reproducing 34.9% of crash bugs end-to-end and outperforming baseline computer-use agents by 37%; our sanity checks follow the same execute-steps-and-observe-the-outcome principle, applied to a browser instead of a game. Chen et al. (Chen et al., 2025) take a complementary retrieval-augmented approach, using bug reports from similar Android apps to guide GUI exploration toward paths likely to trigger a bug, again treating a bug report as an executable specification for an agent. The work closest to ours is Fixpad++ (İr et al., 2026), which verifies whether a candidate patch fixes a reported crash in the GUI-based Notepad++ editor: a multi-agent system reproduces the bug via an extended ReAct loop with self-reflection, then replays the same trajectory on the patched build to judge fix correctness. Fixpad++ and our pipeline share the core idea of verifying a fix by re-executing a reproduction trajectory rather than by comparing text, but differ in what is being verified and how: Fixpad++ judges source-code patches with a purpose-built multi-agent architecture tuned to a single application. We instead judge natural-language no-code fixes with off-the-shelf agentic executors, both open-weight and proprietary, and treat the choice of executor itself as an independent variable (RQ2). Agent-based verification of GUI artifacts has also been studied outside the bug-fixing setting. Kolthoff et al. (Kolthoff et al., 2025) introduce GUISpector, which uses a multi-modal LLM agent, built on a commercial CUA, to autonomously verify whether a GUI prototype satisfies natural-language requirements and to report which acceptance criteria are met or violated; unlike Fixpad++ and our pipeline, GUISpector checks conformance to a specification, not whether a previously observed bug has disappeared. Closer to our concern with executor reliability, Hong et al. (Hong et al., 2026) show that GUI-agent evaluators frequently misattribute their own execution failures to genuine software defects, and propose reusing a failed trajectory to disambiguate the two; this distinction between evaluator error and actual software defect is precisely what our sanity checks and our RQ3 human-comparison are designed to control for in our own pipeline. Taken together, this line of work reproduces bugs in unit-test, game, and mobile-GUI settings (Kang et al., 2024; Cheng et al., 2025; Guo et al., 2026; Yapağcı et al., 2025; Chen et al., 2025; Feng and Chen, 2024), verifies fixes with a bespoke agent tied to a single desktop application (İr et al., 2026), or verifies specification conformance rather than bug resolution (Kolthoff et al., 2025). None evaluates no-code fixes specifically, and none treats the choice of executing agent as an experimental variable, which is the focus of our RQ2. 2.3. LLM Evaluation in Software Engineering Evaluating LLM-based software engineering tools reliably is difficult in its own right. Torun et al. (Torun et al., 2026b) survey current evaluation practices and argue that the open-endedness and non-determinism of LLM outputs undermine the single-ground-truth, deterministic assumptions that conventional SE evaluation relies on. One increasingly common workaround, the LLM-as-a-Judge paradigm, used by Gön et al. (Gon et al., 2026) alongside BERTScore to grade generated no-code fixes, is itself the subject of active scrutiny: He et al. (He et al., 2025) survey LLM-as-a-Judge practice in software engineering and identify a lack of validation against ground truth as one of the paradigm’s central open problems. Unlike execution-free critics (Yadavally et al., 2025), execution-based evaluation sidesteps this problem directly, and it has an established track record for LLM-generated code. SWE-bench (Jimenez et al., 2024) and SWE-bench-Live (Zhang et al., 2026a) evaluate whether a model’s patch to a real codebase makes a GitHub issue’s associated tests pass, instead of scoring the patch’s similarity to a reference diff. The same execution-based principle has recently been extended from source code to the GUI: OSWorld (Xie et al., 2024) scores agents on 369 real desktop and web tasks by checking the resulting operating-system state against a task-specific verification script, and WebArena (Zhou et al., 2024) evaluates web agents on self-hosted, fully functional websites by checking the functional correctness of the completed task. Our pipeline borrows this state-based checking but inverts the role of the agent: in OSWorld and WebArena the agent is the subject under evaluation, whereas here the agentic executor is part of the measurement instrument and the subject is the LLM-generated no-code fix it applies. This is why we treat the executor as an experimental variable (RQ2) and validate it against human execution (RQ3). 3. Methodology This section describes the dataset we work with, the rollback-based evaluation pipeline shown in Figure 1, the use of three agentic executors as pipeline executors, and the metrics we report. 3.1. Dataset We reuse the benchmark released by Gön et al. (Gon et al., 2026), which contains 302 annotated bug reports drawn from the Brave browser GitHub issue tracker. Each report is annotated with an invalid-bug-report subclass and, where available, a link to the maintainer comment containing the successful no-code fix. Of the 302 reports, 188 fall into one of seven invalid subclasses, while the remaining 114 are outliers that the original annotation placed outside the taxonomy (reports found to be valid, internal developer-workflow items, or cases with too little information to classify). The benchmark also includes no-code fixes generated by 12 LLM-based configurations for each bug report, spanning four pipeline variants and six backbone LLMs; we describe these configurations in detail in Section 3.2. Each configuration was executed three times in the original study to account for LLM stochasticity, yielding 36 candidate no-code fixes per bug report (12 configurations × three runs), all of which we evaluate independently in our pipeline. Not every subclass in the benchmark is suitable for execution-based evaluation. Fixes for the Working as Designed (48 reports), Feature Request (41), Non-Reproducible (35), and Question (13) subclasses do not correspond to actions that can be applied and verified in a browser environment: they are explanations, requests for information, or acknowledgments none of which changes the application state. We therefore select the 51 bug reports in the three subclasses whose maintainer fix takes the form of a concrete action on the browser or its environment: Faulty Configuration & Environment Setup (19), Wrong Version (Already Fixed) (12), and External System & Dependency Issues (20). 3.2. LLM-Based Fix-Generation Configurations For each bug report, the benchmark provides no-code fixes generated by 12 distinct configurations, spanning four pipeline variants and six backbone LLMs. Table 1 lists all 12 configurations together with the short identifiers we use to refer to them for the remainder of the paper. Vanilla. The baseline pipeline prompts a single backbone LLM with the bug report and its predicted invalid-subclass label, and is instantiated with six backbone models: three proprietary (Gemini 3.1 Pro Preview (Google, 2026), GPT-5.4 (OpenAI, 2026), and Claude Opus 4.6 (Anthropic, 2026a)) and three open-weight (GLM-5 (Zeng et al., 2026), MiniMax M2.7 (MiniMax, 2026), and Kimi K2.5 (Team et al., 2026)). Ablation (No-Prior). Identical to Vanilla but withholds the invalid-subclass prior from the prompt, isolating its contribution to fix quality. Instantiated with the best-performing proprietary and open-weight backbones from the Vanilla setting (Gemini 3.1 Pro and Kimi K2.5). RAG. Retrieves from two corpora, Brave Browser wiki pages and historical invalid bug reports, embedded with the Zembed-1 model22 2 https://huggingface.co/zeroentropy/zembed-1 and indexed in LanceDB33 3 https://w.lancedb.com/. For each query issue, the pipeline retrieves the top-20 nearest neighbors from each corpus via dense vector search, discards candidates created after the query issue (chronological filtering, to prevent data leakage), and re-ranks the remaining candidates with the Zerank-244 4 https://huggingface.co/zeroentropy/zerank-2 cross-encoder to select the top-5 contexts, which are appended to the prompt. Instantiated with the same two backbones as the ablation setting (Gemini 3.1 Pro and Kimi K2.5). Agentic Web Search. Follows a WebThinker-style “Think-Search-and-Draft” architecture: the backbone LLM first drafts a research brief, then a bounded orchestration loop issues queries against the Serper Search API. The canonical original-issue page is excluded from results to prevent leakage, and the content of any visited page is compressed before being returned. Retrieved evidence is passed back to the LLM in chronological order before it produces the final classification and fix. Instantiated with Gemini 3.1 Pro Preview (paired with Gemini 2.5 Flash for the search loop) and Kimi K2.5. Table 1. The 12 LLM-based fix-generation configurations reused from Gön et al. (Gon et al., 2026), with the short identifiers used throughout this paper. Rows are pipelines and columns are backbone LLMs; a dash marks a combination that was not run. Backbone LLM Pipeline Gemini 3.1 Pro Preview GPT-5.4 Claude Opus 4.6 GLM-5 MiniMax M2.7 Kimi K2.5 Vanilla V-Gemini V-GPT V-Claude V-GLM V-MiniMax V-Kimi Ablation (no subclass prior) Abl-Gemini – – – – Abl-Kimi RAG RAG-Gemini – – – – RAG-Kimi Agentic web search Agentic-Gemini – – – – Agentic-Kimi Figure 1. Overview of the evaluation pipeline.Flow diagram: a bug report, the maintainer's fix and 12 LLM fixes enter a Docker environment; two sanity checks bracket a rollback from fixed to buggy state; each LLM fix is applied by an executor and checked. 3.3. Pipeline Overview For each of the 51 selected bug reports, evaluation proceeds through the pipeline shown in Figure 1. The pipeline takes three inputs on the left: the bug report itself (including its steps-to-reproduce and stated actual result), the maintainer’s ground-truth fix, and the 12 candidate LLM-generated fixes. It produces a per-LLM resolution rate at the right, indicating how often each LLM’s fix resolved the reported bug. The main flow is bracketed by two sanity checks and separated by an explicit state rollback. A further branch, based on whether the ground-truth fix is a version update or a GUI-level change, determines how that fix is applied and rolled back. We describe each stage in turn, following Figure 1 left to right. Fresh environment and ground-truth application. For each bug report, we classify the maintainer’s ground-truth fix as version-type (the bug is resolved simply by moving to a newer Brave release) or GUI-type (the bug is resolved by a configuration change, extension toggle, or other in-browser action). This classification follows the content of the maintainer’s resolution (GT fix: version-type? in Figure 1). The Wrong Version issue is version-type by definition, and so are two external system issues whose maintainer resolved the report by pointing to a later release that contains an upstream Chromium fix; the remaining six issues are GUI-type. For a version-type fix, we provision a Docker container already running the newer, fixed Brave release (Docker: newer / fixed version). For a GUI-type fix, we provision a container running the version originally reported by the issue reporter (Docker (reported version)), and we apply the maintainer’s fix to it through the browser GUI. Either path yields the fixed state (green ellipse in Figure 1). Fixed-state sanity check. Before rolling anything back, we verify that the fixed state is genuinely fixed: the steps-to-reproduce from the bug report are executed and checked against the report’s stated actual result (Is bug reproduced? in Figure 1). If the actual result is not observed, this check passes and the pipeline continues to the rollback stage. If it is observed, the ground-truth fix as applied does not resolve the bug in our environment; the issue is dropped (Drop issue) and no further evaluation is attempted for it. Rollback to the buggy state. Once this check passes, the ground-truth fix is rolled back, branching again on whether it was version-type or GUI-type. For a version-type fix, we switch the Docker container back to the originally reported version (Switch Docker: back to reported version). For a GUI-type fix, a scripted sequence of GUI actions reverts the fix without changing the container version (Revert fix (GUI)). Either path returns the environment to the configuration that exhibited the bug before the fix, which we call the buggy state (red ellipse in Figure 1). The rollback confirms that the state the checker reads in the buggy environment corresponds to the reported symptom. Buggy-state sanity check. We now verify that the rollback actually produced the reported bug: the steps-to-reproduce are executed again and checked against the stated actual result (the second Is bug reproduced? check in Figure 1). If the actual result is observed, this check passes, and the issue proceeds to per-LLM evaluation. If not, the rollback failed to reproduce the bug, and the issue is dropped (Drop issue). Together, these two checks ensure that every issue carried into the LLM evaluation stage has a genuinely fixed reference state and a genuinely buggy state derived from it. Of the 51 selected issues, 38 could not be set up in our environment at all, for example because they were specific to a platform other than Linux or required a live backend service, and four did not exhibit the reported bug when tested; the remaining nine passed both checks and formed the effective evaluation set. Per-LLM fix application and verification. For each of the 36 candidate fixes (12 configurations, three independently generated runs each), every episode starts from a fresh container in the buggy state verified above (Reset to buggy state): for GUI-type issues, a fresh profile on the gate-verified build, with the buggy value pre-seeded where the bug is not Brave’s default state; for the version-type issues, the originally reported build. The executor then applies the LLM’s fix through the browser GUI (Executor applies fix (GUI)). On the version-type issues, a fix may instead call for an update: when the executor opens Brave’s update page (brave://settings/help), the harness switches the container to the fixed build (Switch Docker in Figure 1), which is what a user would obtain by updating. Because the switch is triggered by the executor’s own action, resolution rates on these issues can differ across executors. After the episode, an issue-specific checker reads the resulting browser state and decides whether the reported actual result is still present; we additionally replay the steps-to-reproduce and keep the screenshots as a visual record (Re-run steps-to-reproduce & verify). This verdict is produced by the checker (Section 3.4). Each attempt therefore yields two independent signals: the checker’s verdict on whether the bug persists, and the executor’s own termination status (DONE, BLOCKED, step budget exhausted, or stopped without a verdict), which feed the metrics in Section 3.6. 3.4. Executors and Checkers Executors are used in the per-fix stage: every candidate fix is applied by an agentic executor instead of a bug-specific script, which is what allows the pipeline to consume the natural-language fixes directly. The sanity gates and the rollback (Section 3.3) are scripted, so that the validity of each issue’s environment does not depend on the agents under study. To isolate variance introduced by the choice of executor from variance in the underlying LLM fix quality, we run the per-fix stage once with each of three executors. The three executors we compare are OpenCUA-72B (Wang et al., 2025), an open-weight model trained for computer use; Meta’s Muse Glimmer (Meta Superintelligence Labs, 2026), an open-weight general-purpose agentic model; and Anthropic’s Claude Sonnet 5 (Anthropic, 2026b) with its computer-use tool (bottom left side of Figure 1). All three act on the same browser environment through a shared action interface. OpenCUA and Sonnet 5 use the action formats defined by their developers, while Muse Glimmer, which has no official computer-use harness, is driven through a function-calling schema we designed; implementation details are provided in our replication package (Section 8). All three executors receive the same task prompt, containing the bug report and the fix text, and act for at most 20 steps per episode. Sonnet 5 uses its default sampling settings, OpenCUA-72B greedy decoding (temperature 0) as in its reference evaluation code, and Muse Glimmer temperature 1.0 and top-p 0.95 as recommended by its model card55 5 https://huggingface.co/meta-models/Muse-Glimmer-30B. Checker. The executor applies a fix but does not decide whether the fix worked. For each issue, we implement a deterministic checker that reads the browser state tied to the reported symptom, such as a value in the browser’s preference files or a property of the rendered page queried through the Chrome DevTools Protocol, and reports whether the reported actual result is still present. The executor’s own claim of success is recorded but never used as the verdict. The checkers are the only issue-specific code in the pipeline and are part of our replication package. Repetition and aggregation. Because executor behavior is stochastic, each fix is executed three times by each executor. We count a fix as resolved under an executor if the checker reports the bug as absent in at least one of the three runs. The fix text is identical across runs, so if the checker finds the bug absent after any run, following the fix’s instructions can remove the bug; a run in which the same fix fails may instead reflect an executor error. We therefore treat one successful run as sufficient, an assumption that RQ3 tests against human execution. We count a fix as non-applicable under an executor if the executor declared it BLOCKED in at least two of the three runs. 3.5. Human Execution for Executor Accuracy The pipeline’s verdicts depend on the executors correctly applying LLM-generated fixes and correctly observing whether the reported bug persists afterwards. To measure how reliably they do this, we compare the executors against humans performing the same task manually. We draw a simple random sample of 75 of the 322 individual fixes in the evaluation set, using a fixed random seed. This sample size yields a 95% confidence level with a margin of error of ±10%± 10\% for proportions estimated over the population of 322 fixes, following Cochran’s formula with finite population correction (Cochran, 1977) and assuming maximum variability (p=0.5p=0.5). For each sampled fix, the buggy state is reproduced identically to how the executors experience it: a fresh Brave environment with the ground-truth fix rolled back. A human then executes the LLM’s fix on this environment literally, performing the actions the fix describes without judging their correctness. The human then runs the steps-to-reproduce and records whether the reported actual result is observed. Executor assignment and blinding. The human executors are the first two authors. Each author independently executes all 75 sampled fixes, which allows us to measure inter-rater agreement between the two executors. Before executing a fix, an author sees neither the configuration that generated it nor any executor’s outcome for it. Because the browser state determines the outcome, the comparison does not rest on subjective agreement. If the executor and the human executed the same fix on the same starting state, both should in principle arrive at the same outcome. A divergence may reflect an executor or checker error, an ambiguous fix that admits more than one reasonable reading, or an imperfect human execution, so we treat the human outcome as a reference rather than a ground truth. We compare each executor’s resolved verdict with the human outcome and report the percentage of matching fixes, with 95% Wilson (Wilson, 1927) confidence intervals, as the accuracy of that executor. 3.6. Metrics We report the following metrics computed over the outcomes produced by the pipeline: Resolution rate: For each (configuration, executor) pair, the fraction of the configuration’s individual fixes (27 fixes for 11 configurations and 25 for Agentic-Kimi) that the executor resolved, as defined in Section 3.4. Aggregated across all 12 configurations, this yields a resolution rate over the 322 fixes per executor. This fix-level rate, rather than an issue-level one, is the primary measure of fix quality in this paper. Applicability rate: The fraction of fixes that the executor did not declare BLOCKED in the majority of its runs, regardless of whether the bug was ultimately resolved. This separates fixes the executor judged non-actionable (advice, requests for information, deferrals) from applied fixes that failed to resolve the bug. Because it relies on the executor’s judgment, applicability is reported per executor. Subclass breakdown: Fix-level resolution and applicability rates, computed over the relevant subset of the 322 fixes, broken down by the three subclasses in the evaluation set: Faulty Configuration & Environment Setup, Wrong Version (Already Fixed), and External System & Dependency Issues. For RQ2, we additionally report per-LLM resolution rates side by side across the three executors, and characterize the spread as a measure of executor sensitivity. For RQ3, we compare each executor’s aggregated verdict with the human-executed outcome on the random sample described in Section 3.5, reporting the percentage of matching outcomes with 95% Wilson intervals, Cohen’s κ (Cohen, 1960) on the consensus set, and the direction of mismatches. 3.7. Use of generative AI We used Claude Sonnet 5 (Anthropic, 2026b) to assist in writing the evaluation harness (container orchestration, executor adapters, and the Muse Glimmer function-calling schema), the issue-specific checkers, the human-evaluation tool, the analysis scripts, and the documentation of the replication package. The authors reviewed this code, verified every checker and scripted gate action against a live container before use. 4. Results 4.1. RQ1: Fix-Generation Capability of LLMs Overall. Table 2 summarizes resolution and applicability per executor. Across the 322 fixes, the resolution rate was 49.7% under Sonnet 5, 45.7% under OpenCUA-72B, and 14.6% under Muse Glimmer. Applicability was 66.1%, 99.1%, and 70.5%, respectively. To separate fixes that proposed no executable action from fixes that were applied but did not remove the bug, we also compute the resolution rate among applicable fixes only, that is, the share of applicable fixes that resolved the bug. This rate was 69.5% under Sonnet 5, 45.8% under OpenCUA-72B, and 18.1% under Muse Glimmer. Table 2. Resolution and applicability per executor over the 322 fixes (fix counts in parentheses). The last column restricts the resolution rate to the fixes the executor found applicable. Executor Resolution (%) Applicability (%) Resolution among applicable (%) Claude Sonnet 5 49.7 (160) 66.1 (213) 69.5 (148) OpenCUA-72B 45.7 (147) 99.1 (319) 45.8 (146) Muse Glimmer 14.6 (47) 70.5 (227) 18.1 (41) Per configuration. Table 3 reports resolution and applicability for each of the 12 configurations. V-Claude had the highest resolution rate under Sonnet 5 (74.1%), but not under the other two executors: under OpenCUA-72B, RAG-Kimi and Agentic-Gemini led (55.6%) and V-Claude was third (51.9%, tied with V-Gemini), and under Muse Glimmer, Agentic-Kimi led (28.0%) and V-Claude tied for second (25.9%). The lowest rate was 25.9% under Sonnet 5 (V-MiniMax), 33.3% under OpenCUA-72B (V-Kimi), and 0.0% under Muse Glimmer (V-Kimi). OpenCUA-72B declared at most one fix per configuration non-applicable, whereas under Sonnet 5 applicability ranged from 44.4% (V-MiniMax) to 88.9% (V-Claude). Agentic-Kimi has 25 fixes instead of 27 because its second generation run for issues 25057 and 45912 is absent from the source benchmark. Table 3. Resolution (Res.) and applicability (App.) rate (%) per configuration and executor. Each configuration has 27 fixes (Agentic-Kimi: 25). Backbones: Gem. = Gemini 3.1 Pro, Cla. = Claude Opus 4.6, MiniM. = MiniMax M2.7. Executors: S5 = Sonnet 5, OC = OpenCUA-72B, MG = Muse Glimmer. Vanilla Ablation RAG Agentic Gem. GPT Cla. GLM MiniM. Kimi Gem. Kimi Gem. Kimi Gem. Kimi Res. S5 44.4 40.7 74.1 44.4 25.9 44.4 59.3 51.9 48.1 44.4 63.0 56.0 OC 51.9 48.1 51.9 37.0 44.4 33.3 44.4 40.7 48.1 55.6 55.6 36.0 MG 11.1 11.1 25.9 18.5 11.1 0.0 14.8 11.1 7.4 11.1 25.9 28.0 App. S5 55.6 55.6 88.9 51.9 44.4 66.7 70.4 74.1 63.0 70.4 81.5 72.0 OC 100.0 96.3 100.0 100.0 100.0 100.0 100.0 96.3 96.3 100.0 100.0 100.0 MG 63.0 63.0 81.5 70.4 63.0 81.5 66.7 77.8 70.4 70.4 74.1 64.0 Pipeline variants. Gemini 3.1 Pro and Kimi K2.5 are the only backbones instantiated in all four pipelines (Section 3.2), so they allow comparing pipelines while holding the backbone fixed. With Gemini, Agentic had the highest resolution rate under both Sonnet 5 (63.0%) and OpenCUA-72B (55.6%); the lowest was Vanilla under Sonnet 5 (44.4%) and Ablation under OpenCUA-72B (44.4%). With Kimi, Agentic had the highest rate under Sonnet 5 (56.0%) and RAG under OpenCUA-72B (55.6%); no pipeline resolved fewer fixes than Vanilla (44.4% under Sonnet 5, 33.3% under OpenCUA-72B). Agentic also had the highest rate under Muse Glimmer for both backbones (25.9% and 28.0%). Per subclass. Table 4 breaks the results down by subclass. The subclass with the highest resolution rate differed by executor: External System & Dependency under Sonnet 5 (59.3%) and Muse Glimmer (21.3%), and Faulty Configuration under OpenCUA-72B (65.7%). Faulty Configuration fixes reached only 13.5% under Muse Glimmer. The single Wrong Version issue had the lowest resolution rate under OpenCUA-72B (8.3%) and Muse Glimmer (0.0%), but not under Sonnet 5 (50.0%), which completes the update and the gradient selection that this issue requires. Applicability was lowest for the Wrong Version issue under Sonnet 5 (44.4%) and for External System under Muse Glimmer (67.6%), and close to 100% for OpenCUA-72B in every subclass. Table 4. Resolution and applicability rate (%) per subclass. Sonnet 5 OpenCUA-72B Muse Glimmer Subclass Iss. Fixes Res. App. Res. App. Res. App. Faulty Configuration 5 178 43.8 66.3 65.7 98.9 13.5 69.7 Wrong Version 1 36 50.0 44.4 8.3 100.0 0.0 83.3 External System 3 108 59.3 73.1 25.0 99.1 21.3 67.6 Answer to RQ1. Depending on the executor, 14.6% to 49.7% of the 322 LLM-generated no-code fixes resolved the reported bug, and 66.1% to 99.1% were applicable. V-Claude had the highest resolution rate under Sonnet 5 (74.1%), but ranked third under OpenCUA-72B, where RAG-Kimi and Agentic-Gemini tied for first, and second under Muse Glimmer, where Agentic-Kimi led. The weakest configuration and the best-resolved subclass also depended on the executor. The Wrong Version issue was resolved least often under OpenCUA-72B and Muse Glimmer, but not under Sonnet 5. 4.2. RQ2: Sensitivity to the Choice of Executor Run-to-run stability. Table 5 reports the pass counts of each individual executor run and the number of fixes resolved in at least one run. Within an executor, the pass rate varied by at most 3.4% across runs (Sonnet 5, 38.8% to 42.2%), while Muse Glimmer trailed the other two executors by at least 20.5 points in every run. All three runs reached the same verdict for 92.2% of fixes under Muse Glimmer, 82.3% under Sonnet 5, and 73.9% under OpenCUA-72B. Counting a fix as resolved if any run passed added 24 fixes over the best single run for Sonnet 5 (160 against 136), 38 for OpenCUA-72B (147 against 109), and 12 for Muse Glimmer (47 against 35). Under a majority vote the rates were 39.8%, 30.7%, and 9.9%, so Sonnet 5 leads OpenCUA-72B under both aggregation rules. Table 5. Resolved fixes per individual executor run, fixes resolved in at least one run, and the share of fixes on which all three runs agreed (n=322n=322). Executor Run 1 Run 2 Run 3 Any run All runs agree Claude Sonnet 5 136 (42.2%) 125 (38.8%) 130 (40.4%) 160 (49.7%) 82.3% OpenCUA-72B 100 (31.1%) 109 (33.9%) 100 (31.1%) 147 (45.7%) 73.9% Muse Glimmer 34 (10.6%) 35 (10.9%) 32 (9.9%) 47 (14.6%) 92.2% Agreement between executors. All three executors reached the same verdict on 46.9% of fixes. Sonnet 5 and OpenCUA-72B agreed on 62.4% of fixes (Cohen’s κ=0.25κ=0.25), Sonnet 5 and Muse Glimmer on 64.3% (κ=0.28κ=0.28), and OpenCUA-72B and Muse Glimmer on 67.1% (κ=0.30κ=0.30). The ranking of the 12 configurations was only weakly preserved: Kendall’s τb _b was 0.13 between Sonnet 5 and OpenCUA-72B, 0.45 between Sonnet 5 and Muse Glimmer, and 0.05 between OpenCUA-72B and Muse Glimmer. For a single configuration, the resolution rate differed across executors by 38.8% on average (range 25.9 to 48.1; 95% bootstrap interval 31.2 to 52.1). The disagreement is not concentrated on the three version-dependent issues, where the outcome depends on whether the executor performs the update: without them, agreement drops further (κ = 0.22, 0.24, and 0.17), and all three executors agree on only 40.7% of the fixes. Per-issue differences. Table 6 lists resolved fixes per issue. On the three version-dependent issues, Sonnet 5 resolved the most fixes (16, 18, and 29 for 22252, 25379, and 31108), and the executors no longer resolved the same number of fixes. On 25379, OpenCUA-72B resolved 3 fixes and Muse Glimmer none, because an update alone is not enough there and a gradient must be chosen afterwards. On issue 24747, Sonnet 5 resolved 19 fixes, while neither OpenCUA-72B nor Muse Glimmer resolved any. On issue 37372, OpenCUA-72B resolved all 36 fixes against 20 for Sonnet 5 and 12 for Muse Glimmer, and on 20537 and 25057 it resolved roughly twice as many fixes as Sonnet 5. Muse Glimmer resolved no fix for issues 24747, 25057, 25379, and 45912. Table 6. Resolved fixes per issue and executor. Column headers are Brave GitHub issue numbers, grouped by subclass. Faulty Configuration Wrong Version External System Issue 20537 22987 25057 37372 45912 25379 22252 24747 31108 Total Fixes 36 36 35 36 35 36 36 36 36 322 Sonnet 5 9 13 10 20 26 18 16 19 29 160 OpenCUA-72B 19 20 19 36 23 3 12 0 15 147 Muse Glimmer 3 9 0 12 0 0 10 0 13 47 Episode termination. Each episode ends with the executor’s own verdict, which is independent of the checker. Sonnet 5 ended 83.0% of its 966 episodes with an explicit verdict (50.2% DONE, 32.8% BLOCKED), OpenCUA-72B 59.7% (55.1% DONE, 4.7% BLOCKED), and Muse Glimmer 44.9% (13.4% DONE, 31.6% BLOCKED); nearly all remaining episodes exhausted the 20-step budget (16.3%, 40.2%, and 55.1%). Across all executors, the checker recorded the bug as resolved in 621 of the 1,146 episodes ending in DONE (54.2%), 162 of the 1,077 episodes ending at the step budget (15.0%), and 18 of the 667 episodes ending in BLOCKED (2.7%). Answer to RQ2. The choice of executor changed the measured fix capability substantially: overall resolution ranged from 14.6% to 49.7%, the rate of a single configuration differed across executors by 38.8% on average, and the executors reached the same verdict on only 46.9% of the fixes. Each executor was stable across its own runs (at most 3.4 points), so the differences between executors exceeded their run-to-run variation. Configuration rankings were only weakly preserved (τb _b between 0.05 and 0.45), and V-Claude, first under Sonnet 5, ranked third under OpenCUA-72B and second under Muse Glimmer. 4.3. RQ3: Agreement with Human Execution The two authors independently executed all 75 randomly sampled fixes and reached the same outcome on 78.7% of them (59 fixes, Cohen’s κ=0.54κ=0.54). Table 7 reports, for each executor, the percentage of sampled fixes on which its verdict matched the outcome of each human executor and of the consensus set, with 95% Wilson confidence intervals. Claude Sonnet 5 matched the consensus outcome on 88.1% of fixes (κ=0.74κ=0.74), Muse Glimmer on 79.7% (κ=0.48κ=0.48), and OpenCUA-72B on 66.1% (κ=0.28κ=0.28). The mismatches differed in direction. Of the 20 consensus mismatches of OpenCUA-72B, 13 were fixes it counted as resolved that the humans did not. Of the 12 mismatches of Muse Glimmer, 10 were fixes it did not resolve although the humans did, and Sonnet 5 erred in both directions (5 false positives and 2 false negatives). On the 75 sampled fixes, OpenCUA-72B resolved 33, Muse Glimmer 13, and Sonnet 5 33, against 25 and 29 for the two humans. Replacing “any run” with a majority vote over the three runs changed the agreement with the consensus to 86.4% for Sonnet 5, 72.9% for OpenCUA-72B, and 81.4% for Muse Glimmer, so the aggregation rule did not alter the ordering. The human outcomes differed on the remaining 16 fixes, which are therefore not part of the consensus set. Table 7. Agreement (%) between each executor and human execution on the 75 randomly sampled fixes, with 95% Wilson confidence intervals. Consensus: the 59 fixes on which both humans reached the same outcome. Executor Human 1 Human 2 Consensus κ (consensus) Claude Sonnet 5 81.3 [71.1, 88.5] 78.7 [68.1, 86.4] 88.1 [77.5, 94.1] 0.74 OpenCUA-72B 68.0 [56.8, 77.5] 57.3 [46.1, 67.9] 66.1 [53.4, 76.9] 0.28 Muse Glimmer 78.7 [68.1, 86.4] 68.0 [56.8, 77.5] 79.7 [67.7, 88.0] 0.48 Human 1 vs. Human 2 78.7 [68.1, 86.4] 0.54 Answer to RQ3. The authors agreed on 78.7% of the 75 sampled fixes (κ=0.54κ=0.54). On the fixes with a shared human outcome, Sonnet 5 matched on 88.1% (κ=0.74κ=0.74), Muse Glimmer on 79.7% (κ=0.48κ=0.48), and OpenCUA-72B on 66.1% (κ=0.28κ=0.28). OpenCUA-72B and Sonnet 5 overstated and Muse Glimmer understated the number of resolved fixes; Sonnet 5 had both the highest resolution rate and the highest agreement with the humans. 5. Discussion 5.1. The Executor Is Part of the Measurement RQ2 shows that an executor-based evaluation does not measure fix quality in isolation. The same 322 fixes produced resolution rates between 14.6% and 49.7%, and the rate of a single configuration moved by 38.8% on average when only the executor changed. These gaps are much larger than the run-to-run variation within any executor (at most 3.4 points), so they reflect systematic properties of the executors rather than noise. Conclusions about which LLM produces better no-code fixes are therefore conditional on the executor. Even the strongest configuration was not stable: V-Claude led under Sonnet 5 but ranked third under OpenCUA-72B and second under Muse Glimmer, and the rankings of the 12 configurations were nearly unrelated between Sonnet 5 and OpenCUA-72B (τb=0.13 _b=0.13) and between OpenCUA-72B and Muse Glimmer (τb=0.05 _b=0.05). Some comparisons changed substantially: RAG-Kimi resolved 22.3% more fixes than V-Kimi under OpenCUA-72B (55.6% against 33.3%) and exactly as many under Sonnet 5. The executors also differed in where they succeeded. OpenCUA-72B resolved every fix for issue 3737266 6 https://github.com/brave/brave-browser/issues/37372, whose checker is lenient (Section 6), while Sonnet 5 resolved 20 of 36, whereas Sonnet 5 was the only executor to resolve any fix for issue 2474777 7 https://github.com/brave/brave-browser/issues/24747. Muse Glimmer exhausted its step budget in 55.1% of its episodes and resolved no fix for four of the nine issues. No single executor is thus a neutral instrument, and a verdict of “fix failed” can mean that the executor failed. We draw three practical implications for execution-based evaluation with agentic executors. First, the executor and its harness should be reported as part of the experimental setup, in the same way as the model under evaluation. Second, results should be obtained with more than one executor or calibrated against human execution, as we do in RQ3. Third, run-level results and the aggregation rule should be reported, because the rule interacts with executor stability: counting a fix as resolved if any run passed added 38 fixes over the best single run for OpenCUA-72B, the least stable executor, but only 12 to 24 for the other two. RQ3 indicates which of the three executors agrees best with human execution. Sonnet 5, which also had the highest resolution rate (49.7%), agreed most with the humans (88.1% with the consensus, κ=0.74κ=0.74), although it resolved more fixes than the humans on the sample (33 against 25 and 29). OpenCUA-72B had the lowest agreement (66.1%, κ=0.28κ=0.28), and most of its mismatches were false positives (13 of 20). Muse Glimmer mismatched in the opposite direction (10 of its 12 mismatches were fixes it did not resolve although the humans did), which is consistent with the 55.1% of its episodes that exhausted the step budget. Rates under OpenCUA-72B and, to a lesser extent, Sonnet 5 are therefore likely to overstate, and those under Muse Glimmer to understate, the share of fixes that work when a human applies them. The executors’ termination status also shows why the verdict should not be left to the executor. An executor reports DONE when it believes it has applied the fix, not when the bug is gone, and the two diverge in both directions: the checker found the bug resolved in only 621 of the 1,146 episodes that ended in DONE, while 162 of the 1,077 episodes that ran out of steps nevertheless left the browser in a resolved state. On the version-dependent issues alone, Sonnet 5, OpenCUA-72B, and Muse Glimmer said DONE while the checker failed in 7, 121, and 1 of 324 episodes, and the checker passed without a DONE report in 54, 38, and 26. Using the agent’s own completion signal as the outcome would therefore both inflate and deflate the measured capability, which is consistent with the observation of Hong et al. (Hong et al., 2026) that GUI agents often misattribute their own execution failures. Separating the executor from an independent checker keeps the verdict reproducible, at the cost of per-issue setup and checker code (Section 6). 5.2. What Text Similarity Misses As argued in Section 1 reference-based metrics cannot show whether a no-code fix works. Our execution results allow a first look at this claim. For each LLM-generated fix, we took the BERTScore (Zhang et al., 2019) value against the maintainer’s ground-truth fix released in the replication package of Gön et al. (Gon et al., 2026), and examined the 108 fixes that failed in all nine attempts, that is, in every run under every executor. Because no executor resolved these fixes, their failure does not depend on the executor sensitivity discussed in Section 5.1. These fixes received BERTScore F1 values between 0.757 and 0.867 (median 0.831), within or above the range of 0.81 to 0.82 that Gön et al. (Gon et al., 2026) report for fixes across their benchmark. A fix that never removed the reported symptom was thus scored as close to the maintainer’s resolution as a typical fix. The 214 fixes that at least one executor resolved scored the same (median 0.832, mean 0.824 against 0.823). Scores also varied far more between issues than within them. Fixes for issue 25379 averaged 0.762, while those for every other issue with such fixes averaged between 0.818 and 0.846 (issue 37372 has none, because OpenCUA-72B resolved all its fixes), which suggests that the score reflects the wording of the reference resolution more than the correctness of the fix. A single BERTScore threshold therefore cannot separate working from failing fixes across issues, and a comparison of configurations by BERTScore partly reflects which issues their fixes belong to. Execution-based evaluation does not replace text-based evaluation everywhere. It applies only to reports whose bug can be re-established in a controlled environment (Section 5.4), whereas BERTScore and an LLM judge apply to any report with a reference fix. For reports that can be executed, however, these fixes show that a high similarity to the maintainer’s resolution does not indicate that a fix works. 5.3. What the Results Say About No-Code Fix Generation Even under the executor with the highest resolution rate and the highest agreement with the humans (Sonnet 5, 49.7%), just under half of the LLM-generated fixes resolved the reported bug (39.8% under a majority vote), and the best configuration reached 74.1%. Automatically generated no-code fixes are therefore not yet reliable enough to be sent to reporters without review, although the best configurations resolve a majority of cases under two of the three executors. No configuration led under every executor. V-Claude, which also had the highest applicability under Sonnet 5 (88.9%), and Agentic-Gemini were the only configurations in the top three under all three executors, but the intervals overlap heavily, so we do not claim a robust winner. The pipeline enhancements proposed by Gön et al. (Gon et al., 2026) did not translate into consistent gains: for Gemini, RAG resolved fewer fixes than Vanilla under OpenCUA-72B and Muse Glimmer but more under Sonnet 5, whereas for Kimi neither RAG nor Agentic web search resolved fewer fixes than Vanilla under Sonnet 5 or OpenCUA-72B. Agentic web search had the highest rate for both backbones under Sonnet 5 and Muse Glimmer, but not for Kimi under OpenCUA-72B. Applicability and resolution capture different ways in which a fix can fail. A configuration can fail by not proposing an executable action, as V-MiniMax did for almost half of its fixes under Sonnet 5, or by proposing an action that does not remove the symptom. The two failure modes call for different remedies: the first concerns the model’s willingness to commit to a concrete instruction, the second its knowledge of the application. At the same time, applicability as measured here depends on the executor, since OpenCUA-72B rarely declared any fix non-applicable. 5.4. The Scarcity of Executable No-Code Fixes The reduction from 302 annotated reports to 51 candidates and finally to nine executable issues is worth reading as a finding rather than merely as a description of our sample. Bug reports resolved without a code change are only part of what an issue tracker receives (Aktas et al., 2023; Erfani Joorabchi et al., 2014), and only a subset of those are resolved by an action that can be carried out in an environment at all; the remainder are answered with an explanation, a request for further information, or a decision that the reported behavior is intended (Erfani Joorabchi et al., 2014) (Section 3.1). What makes such reports costly to collect is that this distinction leaves no structural trace. Issue trackers do not label invalid reports as such, and their resolutions are not linked to commits, so there is no signal analogous to a fixing commit that can be mined automatically, as is routinely done when constructing code-level benchmarks. Identifying a no-code fix requires reading the maintainer discussion of a closed issue and judging what the resolution consisted of. Any effort to scale execution-based evaluation of no-code fixes will therefore have to confront dataset construction as a research problem in its own right, most plausibly by automating the recognition of no-code resolutions in maintainer comments. The reduction observed before and at the sanity gates is informative for a second reason. Of the 51 candidates, 38 could not be set up in a Linux browser environment at all, because they depended on another platform, a live backend, or information the report did not provide (Bruno et al., 2026), and 4 more did not exhibit the reported bug when tested. These are reports whose resolution was accepted by both the reporter and the maintainer, yet whose bug-fix pair could not be re-established outside the original reporting context. This suggests that no-code bugs are frequently entangled with state that the report does not capture, such as an accumulated user profile, a particular operating system or hardware configuration, the behavior of a third-party service at the time of reporting, or a server-side change that no local rollback can undo. Seen this way, the gates are not only a filter that reduced our evaluation set but a measurement instrument: they quantify how often a documented no-code fix can be independently re-executed at all, which is a precondition that any execution-based evaluation of this task, ours or otherwise, must satisfy. We consequently kept them strict, on the view that a verdict grounded in a verified buggy state and a verified fixed state carries more evidential weight than a larger set of verdicts resting on unchecked assumptions about the environment. Threats arising from the resulting set size are discussed in Section 6. 6. Threats to Validity 6.1. Internal Validity Executor reliability. Every verdict in our study depends on an executor carrying out a natural-language fix in a real browser. An executor may misapply a fix, apply it only partially, or change unrelated state, in which case a failed fix says nothing about the fix itself. The 20-step budget adds a related threat: an episode that is cut off may have succeeded with more steps, which matters most for Muse Glimmer, whose episodes exhausted the budget in 55.1% of cases. To reduce the effect of individual execution failures, we executed every fix three times per executor and counted a fix as resolved if any run succeeded, and we compare executor outcomes with human execution on a random sample of fixes. Executors could also act outside the browser: the container desktop offers a shell, and 10 OpenCUA-72B episodes typed shell-like commands. Nine of them failed without an update and one passed through the settings-page update, so no resolved episode depended on a shell. Harness differences. The three executors could not be driven by an identical harness. Sonnet 5 used its native computer-use tool, OpenCUA-72B used its own action format with the screenshot history window of its reference harness, and Muse Glimmer, which has no official computer-use harness, was driven through a function-calling schema that we designed. The executors also used different sampling settings (Section 3.4), which we did not tune. A higher temperature yields more varied runs and thus benefits more from counting a fix as resolved if any of three runs succeeds. Differences between executors may therefore partly stem from their harnesses and sampling settings, and our own schema may disadvantage Muse Glimmer. We kept the task prompt, the step budget, and the environment identical across executors, and we interpret RQ2 as a comparison of executors together with their harnesses rather than of the underlying models alone. Checker validity. Whether a fix resolves the bug is decided by an issue-specific checker that reads the browser state tied to the symptom in the bug report, such as a preference value or a property of the rendered page. A checker may accept a state that was reached without addressing the reported problem, or reject a valid fix that removes the symptom through a different route. In particular, for issues 25057 and 37372, the expected value is the browser’s default, which the browser does not store explicitly, so the checker also accepts the absence of the preference; any action that removes it, including a general reset of the browser settings, therefore counts as a resolution. We derived each checker from the actual result stated in the bug report, release all checkers and execution traces in our replication package, and use the human execution in RQ3 as an independent check of their verdicts. Metric definitions. Two of our metrics involve design choices that affect the reported values. Applicability relies on the executor’s decision to declare a fix BLOCKED and thus reflects the executor’s judgment as well as the fix; OpenCUA-72B, for instance, ended only 4.7% of its episodes in BLOCKED. Counting a fix as resolved if any of three runs succeeded yields higher rates than a majority vote, and the gain is larger for less stable executors. We therefore report every individual run (Table 5) and provide majority-vote results in the replication package. On the RQ3 sample, a majority vote changed the agreement with the human consensus by at most 6.8% per executor, and dropping the six fixes on which the first human recorded partial or not applicable changed the agreement by at most 4.2 points. The BERTScore analysis in Section 5.2 covers only fixes that failed in every attempt, whereas the range reported by Gön et al. (Gon et al., 2026) spans all seven subclasses and their reference fixes. Buggy-state fidelity. Per-fix episodes start from a reconstructed buggy state: a fresh profile on the gate-verified build, with the buggy value pre-seeded for three issues, or the reported build for version-type issues. The sanity gates confirm that this state shows the reported symptom and that the maintainer’s fix removes it, but the reporter’s original state may have depended on context we cannot recreate, such as an accumulated profile or a server-side condition. Two reports state no version (25057 and 37372), so we used a nearby pinned build for them. Human reference. The human executors in RQ3 are the first two authors. Their familiarity with the pipeline, and in some cases with a bug’s ground-truth resolution, may influence how they execute a fix. We blinded both executors to the configuration that generated each fix and to the executors’ outcomes, and had both execute every sampled fix independently. Their agreement was moderate (78.7%, κ=0.54κ=0.54), and the second human resolved more fixes than the first (29 against 25), so the human outcome is a reference rather than a validated ground truth. Restricting the comparison to the 59 fixes with a shared outcome removes the fixes that were hardest to judge and may therefore overstate executor accuracy; we report the agreement with each human separately for this reason. Each human executed a fix once, whereas an executor’s verdict aggregates three runs, and the consensus set contains only 19 resolved fixes, so the intervals in Table 7 are wide and we do not test differences between executors for significance. 6.2. External Validity Application and bug types. All issues come from a single application, the Brave browser, and a single prior benchmark (Gon et al., 2026), and are restricted to the three subclasses whose resolutions are executable actions. Our findings may not transfer to other browsers, to desktop or mobile applications with different interaction models, or to the subclasses we excluded by design. Moreover, our environment runs on Linux, and 17 of the 51 candidate issues were excluded because they were specific to another platform; the evaluated issues may therefore differ systematically from those that could not be reproduced. Size of the evaluation set. The effective evaluation set contains nine issues and 322 fixes, further divided across 12 configurations and three executors. The subclass breakdown rests on five, one, and three issues, so the Wrong Version result describes a single issue. We report agreement statistics and rank correlations descriptively, together with issue-clustered bootstrap intervals, and do not perform significance tests. The intervals are wide and overlap (for example 38.6 to 61.3 for Sonnet 5 and 27.6 to 64.5 for OpenCUA-72B), so differences between configurations and between executors should be read as descriptive rather than statistically confirmed. 7. Conclusion We presented a rollback-based, automated execution-driven pipeline that verifies LLM-generated no-code fixes by applying them with an executor in a real browser and checking the resulting state. Applied to 322 fixes from 12 LLM-based configurations on nine reproducible Brave bug reports, the pipeline shows that current LLMs resolve between 14.6% and 49.7% of the reported bugs depending on the executor, with V-Claude the strongest configuration under Sonnet 5 but not under the other two executors. The choice of executor shifted the measured capability of a configuration by 38.8% on average and only weakly preserved configuration rankings, which makes the executor a first-order variable in any execution-based evaluation of this kind. Only nine of 51 candidate reports could be re-established in a controlled environment, so scaling this form of evaluation will require automated construction of executable no-code benchmarks. Compared with human execution on a random sample, Sonnet 5 was the most accurate executor (88.1% agreement with the human consensus), whereas OpenCUA-72B overstated resolution and agreed on 66.1%. Future work will extend the pipeline to more applications and bug types, and study how executor harnesses can be standardized so that verdicts become comparable across executors. 8. Data Availability Our replication package is available at https://figshare.com/s/eda4e478ccbf242fbdab. It contains the Dockerfile and build scripts for the pinned Brave builds of the evaluated issues, the per-issue environment setup (profile seeds and scripted gate actions), the issue-specific checkers, the executor prompts and action interfaces (including our function-calling schema for Muse Glimmer), the verdicts of executor episodes and the text traces of them, the 322 evaluated fix texts, the human verdicts and sampling seed for RQ3, the status of all 51 candidate issues with the reason each excluded issue was dropped, and the scripts that produce every table, including the majority-vote variants. Acknowledgements We thank OpenZeka for generously providing access to their GPU resources, which made the experiments in this work possible. References Aktas et al. (2023) E. U. Aktas, E. Cakmak, M. C. Inan, and C. Yilmaz Issue report validation in an industrial context. In Proceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering, p. 2026–2031. Cited by: §5.4. Anthropic (2026a) Anthropic Introducing Claude Opus 4.6. Note: https://w.anthropic.com/news/claude-opus-4-6Accessed: 2026-10-01 Cited by: §3.2. Anthropic (2026b) Anthropic Introducing Claude Sonnet 5. Note: https://w.anthropic.com/news/claude-sonnet-5Accessed: 2026-09-24 Cited by: §1, §3.4, §3.7. Antoniol et al. (2008) G. Antoniol, K. Ayari, M. Di Penta, F. Khomh, and Y. Guéhéneuc Is it a bug or an enhancement? a text-based approach to classify change requests. In Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds, p. 304–318. Cited by: §2.1. Aracena et al. (2024) G. Aracena, K. Luster, F. Santos, I. Steinmacher, and M. A. Gerosa Applying large language models to issue classification. In Proceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering, p. 57–60. Cited by: §1, §2.1. Bruno et al. (2026) V. L. Bruno, A. Giagnorio, D. Bifolco, L. Wienges, M. Di Penta, and G. Bavota Writing bug reports for software repair agents: what information matters most?. arXiv preprint arXiv:2607.09553. Cited by: §5.4. Celik and Mahmoud (2025) A. Celik and Q. H. Mahmoud A review of large language models for automated test case generation. Machine Learning and Knowledge Extraction 7 (3), p. 97. Cited by: §1. Chen et al. (2025) M. Chen, Z. Liu, C. Chen, J. Wang, B. Wu, J. Hu, and Q. Wang Standing on the shoulders of giants: bug-aware automated gui testing via retrieval augmentation. Proceedings of the ACM on Software Engineering 2 (FSE), p. 825–846. Cited by: §2.2, §2.2. Cheng et al. (2025) R. Cheng, M. Tufano, J. Cito, J. Cambronero, P. Rondon, R. Wei, A. Sun, and S. Chandra Agentic bug reproduction for effective automated program repair at google. arXiv preprint arXiv:2502.01821. Cited by: §2.2, §2.2. Cihan et al. (2025) U. Cihan, V. Haratian, A. İçöz, M. K. Gül, Ö. Devran, E. F. Bayendur, B. M. Uçar, and E. Tüzün Automated code review in practice. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), p. 425–436. Cited by: §1. Cochran (1977) W. G. Cochran Sampling techniques. Vol. 3, wiley New York. Cited by: §3.5. Cohen (1960) J. Cohen A coefficient of agreement for nominal scales. Educational and psychological measurement 20 (1), p. 37–46. Cited by: §3.6. Colavito et al. (2025) G. Colavito, F. Lanubile, and N. Novielli Benchmarking large language models for automated labeling: the case of issue report classification. Information and Software Technology 184, p. 107758. Cited by: §1, §2.1. Dinç and Tüzün (2025) E. Dinç and E. Tüzün Judge the votes: a system to classify bug reports and give suggestions. In 2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware), p. 01–10. Cited by: §2.1. Du et al. (2024) X. Du, Z. Liu, C. Li, X. Ma, Y. Li, and X. Wang LLM-brc: a large language model-based bug report classification framework. Software Quality Journal 32 (3), p. 985–1005. Cited by: §1, §2.1. Erfani Joorabchi et al. (2014) M. Erfani Joorabchi, M. Mirzaaghaei, and A. Mesbah Works for me! characterizing non-reproducible bug reports. In Proceedings of the 11th working conference on mining software repositories, p. 62–71. Cited by: §5.4. Fan et al. (2018) Y. Fan, X. Xia, D. Lo, and A. E. Hassan Chaff from the wheat: characterizing and determining valid bug reports. IEEE transactions on software engineering 46 (5), p. 495–525. Cited by: §1, §2.1. Feng and Chen (2024) S. Feng and C. Chen Prompting is all you need: automated android bug replay with large language models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, p. 1–13. Cited by: §2.2, §2.2. Gon et al. (2026) M. F. Gon, E. Dinc, T. E. Sungur, and E. Tuzun Automated root-cause subclassification and no-code fix generation for invalid bug reports. External Links: 2605.17561, Link Cited by: §1, §1, §2.1, §2.3, §3.1, Table 1, Table 1, §5.2, §5.2, §5.3, §6.1, §6.2. Google (2026) Google Gemini 3.1 Pro: a smarter model for your most complex tasks. Note: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/Accessed: 2026-10-01 Cited by: §3.2. Guo et al. (2026) Y. Guo, Y. Liu, J. M. Zhang, Y. Ma, Y. Lou, and Z. Chen SWE-doctor: guiding software engineering agents with runtime diagnosis from multi-faceted bug reproduction tests. arXiv preprint arXiv:2607.00990. Cited by: §2.2, §2.2. He et al. (2025) J. He, J. Shi, T. Y. Zhuo, C. Treude, J. Sun, Z. Xing, X. Du, and D. Lo From code to courtroom: llms as the new software judges. arXiv preprint arXiv:2503.02246. Cited by: §2.3. Herzig et al. (2013) K. Herzig, S. Just, and A. Zeller It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In 2013 35th international conference on software engineering (ICSE), p. 392–401. Cited by: §1, §2.1. Hong et al. (2026) S. Hong, Z. Liu, T. Li, W. Tao, Y. Wu, and C. Wu DiagEval: trajectory-conditioned diagnosis for reliable software evaluation with gui agents. arXiv preprint arXiv:2605.17439. Cited by: §2.2, §5.1. İr et al. (2026) M. Ö. İr, M. Dedeler, A. Koyuncu, and E. Tüzün Fixpad++: automated bug fix verification using llm agents. In Proceedings of the 3rd ACM International Conference on AI-Powered Software, p. 248–258. Cited by: §2.2, §2.2. Jiang et al. (2026) J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35 (2), p. 1–72. Cited by: §1. Jiang et al. (2024) X. Jiang, Y. Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology 33 (7), p. 1–30. Cited by: §1. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan Swe-bench: can language models resolve real-world github issues?. In International Conference on Learning Representations, Vol. 2024, p. 54107–54157. Cited by: §1, §2.3. Kang et al. (2024) S. Kang, J. Yoon, N. Askarbekkyzy, and S. Yoo Evaluating diverse large language models for automatic and general bug reproduction. IEEE Transactions on Software Engineering 50 (10), p. 2677–2694. Cited by: §2.2, §2.2. Kolthoff et al. (2025) K. Kolthoff, F. Kretzer, S. P. Ponzetto, A. Maedche, and C. Bartelt GUISpector: an mllm agent framework for automated verification of natural language requirements in gui prototypes. arXiv preprint arXiv:2510.04791. Cited by: §2.2, §2.2. Laiq et al. (2022) M. Laiq, N. b. Ali, J. Böstler, and E. Engström Early identification of invalid bug reports in industrial settings–a case study. In International Conference on Product-Focused Software Process Improvement, p. 497–507. Cited by: §1. Laiq et al. (2025) M. Laiq, N. bin Ali, J. Börstler, and E. Engström A comparative analysis of ml techniques for bug report classification. Journal of Systems and Software 227, p. 112457. Cited by: §2.1. Meng and Visser (2023) Q. Meng and J. Visser Which bug reports are valid and why? using the bert transformer to classify bug reports and explain their validity. In Proceedings of the 4th European Symposium on Software Engineering, p. 52–60. Cited by: §2.1. Meta Superintelligence Labs (2026) Meta Superintelligence Labs Introducing muse glimmer: an open agentic model. Note: https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-modelAccessed: 2026-09-24 Cited by: §1, §3.4. MiniMax (2026) MiniMax MiniMax M2.7: early echoes of self-evolution. Note: https://w.minimax.io/news/minimax-m27-enAccessed: 2026-10-01 Cited by: §3.2. OpenAI (2026) OpenAI Introducing GPT-5.4. Note: https://openai.com/index/introducing-gpt-5-4/Accessed: 2026-10-01 Cited by: §3.2. Otoom et al. (2019) A. F. Otoom, S. Al-jdaeh, and M. Hammad Automated classification of software bug reports. In Proceedings of the 9th international conference on information communication and management, p. 17–21. Cited by: §2.1. Pandey et al. (2017) N. Pandey, D. K. Sanyal, A. Hudait, and A. Sen Automated classification of software issue reports using machine learning techniques: an empirical study. Innovations in Systems and Software Engineering 13 (4), p. 279–297. Cited by: §2.1. Pornprasit and Tantithamthavorn (2024) C. Pornprasit and C. Tantithamthavorn Fine-tuning and prompt engineering for large language models-based code review automation. Information and Software Technology 175, p. 107523. Cited by: §1. Schäfer et al. (2023) M. Schäfer, S. Nadi, A. Eghbali, and F. Tip An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering 50 (1), p. 85–105. Cited by: §1. Shi et al. (2025) L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. Vosoughi Judging the judges: a systematic study of position bias in llm-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, p. 292–314. Cited by: §1. Stack Overflow (2025) Stack Overflow Stack overflow developer survey 2025: ai. Note: https://survey.stackoverflow.co/2025/aiAccessed: 2026-09-08 Cited by: §1. Team et al. (2026) K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Z. Chai, Y. Charles, H. Che, C. Chen, et al. Kimi k2. 5: visual agentic intelligence. arXiv preprint arXiv:2602.02276. Cited by: §3.2. Torun et al. (2026a) U. B. Torun, M. T. Demircan, M. F. Gön, and E. Tüzün Past, present, and future of bug tracking in the generative ai era. ACM Transactions on Software Engineering and Methodology. Cited by: §2.1. Torun et al. (2026b) U. B. Torun, V. Karakaya, A. Babar, and E. Tüzün Evaluation of llm-based software engineering tools: practices, challenges, and future directions. External Links: 2604.24621, Link Cited by: §1, §2.3. Wang et al. (2025) X. Wang, B. Wang, D. Lu, J. Yang, T. Xie, J. Wang, J. Deng, X. Guo, Y. Xu, C. H. Wu, Z. Shen, Z. Li, R. Li, X. Li, J. Chen, B. Zheng, P. Li, F. Lei, R. Cao, Y. Fu, D. Shin, M. Shin, J. Hu, Y. Wang, J. Chen, Y. Ye, D. Zhang, D. Du, H. Hu, H. Chen, Z. Zhou, H. Yao, Z. Chen, Q. Gu, Y. Wang, H. Wang, D. Yang, V. Zhong, F. Sung, Y. Charles, Z. Yang, and T. Yu OpenCUA: open foundations for computer-use agents. External Links: 2508.09123, Link Cited by: §1, §3.4. Wilson (1927) E. B. Wilson Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22 (158), p. 209–212. Cited by: §3.5. Xie et al. (2024) T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37, p. 52040–52094. Cited by: §2.3. Xuan et al. (2014) J. Xuan, H. Jiang, Y. Hu, Z. Ren, W. Zou, Z. Luo, and X. Wu Towards effective bug triage with software data reduction techniques. IEEE transactions on knowledge and data engineering 27 (1), p. 264–280. Cited by: §1. Yadavally et al. (2025) A. Yadavally, H. Nguyen, L. Callot, and G. Guinet Large language model critics for execution-free evaluation of code changes. arXiv preprint arXiv:2501.16655. Cited by: §2.3. Yapağcı et al. (2025) E. Yapağcı, Y. A. S. Öztürk, and E. Tüzün Agents in the sandbox: end-to-end crash bug reproduction for minecraft. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 3095–3107. Cited by: §2.2, §2.2. Yu et al. (2024) Y. Yu, G. Rong, H. Shen, H. Zhang, D. Shao, M. Wang, Z. Wei, Y. Xu, and J. Wang Fine-tuning large language models to improve accuracy and comprehensibility of automated code review. ACM transactions on software engineering and methodology 34 (1), p. 1–26. Cited by: §1. Zanetti et al. (2013) M. S. Zanetti, I. Scholtes, C. J. Tessone, and F. Schweitzer Categorizing bugs with social networks: a case study on four open source software communities. In 2013 35th International Conference on Software Engineering (ICSE), p. 1032–1041. Cited by: §2.1. Zeng et al. (2026) A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Cited by: §3.2. Zhang et al. (2026a) L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, et al. Swe-bench goes live!. Advances in Neural Information Processing Systems 38. Cited by: §2.3. Zhang et al. (2024) Q. Zhang, Y. Shang, C. Fang, S. Gu, J. Zhou, and Z. Chen Testbench: evaluating class-level test case generation capability of large language models. arXiv preprint arXiv:2409.17561. Cited by: §1. Zhang et al. (2023) S. Zhang, Z. Chen, Y. Shen, M. Ding, J. B. Tenenbaum, and C. Gan Planning with large language models for code generation. arXiv preprint arXiv:2303.05510. Cited by: §1. Zhang et al. (2019) T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi Bertscore: evaluating text generation with bert. arXiv preprint arXiv:1904.09675. Cited by: §1, §5.2. Zhang et al. (2026b) Y. Zhang, Z. Pan, I. N. B. Yusuf, H. Ruan, R. Shariffdeen, and A. Roychoudhury Code review agent benchmark. arXiv preprint arXiv:2603.23448. Cited by: §1. Zhou et al. (2024) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. Webarena: a realistic web environment for building autonomous agents. In International Conference on Learning Representations, Vol. 2024, p. 15585–15606. Cited by: §2.3.

Instant research discovery