Paper deep dive
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
Weijia Zhang, Kunlun Zhu, Zeyi Liu, Yinting Chen, Tianyi Ma, Jiateng Liu, Jiaxun Zhang, Bingxuan Li, Xiangru Tang, Heng Ji, Jiaxuan You
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/5/2026, 3:43:48 AM
Summary
The paper introduces CUADebug, a framework for diagnosing and repairing Computer-Use Agent (CUA) failures. It presents CUAErrorBench, a human-annotated benchmark of 204 failed OSWorld trajectories, and CUADebugger, a tool-augmented debugger that uses multimodal step inspection and structured root-cause analysis to improve diagnosis accuracy and re-execution success rates compared to naive prompting.
Entities (13)
Relation Signals (10)
CUADebug → includes → CUAErrorBench
confidence 95% · CUADebug includes a CUA-specific error taxonomy, CUAErrorBench
CUADebug → includes → CUADebugger
confidence 95% · CUADebug includes ... CUADebugger, a tool-augmented debugger.
Task Reasoning and Control → islargestfailurefamilyin → CUAErrorBench
confidence 95% · task reasoning and control is the largest failure family (110/204)
CUAErrorBench → contains → OSWorld
confidence 90% · human-annotated OSWorld failure benchmark
CUADebugger → guides → re-execution
confidence 90% · our method improves success from 12.2% to 25.86%
CUADebugger → improves → joint subtype-and-step diagnosis
confidence 90% · CUADebugger improves joint subtype-and-step diagnosis from 11.2% to 19.6%
CUADebugger → uses → ReAct
confidence 90% · CUADebugger runs a ReAct (Yao et al., 2023) loop
CUAErrorBench → annotates → Claude 4.5 Sonnet
confidence 85% · 144 Claude 4.5 Sonnet 50-step trajectories
CUAErrorBench → annotates →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Computer-use agents (CUAs) operate real desktop and web interfaces through screenshots, mouse and keyboard actions, and stateful UI feedback, yet their failures remain difficult to diagnose and repair. Unlike text-only agents, CUA failures arise from coupled visual perception, spatial grounding, low-level interaction, task reasoning, and environment dynamics, making debugging a distinctive multimodal causal localization problem. We introduce CUADebug, a framework for diagnosing and repairing CUA failures. CUADebug includes a CUA-specific error taxonomy, CUAErrorBench, a human-annotated OSWorld failure benchmark, and CUADebugger, a tool-augmented debugger. Instead of prompting over the full trajectory once, CUADebugger actively inspects suspicious steps with paired before/after screenshots and action traces, then submits a structured diagnosis containing the root-cause step, error type, grounded evidence, and corrective strategy for re-execution. Human annotations over 204 failed trajectories show that task reasoning and control is the largest failure family (110/204), followed by perception (36), grounding/interaction (25), external/system (13), and an others category of 20 OSWorld infeasible-task cases. On the main Claude-agent split, CUADebugger improves joint subtype-and-step diagnosis from 11.2% to 19.6% with Gemini 2.5 Pro and improves consistently across debugger backbones. In single re-execution package evaluation, RCA-based conditions achieve higher task completion than history-only continuation (28.47% with machine RCA and 29.90% with our method, versus 13.89%); in continual re-execution, our method improves success from 12.2% to 25.86%, while human-oracle guidance reaches 29.21%. These results show that CUA root-cause diagnosis can provide actionable repair signals rather than merely post-hoc explanations.
Tags
Links
- Source: https://arxiv.org/abs/2608.02643v1
- Canonical: https://arxiv.org/abs/2608.02643v1
Trouble viewing inline? Open PDF directly →
Full Text
68,105 characters extracted from source content.
Expand or collapse full text
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures Weijia Zhang 1 , Kunlun Zhu 1 , Zeyi Liu 1 , Yinting Chen 1 , Tianyi Ma 1 , Jiateng Liu 1 , Jiaxun Zhang 1 , Bingxuan Li 1 , Xiangru Tang 2 , Heng Ji 1,† , Jiaxuan You 1,† 1 University of Illinois Urbana-Champaign 2 Yale University † Corresponding authors weijia4,kunlunz2,hengji,jiaxuan@illinois.edu Abstract Computer-use agents (CUAs) operate real desk- top and web interfaces through screenshots, mouse and keyboard actions, and stateful UI feedback, yet their failures remain difficult to diagnose and repair. Unlike text-only agents, CUA failures arise from coupled visual per- ception, spatial grounding, low-level inter- action, task reasoning, and environment dy- namics, making debugging a distinctive mul- timodal causal localization problem. We in- troduce CUADebug, a framework for diag- nosing and repairing CUA failures. CUADe- bug includes a CUA-specific error taxonomy, CUAErrorBench, a human-annotated OSWorld failure benchmark, and CUADebugger, a tool- augmented debugger. Instead of prompting over the full trajectory once, CUADebugger actively inspects suspicious steps with paired before/after screenshots and action traces, then submits a structured diagnosis containing the root-cause step, error type, grounded evidence, and corrective strategy for re-execution. Hu- man annotations over 204 failed trajectories show that task reasoning and control is the largest failure family (110/204), followed by perception (36), grounding/interaction (25), ex- ternal/system (13), and an others category of 20 OSWorld infeasible-task cases. On the main Claude-agent split, CUADebugger im- proves joint subtype-and-step diagnosis from 11.2% to 19.6% with Gemini 2.5 Pro and improves consistently across debugger back- bones. In single re-execution package evalu- ation, RCA-based conditions achieve higher task completion than history-only continuation (28.47% with machine RCA and 29.90% with our method, versus 13.89%); in continual re- execution, our method improves success from 12.2% to 25.86%, while human-oracle guid- ance reaches 29.21%. These results show that CUA root-cause diagnosis can provide action- able repair signals rather than merely post-hoc explanations. Why CUA Failures Are Hard to Diagnose Three challenges in this G1 case; CUAErrorBench also covers planning and system failures. REAL CASE Instruction: change barely visible slide numbers to red Environment: LibreOffice Impress human: G1 1Context The editable slide-number text is tiny inside a larger placeholder object. The simplified view contains both the target text and the surrounding frame. → CHALLENGE 1 Perception cue the real target is a tiny visual detail inside a larger object CHALLENGE 2 Interaction mechanics single-click, double-click, and focus lead to different UI states ⏱ CHALLENGE 3 Delayed symptom the symptom appears only after later formatting actions <number> 2 Root Step Intent: select the text. Actual: a single click selects the frame. text target frame selected <number> 3 ⏱ Later Many steps later, the slide number is still faint instead of red. The final state shows failure, but not which earlier action caused it. 5 Figure 1: Teaser example of an early UI-selection mis- take causing later formatting failure. 1 Introduction Computer-use agents (CUA agents) are a natural interface for applying language and vision models to real software, because they can operate desktop, web, and mobile GUIs through screenshots and mouse or keyboard actions. Compared with text- only or tool-only agents, CUA agents must jointly solve visual perception, spatial grounding, low- level interaction mechanics, and long-horizon state tracking. These coupled requirements make fail- ures difficult to diagnose: a final wrong file, chart, or UI setting may only be the visible symptom of an earlier perception, grounding, reasoning, or envi- ronment error. Benchmarks such as OSWorld (Xie et al., 2024), WebArena (Zhou et al., 2024), Visu- alWebArena (Koh et al., 2024), Mind2Web (Deng et al., 2023), AndroidWorld (Rawles et al., 2025), and Windows Agent Arena (Bonatti et al., 2024) make end-to-end success measurable, but they do not by themselves explain where failed CUA tra- jectories go wrong. Existing agent evaluation pipelines are therefore not sufficient for CUA debugging. End-to-end task success only tells us whether an agent finished 1 arXiv:2608.02643v1 [cs.SE] 31 Jul 2026 204 annotations Overall 050100 Share of annotations (%) Claude 4.5 Sonnet Gemini 2.5 Pro Qwen 3.5 25 8 3 6 8 76 15 19 1319n=144 n=30 n=30 By trajectory source P 36G 25R 110S 13O 20 Figure 2: CUAErrorBench annotation statistics by source. a task, not why it failed. Error taxonomies and failure-attribution studies expose useful categories of agent mistakes (Zhu et al., 2025; Cemri et al., 2025; Shah et al., 2026; Zhang et al., 2025a; Zhu et al., 2026; Ma et al., 2026), but CUA trajecto- ries require causal judgments over screenshots, spa- tial targets, low-level actions, and temporal screen transitions. As illustrated in Figure 1, the action- able error may occur before the final failure sig- nal. Because these signals are difficult to infer from rewards alone, CUA debugging needs human- grounded supervision over the root-cause step, sup- porting evidence, and actionable correction. This paper studies CUA root-cause debugging through three research questions. First, where do CUA agents fail? We answer this with a CUA- specific taxonomy and human failure analysis over annotated OSWorld trajectories. Second, can a tool- augmented RCA agent improve over naive prompt- ing while matching human root-cause judgments? We compare CUADebugger, our debugging agent, against a prompt-only baseline that predicts the same RCA record from the full trajectory and tax- onomy, using both taxonomy and step-localization metrics against human annotations. Third, can RCA guide re-execution? We test whether diag- nostic context can improve continual or localized re-rollout rather than merely producing plausible explanations. To support these questions, we introduce CUAEr- rorBench, a human-annotated OSWorld failure benchmark. Each example labels the root-cause step, L1/L2 taxonomy label, supporting evidence, corrective strategy, and confidence. CUAError- Bench is not another task-success benchmark; it converts failed trajectories into causal supervision for comparing human and model root-cause judg- ments. Our benchmark contains 204 usable anno- tations across Claude 4.5 Sonnet, Gemini 2.5 Pro, and Qwen 3.5 trajectory sources. We also introduce CUADebugger, a tool- augmented RCA agent for CUA trajectories (Fig- ure 3). Rather than asking a model to infer the root cause from the full trajectory in one pass, CUADebugger runs a ReAct (Yao et al., 2023) loop with two CUA-specific debugging tools: a multimodal step-inspection tool that returns paired before/after screenshots together with the action, reasoning, and execution status at any chosen step; and a structured root-cause submission tool whose schema enforces coarse-to-fine taxonomy selection together with grounded evidence, a concrete cor- rection, a calibrated confidence score, and per-step summaries that are passed forward to re-rollout. When episodic memory is enabled, CUADebugger also retrieves transferable diagnoses distilled from prior failures and human annotations as optional RCA context. This design links diagnosis with re-execution: a useful diagnosis should not only describe a failure, but also provide a repair signal that can be tested by re-rollout. Our experiments support three main findings. First, human annotations show that failures concen- trate in task reasoning and control but remain gen- uinely multimodal, spanning perception, ground- ing/interaction, external/system, and infeasible- task cases (Figure 2). Second, CUADebugger im- proves fine-grained and joint diagnosis consistently across debugger backbones on the main Claude- agent split (e.g., joint Tag+Step Exact rises from 11.2% to 19.6% with Gemini 2.5 Pro). Third, diag- nostic context can support repair: single re-rollout with structured RCA reaches 29.90% task comple- tion versus 13.89% for history-only continuation, and continual re-rollout improves from 12.2% to 25.86%, approaching the 29.21% human-oracle result. Our core contributions are threefold. First, we analyze where CUA agents fail and introduce a five-category CUA-specific root-cause taxon- omy (Table 1). Second, we introduce CUAError- Bench, a human-annotated benchmark that turns failed OSWorld trajectories into root-cause super- vision. Third, we develop CUADebugger, a tool- augmented RCA agent that combines a ReAct loop with multimodal step-inspection, structured root- cause submission, and optional episodic memory retrieval, and evaluate it against a naive prompt baseline and human judgments. 2 CUADebug Pipeline: From Failed Trajectory to Repairable Root Cause Failed CUA trajectory INSTRUCTION make slide numbers red ENVIRONMENTLibreOffice Impress Step 2: click <number> STEP 1 open Master View STEP 2 intent: select text; actual: frame selected LATER agent formats and reports success FINAL slide number remains faint gray Slide Number<number> Tool-guided multimodal inspection Tool: get_step_details(n) Returns before screenshot, action, reasoning, execution status, and after screenshot as one paired observation. screen before action screen after ReAct search over candidate steps The debugger writes persistent observation notes and localizes the earliest causal error. Structured RCA record ROOT STEP earliest causal error TAXONOMYL1/L2 SUBTYPE EVIDENCE grounded trajectory evidence CORRECTION what should change at that step SUMMARIES inspected step intentions + outcomes Memory Store reusable memory trigger condition, failed action, corrected action, distinguishing feature, evidence, and source episode. query with task and failure cues retrieve matched memories for RCA Use with caution Retrieved memories must match the current screenshots and trace before they influence RCA. Debugger- guided re- rollout Repair recipe Inject root step, error tag, evidence, correction, and inspected-step summaries into the acting agent. avoid repeating the root error measure re-rollout score and task success turn explanation into a testable intervention Figure 3: CUADebugger pipeline: inspect failed steps, produce structured RCA, retrieve memories when enabled, and guide re-rollout. 2 CUAErrorBench We construct CUAErrorBench from failed OS- World trajectories produced by three CUA agent sources. Each trajectory contains a task instruction, visual observations, screenshots, model reasoning traces, low-level actions, rewards, and terminal sta- tus. The active benchmark excludes the deprecated 15-step split and contains 204 usable annotations: 144 Claude 4.5 Sonnet 50-step trajectories, 30 Gem- ini 2.5 Pro trajectories, and 30 Qwen 3.5 trajecto- ries. Figure 2 shows that the largest human-labeled failure family is task reasoning and control, fol- lowed by perception and grounding/interaction. The annotation pipeline stores debugger- generated RCA proposals together with human an- notations; CUAErrorBench uses the human fields as reference labels. For each failed trajectory, anno- tators provide five fields: (1) the root-cause step, (2) a diagnostic label, (3) textual evidence, (4) a correc- tive action or strategy, and (5) confidence. All 204 paper-usable examples are annotated by multiple annotators; disagreements are discussed and re- solved into one final reference label. Appendix A.4 summarizes the human annotation instructions and consent procedure, and Appendix A.5 gives one complete annotated example. Benchmark curation and annotation protocol. Annotators identify the root cause by inspecting the task instruction, terminal failure signal, action and reasoning trace, and before/after screenshots. They first verify the final failure symptom, then work backward to find the earliest step that introduces a new causal mistake rather than merely inheriting an earlier error. Evidence must cite concrete trajectory content, such as a visible UI state, selected element, action code, or mismatch between the agent’s stated intention and the screen outcome. The correction field describes an actionable alternative at that step, while confidence records whether competing root causes remain plausible. Taxonomy labels.We use a two-level CUA error taxonomy as the common label space for annota- tion, RCA, memory retrieval, and evaluation. The top level identifies the causal module that deter- mines the repair family, while the subtype captures the concrete failure pattern needed for fine-grained evaluation and memory retrieval. Table 1 shows the five top-level categories; Appendix A.6 gives the full 30-subtype taxonomy. CodeError module Diagnostic questionTypical evidence PPerceptionDid the agent misunderstand what was visible in the observation? Hallucinated or missed UI content, OCR mistakes, cross-region binding errors, semantic misreadings. GGrounding and Interaction Did the agent know the intended operation but execute it on the wrong target or with the wrong mechanics? Wrong coordinates or element, hidden or disabled target, incorrect click, drag, gesture, or text-entry behavior. RTask Reasoning and Control Did the agent choose, maintain, or revise the wrong plan? Constraint violations, missing subgoals, action-intent mismatch, memory loss, progress misjudgment, failed self-correction. SExternal/SystemDid the environment, tool, or benchmark setup prevent otherwise valid progress? Rendering or timing failures, unexpected system dialogs, resource limits, tool/API failures, benchmark artifacts. OOthersDoes the failure fall outside the four execution modules above? Cases not attributable to P/G/R/S; currently OSWorld infeasible-task cases where the agent should recognize the task is impossible and report it. Table 1: Top-level CUA error taxonomy used in CUAEr- rorBench. 3 3 Method CUADebugger operationalizes CUA root-cause an- notations in three stages: root-cause prediction, memory retrieval, and debugger-guided re-rollout. 3.1 Root-Cause Analysis RCA predicts the root-cause step in a failed tra- jectory. CUADebugger performs this prediction by running a ReAct (Yao et al., 2023) agent loop equipped with CUA-specific debugging tools, rather than a one-shot prompt over the full trajec- tory. The agent is initialised with the task instruc- tion, trajectory metadata, terminal failure stepF, L1/L2 taxonomy definitions, an output schema, and optionally retrieved memories. At each turn it de- cides which step to inspect next; older screenshots are compressed out of the context window, while the agent’s textual observation notes are retained. Multimodal step-inspection tool. The core in- spection primitive,get_step_details(n), re- turns a single paired observation for stepn: the action code, the agent’s reasoning, the execution error and reward, and two screenshots: the input screenshot showing the screen state before acting and the result screenshot showing the state after the action executed. Pairing the screenshots against the agent’s stated intention exposes perception, ground- ing, and outcome-misinterpretation errors that text traces alone would hide. After each inspection, the agent writes a short observation note that persists across context compression. Structured root-cause submission tool. The agent terminates by callingfinish(·). Its schema enforces the full RCA record in one structured object: the root-cause stepN, L1/L2 taxonomy tag, grounded evidence, concrete correction, cali- brated confidence score in[0, 1], and one per-step summary for each inspected step. The per-step summaries are passed forward to re-rollout (Sec- tion 3.5) so the acting agent inherits the debugger’s interpretation of the failed trajectory. Memory retrieval. When episodic memory is enabled (Section 3.4), the initial prompt also in- cludes the top-kretrieved memories, marked as candidate evidence: the agent is instructed to com- pare each memory’s trigger condition against the current trajectory and discard non-matching memo- ries rather than copy them. Appendix A.1 gives the RCA prompt template, memory-retrieval prompt block, and tool schemas. 3.2 Error Taxonomy Module The error taxonomy module makes RCA a coarse- to-fine diagnosis. Given a candidate root-cause step, the debugger first selects one of the five top- level modules in Table 1, then selects a subtype within that module from Appendix A.6. This struc- ture separates causal source from terminal symp- tom, while the subtype indexes memories for re- trieval and the top-level module gives the acting agent an interpretable warning during re-rollout. 3.3 Multimodal Trajectory Inspection CUADebugger treats screenshots and action traces as paired evidence rather than separate inputs. For each tool-inspected step, the debugger records the visible state before the action, the intended UI op- eration, the executed low-level action, and the ob- served state afterward. This inspection exposes failures that are easy to miss from text traces alone, such as selecting the wrong chart subtype, clicking a nearby control, or assuming that a dialog changed state when the screenshot shows otherwise. The resulting step summaries are then used by RCA, memory distillation, and re-rollout guidance. 3.4 Episodic Memory and Retrieval-Augmented Debugging Episodic memory converts individual trajectories into reusable debugging knowledge. For each failed trajectory, CUADebugger stores an episode with task and application metadata, local error con- text, inferred agent intention, screen-outcome sum- mary, taxonomy tag, RCA output, and optional human annotation. This episode is the durable ref- erence object: later retrieval can return not only an abstract rule, but also the concrete trajectory evidence from which the rule was distilled. Retrieval-augmented debugging exposes this memory through a tool-use interface. From each episode, CUADebugger distills a memory with a short title, trigger condition, failed action, corrected action, taxonomy tag, distinguishing feature, evi- dence, and episodic references. Memories are in- dexed by transferable fields such as application id, taxonomy tag, trigger condition, failed action, and memory text. During RCA or re-rollout, the de- bugger or acting agent may query similar failures, but retrieved memories are treated as candidate ev- idence rather than authoritative labels: the model must compare each trigger and distinguishing fea- ture against the current screenshots and action trace. 4 In cold start, memories are distilled from human an- notations; after cold start, the same representation can absorb successful RCA outputs or contrastive failure/success trajectory pairs. Our experiments instantiate this design through a Machine RCA con- dition and our memory-enabled condition. 3.5 Debugger-Guided Re-Rollout Re-rollout tests whether RCA helps the agent act better, not only explain better. Both the direct base- line and the debugger-guided method replay the original OSWorld trajectory to the same cutoff state and continue with the same remaining step bud- get. The debugger-guided method keeps this state and budget fixed, but augments the acting prompt with the previous trajectory summary, debugger- inspected step summaries, and a repair recipe con- taining the root-cause step, error type, evidence, and correction. In the retrieval-enabled variant, the acting agent can additionally query episodic memory for similar prior failures before choosing its next action. Appendix A.2 lists the re-rollout prompt variants and visual side-channel tools. This design tests whether RCA is actionable: if the diagnosis contains causal repair information, in- jecting it during re-rollout should reduce repeated failure patterns; otherwise, it remains plausible prose without behavioral effect. 4 Experiments 4.1 Evaluation Protocol We evaluate CUA debugging along three research questions. First, failure analysis asks where CUA agents fail by measuring the human-labeled dis- tribution of root-cause modules across trajectory sources. Second, RCA evaluation asks whether CUADebugger improves over a naive prompt base- line and matches human root-cause judgments. Third, re-rollout evaluation asks whether RCA can guide subsequent execution. Unless otherwise stated, human annotations are the reference labels. 4.2 Failure Analysis Failure analysis uses the human labels in CUAEr- rorBench to characterize where CUA agents fail. We report top-level taxonomy distributions by agent trajectory source in Figure 2. This analy- sis answers a descriptive question rather than a model-ranking question: it provides the reference view of CUA failure modes against which model debugger behavior is later compared. Agent traj. RCA methodDebuggerL1L2 Step Exact Step ±2 Tag+ Step Claude 4.5NaiveGemini 2.5 Pro51.030.121.038.511.2 Claude 4.5CUADebugGemini 2.5 Pro53.837.129.447.619.6 Claude 4.5NaiveQwen 3.544.419.423.634.77.6 Claude 4.5CUADebugQwen 3.557.333.025.246.614.6 Claude 4.5NaiveClaude 4.549.314.622.238.24.9 Claude 4.5CUADebugClaude 4.561.331.429.245.315.3 Gemini 2.5NaiveQwen 3.546.720.026.750.06.7 Gemini 2.5CUADebugQwen 3.544.820.724.141.43.4 Gemini 2.5NaiveClaude 4.553.323.326.750.010.0 Gemini 2.5CUADebugClaude 4.543.323.333.350.016.7 Gemini 2.5NaiveGemini 2.5 Pro50.020.030.040.010.0 Gemini 2.5CUADebugGemini 2.5 Pro53.316.740.050.010.0 Qwen 3.5NaiveGemini 2.5 Pro53.323.333.343.310.0 Qwen 3.5CUADebugGemini 2.5 Pro54.536.445.563.418.2 Qwen 3.5NaiveQwen 3.546.720.016.733.33.3 Qwen 3.5CUADebugQwen 3.544.020.020.036.012.0 Qwen 3.5NaiveClaude 4.556.730.030.036.710.0 Qwen 3.5CUADebugClaude 4.545.536.427.336.49.1 Table 2: RCA accuracy against human annotations (per- cent). CUADebug denotes our tool-augmented RCA method; Tag+Step requires exact subtype and exact root-cause-step match. 4.3 RCA Evaluation The RCA evaluation reports both taxonomy and localization accuracy. We compare two RCA meth- ods. The Naive Baseline receives the task, full trajectory package, and taxonomy definitions, and produces the RCA record in one pass. CUADe- bugger receives the same task metadata and taxon- omy definitions but must use the step-inspection and structured-submission tools described in Sec- tion 3.1 to inspect step details before producing the same RCA record. Table 2 reports both methods. L1 accuracy measures top-level taxonomy agree- ment with the human annotation, while L2 accu- racy measures exact subtype agreement. Step Exact measures exact root-cause-step match, and Step±2 gives credit when the predicted root-cause step is within two steps of the human-labeled root-cause step. Tag+Step Exact requires both L2 subtype match and exact root-cause-step match. 4.4 Single Re-Rollout Evaluation Single re-rollout isolates whether a diagnosis can support behavioral repair. The rollout agent restarts one step before the root-cause step and runs with a fixed step and token budget.The Machine RCA condition uses the machine-predicted root- cause step and machine-generated diagnosis with- out memory retrieval to test the automatic RCA- to-repair pipeline, while the baseline, self-debug, 5 ConditionCostTaskRoot error MethodRCA sourceBudget Input tok. Output tok. Acc. (%) New+ Fixed Baselinenone28.01 272.7K3.5K 13.8920+11 Self-debugself31.53 324.4K3.9K 15.2827+12 Machine RCA machine RCA23.03 290.8K2.9K 28.4729+35 Human oraclehumanannota- tion 22.37 296.9K3.7K 20.1043+29 Our methodRCA + memory28.28 319.1K2.9K 29.9044+24 Table 3: Single re-rollout results on Claude 4.5 Sonnet trajectories. human-oracle, and our method conditions use the human-labeled root-cause step to fix the restart point and compare diagnostic context. This con- trols for localization error: if the restart step itself is wrong, the re-rollout may be uninformative re- gardless of the debugging method. Because the rows differ in diagnostic prompt format, Table 3 should be read as a re-execution-package compar- ison rather than a clean ablation of RCA source quality. The main root-error metric is New+Fixed: a post-analysis judge compares the original human annotation, including root-cause step, evidence, and correction, against the new trajectory; the first number counts cases where the original critical er- ror is fixed but a new critical error appears, and the second counts cases where the original critical error is fixed without a new critical error. When the budget reaches a task-level outcome we addition- ally report Re-rollout Acc as task completion rate. The agent trajectory and rollout agent are fixed to Claude 4.5 Sonnet, and Machine RCA diagnoses are produced by Gemini 2.5 Pro. 4.5 Continual Re-Rollout Evaluation Continual re-rollout tests whether diagnostic con- text helps an agent continue after failure. We use Claude 4.5 Sonnet 50-step failures, restart Claude 4.5 Sonnet from step 51, and compare direct con- tinuation, self-debugging, Machine RCA, our full method, and human annotation guidance. Machine RCA diagnoses are produced by Gemini 2.5 Pro. Re-rollout accuracy measures whether the task is completed after re-execution, while New+Fixed uses the same order as Table 3: fixed with a new critical error, followed by fixed without a new criti- cal error. Method Original turn Spend turnsInput OutputAcc. New+ Fixed Baseline5038.92 390.3K3.8K 12.2026+7 Self-debug5043.76 401.6K4.4K 14.20 30+11 Machine RCA5047.13 405.9K5.1K 21.53 21+44 Our method5047.44 426.2K5.4K 25.86 27+58 Human oracle5045.75 419.1K4.8K 29.21 29+44 Table 4: Continual re-rollout results on Claude 4.5 Son- net failures. 5 Results and Analysis CUA failures are concentrated in reasoning/con- trol but remain multimodal. Figure 2 shows that task reasoning and control accounts for the largest share of human-labeled failures in the active benchmark (110/204, 53.9%). This pattern is con- sistent across the three trajectory sources: Claude 4.5 Sonnet has 76 reasoning/control failures out of 144 annotations, Gemini 2.5 Pro has 15 out of 30, and Qwen 3.5 has 19 out of 30. At the same time, the remaining failures are not reducible to text- only planning mistakes: perception accounts for 36 cases, grounding and interaction for 25 cases, ex- ternal/system issues for 13 cases, and an others cat- egory for 20 OSWorld infeasible-task cases. This distribution motivates a CUA-specific taxonomy that explicitly separates visual perception, spatial interaction, reasoning/control, and system causes, plus an others category for failures outside these four modules. CUADebugger improves joint diagnosis on the main annotated split. Table 2 shows that the tool-augmented RCA agent improves RCA on the most complete Claude 4.5 Sonnet split. Relative to the naive prompt baseline, CUADebugger im- proves L2 accuracy and Tag+Step Exact for all three debugger backbones. Gemini 2.5 Pro im- proves from 30.1% to 37.1% L2 accuracy and from 11.2% to 19.6% Tag+Step Exact; Qwen 3.5 im- proves from 19.4% to 33.0% L2 and from 7.6% to 14.6% Tag+Step; Claude 4.5 Sonnet improves from 14.6% to 31.4% L2 and from 4.9% to 15.3% Tag+Step. This suggests that actively inspecting step details with paired before/after screenshots is more useful than simply providing the full trajec- tory and taxonomy in one prompt. Figure 4 further shows where the debugger agrees with human la- bels at the category level. Appendix A.7 reports the corresponding subtype-frequency comparison. 6 Perception Grounding Interaction Reasoning Control External System Others 0.0 0.2 0.4 0.6 0.8 1.0 F1 0.78 n=25 0.59 n=11 0.85 n=76 0.35 n=13 0.86 n=19 Category-level RCA F1 Figure 4: Category-level debugger F1 on the Claude 4.5 Sonnet split (N = 144), including the Others (O) category. Cross-agent RCA remains an important stress test. The Gemini-agent and Qwen-agent rows show that RCA behavior changes with both the failed agent and debugger backbone. Unlike the Claude split, CUADebugger is not uniformly better on every cross-agent row; for example, Qwen 3.5 on Gemini-agent trajectories has a higher Tag+Step score under the naive baseline than under CUADe- bugger, while Gemini 2.5 Pro on Qwen-agent tra- jectories benefits from CUADebugger despite the smaller evaluated subset. These mixed rows also highlight a dependency of CUADebugger on the underlying model’s capability: tool inspection can surface the relevant evidence, but the model must still interpret that evidence and revise its causal hypothesis. Stronger backbones appear better able to turn inspected evidence into corrected diagnoses, whereas weaker backbones may show limited gains or no improvement on small cross-agent splits. Be- cause these cross-agent rows have uneven sample sizes, we treat them as a stress test rather than as a final model ranking. Across settings, Tag+Step remains low, confirming that matching both the fine-grained cause and the exact root step is sub- stantially harder than predicting a coarse error fam- ily. Single re-rollout: structured RCA improves be- havioral repair. Table 3 measures whether a di- agnosis lets the rollout agent fix the original root- cause mistake. The structured RCA conditions pro- vide the strongest task-level recovery: Machine RCA reaches 28.47% Re-rollout Acc, and our method reaches 29.90%, compared with 13.89% for the history-only baseline and 15.28% for self- debug. In root-error outcomes, the same root error is fixed but followed by a new error in 29 Machine RCA cases and 44 our-method cases, and is fixed without a new critical error in 35 and 24 cases re- spectively. This repair setting reflects the same capability dependence as RCA: pointing out the root error is most useful when the rollout model can translate the diagnosis into a different action se- quence; weaker models may still repeat the failure or only partially avoid it. Because these rows differ in restart source and diagnostic prompt format, we treat these gains as evidence that structured repair packages can be useful, not as a direct comparison of diagnosis quality against human annotations. Continual re-rollout benefits from accurate RCA. Table 4 reports continual re-rollout from Claude 4.5 Sonnet 50-step failures. Direct con- tinuation reaches 12.20% task success, and self- debugging gives a modest gain to 14.20%. Ma- chine RCA improves success to 21.53%, while our method reaches 25.86%, narrowing much of the gap to the human-oracle row at 29.21%. The same pattern appears in root-error repair: our method fixes the first critical error in 85 cases total (27 with a new critical error and 58 without), compared with 33 total for the baseline and 41 total for self- debugging. Why human annotation: evolved taxonomies are path-dependent. A natural alternative to human annotation is to let a strong LLM evolve its own taxonomy directly from failed trajectories. We test this directly by running the debugger in an empty-start configuration: Gemini 2.5 Pro sees no taxonomy seed and must commit one of five operations per case (REUSE, DISCOVER, RENAME, SPLIT, MERGE) over the CUAErrorBench 50-step Claude split, with a running taxonomy state that persists across cases. We run four random case orderings; seed 3 completes all 144 cases on a paid API quota, while seeds 0 to 2 each cover 47 to 55 cases before truncation. Three findings emerge. (A) Evolved taxonomies do not converge across orderings. Across the 4 2 = 6 seed pairs, an LLM-as-judge under name-strict alignment finds zero matching subtypes; seed 0 organizes failures asP- * /OI- * /I- * , while seed 3 uses an eight- wayACTION/POLICY/EVALUATOR/VISUAL scheme, and the two cannot be unified by name. (B) The complete seed does not saturate. Seed 3 ends with 40 subtypes after 144 cases with the curve still rising, ruling out a data-quantity explanation for the divergence. (C) Semantic structure partially converges even though 7 naming does not. Projecting each run through a separate LLM-as-judge mapping back into CUAErrorBench’s label space recovers 15% to 28% of human subtypes per seed and roughly 40% in the four-seed union. Together, these results frame human annotation not as a labor convenience but as a structural requirement: autonomous discovery finds error structure, but cannot find a stable schema.CUAErrorBench’s value is therefore not just the 204 labels but the canonical schema those labels anchor. 6 Related Work Computer-Use Agent Benchmarks. Recent benchmarks evaluate agents in realistic computer- use environments, including desktop systems, web interfaces, and mobile applications (Xie et al., 2024; Zhou et al., 2024; Koh et al., 2024; Deng et al., 2023; Rawles et al., 2025; Bonatti et al., 2024).These benchmarks have made end-to- end task success measurable, but they do not by themselves explain why a failed trajectory failed. CUAErrorBench complements these benchmarks by adding human root-cause annotations over failed trajectories, and CUADebugger uses those annota- tions as corrective supervision. Agent failure analysis and debugging. Prior work studies agent failure taxonomies, multi-agent failure attribution, and CUA-specific trust or di- agnosis problems (Cemri et al., 2025; Shah et al., 2026; Zhang et al., 2025a; Zhu et al., 2026; Ma et al., 2026; Liu et al., 2026; Shi et al., 2026; Zhai et al., 2026). Most closely, AgentErrorBench and AgentDebug study root-cause debugging for gen- eral LLM agents over ALFWorld, GAIA, and Web- Shop (Zhu et al., 2025). Our work focuses on the CUA setting, where decisive evidence is often mul- timodal and interaction-level: screenshots, target grounding, temporal screen transitions, and low- level GUI mechanics. CUADebugger pairs CUA- specific human annotations with visual inspection tools, episodic memory over agent intentions and screen outcomes, and OSWorld state replay with re-execution guidance. Multimodal information flow. Recent modular multimodal systems decouple visual evidence ex- traction from downstream reasoning. SeeingEye, for example, uses an agentic vision translator with tools such as OCR and crop to produce structured intermediate representations for text-only LLM rea- soning (Zhang et al., 2025b). CUADebugger sim- ilarly separates visual trajectory inspection from causal reasoning, but targets failed CUA trajecto- ries rather than single-query multimodal answer- ing. Learning from failure.Language agents can im- prove from reflection, experience, retrieval, and memory (Yao et al., 2023; Shinn et al., 2023; Zhao et al., 2024; Wang et al., 2024; Hu et al., 2026). BacktrackAgent is especially related because it improves GUI agents through online error detec- tion and backtracking during task execution (Wu et al., 2025). CUADebugger addresses a differ- ent problem: rather than adding an online roll- back mechanism to a running agent, we build a human-annotated CUA failure benchmark and a tool-augmented RCA agent that performs after- execution causal localization. The output is a struc- tured diagnosis with evidence, taxonomy label, and correction, which can then be evaluated through controlled re-rollout and stored as reusable mem- ory. 7 Conclusion We studied CUA root-cause debugging through three questions: where CUA agents fail, whether a tool-augmented RCA agent improves over naive prompt-based diagnosis while matching human root-cause judgments, and whether RCA can guide re-execution. We introduced a CUA-specific tax- onomy, CUAErrorBench as a human-annotated failure benchmark, and CUADebugger as a tool- augmented RCA agent that produces re-execution guidance. Our experiments show that CUA fail- ures are dominated by reasoning/control but still include substantial perception and grounding/inter- action cases, that CUADebugger’s tool-based step inspection improves joint diagnosis on the main an- notated split, that RCA-based re-rollout packages improve task-level recovery under this protocol, and that our method substantially improves con- tinual re-execution over direct continuation while approaching the human-oracle result. Effective CUA debugging should therefore not stop at ex- plaining failures; it should convert explanations into reusable corrections that can be tested by re- execution. Limitations Our results are strongest for the failure analysis, Naive-vs-CUADebugger RCA comparison, single 8 re-rollout task accuracy, and continual re-rollout results on Claude 4.5 Sonnet failures. RCA sam- ple sizes remain uneven outside the Claude-agent trajectory split, so cross-agent debugger rankings should be interpreted as preliminary. Single re- rollout includes task-accuracy and critical-error out- come cells for the baseline, self-debug, Machine RCA, human-oracle, and our method settings, but these conditions are not fully matched: they vary in restart-step source and diagnostic prompt for- mat as well as in diagnosis source. Future work should also tighten the connection between subtype prediction, exact root-step localization, evidence quality, and behavioral repair during re-execution through matched re-rollout ablations. While there exist other Computer Use Agent benchmarks, we focus on using OSWorld to annotate and test our method; future work can try to expand to other benchmarks. References Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. 2024. Windows agent arena: Evaluating multi-modal OS agents at scale. arXiv preprint arXiv:2409.08264. Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why do multi-agent LLM systems fail? arXiv preprint arXiv:2503.13657. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a generalist agent for the web. In Advances in Neural Information Processing Sys- tems 36 (NeurIPS 2023), Datasets and Benchmarks Track. Zhanghao Hu, Qinglin Zhu, Runcong Zhao, Di Liang, Hanqi Yan, Yulan He, and Lin Gui. 2026. Beyond rag for agent memory: Retrieval by decoupling and aggregation. arXiv preprint arXiv:2602.02007. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neu- big, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating multi- modal agents on realistic visual web tasks. In Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 881–905, Bangkok, Thailand. Associ- ation for Computational Linguistics. Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, and Huamin Chen. 2026. Visual confused deputy: Exploiting and defending perception fail- ures in computer-using agents.arXiv preprint arXiv:2603.14707. Xuyan Ma, Xiaofei Xie, Yawen Wang, Junjie Wang, Boyu Wu, Mingyang Li, and Qing Wang. 2026.Demystifying the lifecycle of failures in platform-orchestrated agentic workflows. Preprint, arXiv:2509.23735. Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell- Ajala, Daniel Toyama, Robert Berry, Divya Tyam- agundlu, Timothy Lillicrap, and Oriana Riva. 2025. AndroidWorld: A dynamic benchmarking environ- ment for autonomous agents. In International Con- ference on Learning Representations. Mehil B Shah, Mohammad Mehdi Morovati, Moham- mad Masudur Rahman, and Foutse Khomh. 2026. Characterizing faults in agentic ai: A taxonomy of types, symptoms, and root causes.Preprint, arXiv:2603.06847. Yucheng Shi, Wenhao Yu, Jingyuan Huang, Wenlin Yao, Wenhu Chen, and Ninghao Liu. 2026. To- wards trustworthy gui agents: A survey. Preprint, arXiv:2503.23434. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Re- flexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024. Voyager: An open-ended embodied agent with large language models. Trans- actions on Machine Learning Research. Qinzhuo Wu, Pengzhi Gao, Wei Liu, and Jian Luan. 2025. Backtrackagent: Enhancing gui agent with error detection and backtracking mechanism. In Pro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 4250– 4272. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhou- jun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). 9 Yuwen Zhai, Runze Li, Liang Wang, Nian Shi, Liwu Xu, Wei Zhang, Ran Lin, Bo Xu, and Benlei Cui. 2026. Guide: Interpretable gui agent evaluation via hierarchical diagnosis. Preprint, arXiv:2604.04399. Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. 2025a. Which agent causes task failures and when? on automated failure attribution of LLM multi- agent systems. In Forty-second International Confer- ence on Machine Learning. Weijia Zhang, Zijia Liu, Haoru Li, Haoqi Chen, and Jiaxuan You. 2025b. Seeingeye: Agentic information flow unlocks multimodal reasoning in text-only llms. Preprint, arXiv:2510.25092. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. Expel: Llm agents are experiential learners. In Proceed- ings of the Thirty-Eighth AAAI Conference on Ar- tificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. AAAI Press. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Gra- ham Neubig. 2024. WebArena: A realistic web envi- ronment for building autonomous agents. In Interna- tional Conference on Learning Representations. Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Yuhui Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, and Daben Liu. 2026. RAFFLES: Reasoning-based attribution of faults for LLM sys- tems. In Proceedings of the 19th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 7659–7688, Rabat, Morocco. Association for Com- putational Linguistics. Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingx- uan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You. 2025. Where LLM agents fail and how they can learn from failures. arXiv preprint arXiv:2509.25370. 10 A Appendix A.1 RCA Prompt and Tool Schemas This appendix summarizes the prompt and tool interface used by the RCA methods in Section 3.1.The implementation sources are debugger/profiling/run_plain.py, debugger/rca.py,and debugger/tools/__init__.py.The taxonomy block in the live prompt is the subtype inventory in Appendix A.6; we reference it below to avoid duplicating the full taxonomy table. The naive RCA baseline is a single-call prompt with no tool access and no memory. It receives the task, the textual trajectory, the taxonomy table, and a small trailing screenshot window, then returns the same RCA JSON fields used by CUADebugger. RCA prompt-only baseline System: You are a debugging assistant analysing a failed GUI-agent trajectory. Your job is to identify: (a) the root error step: the single step number where the failure originates, not just where it surfaces; (b) the taxonomy_tag: the subtype code from the taxonomy table that best describes the root cause. Error Taxonomy: [P/G/R/S/O subtype table from Appendix CUA Error Taxonomy Subtypes] Return ONLY a single valid ASCII JSON object with no markdown fences, no prose, and no trailing text: "root_error_step": [int], "taxonomy_tag": [str; one subtype code], "evidence": [str; 1-2 sentences citing action/error/state evidence], "correction": [str; what the agent should have done at that step], "confidence": [float between 0 and 1] User: [task instruction] [full textual trajectory: per-step action type, action code, error, reward/done status, reasoning, and optional tool use] [result screenshots of the last N steps, when enabled] Identify the root error step and taxonomy tag. Output JSON only. Tool-augmented RCA system prompt <role> You are an expert GUI agent trajectory debugger. Use the provided tools to analyze a trajectory step by step, then submit your findings. </role> <workflow> 1. The user first message includes a step index showing each step’s action_type, whether it had an execution error, and whether a screenshot exists. Use it to pick which steps to inspect. 2. For each suspicious step, such as a step with an execution error, a step after a failed step, a repeated action, or the terminal step of a failed trajectory, call get_step_details. 3. get_step_details returns the input screenshot (what the agent saw before acting) and the result screenshot (screen state after the action). Compare them to identify perception, grounding, interaction, or outcome errors. 4. Look for patterns across steps: repeated errors without correction, stale variable references, wrong UI targets, or an environment that was not ready. 5. When you have inspected all relevant steps, call finish() with the complete structured RCA record. </workflow> <taxonomy> Use the CUA taxonomy appendix: P, G, R, S, and O top-level labels with their fine-grained subtype codes. </taxonomy> <annotation_rules> - Label the root-cause error, even when downstream effects manifest differently. - Perception vs. Grounding: misunderstanding visible content is P; understanding the content but targeting the wrong location or interacting incorrectly is G. - Prefer fine-grained subtype codes (P1, G2, R10, S4, etc.). </annotation_rules> ## RCA Mode: Find the Single Root Error Step Your task is Root Cause Analysis, not general debugging. Goal: identify the single earliest step N such that N <= F, where F is the Terminal Failure Step (the last step of a failed trajectory), and the mistake at step N is the direct cause of subsequent failures. Backward-tracing procedure: 1. Use get_step_details to inspect steps. It returns textual details plus the input screenshot and result screenshot. 2. Identify F = the terminal failure step from the trajectory summary. 3. Walk backwards from F. For each candidate step, call get_step_details to check whether it contains an independent mistake or merely inherited the failure from an earlier step. Compare the input screenshot against the action to identify perception or grounding errors. 4. Stop when you find the earliest step that introduced a new mistake. That is step N = root_error_step. 5. Call finish() with root_error_step=N, taxonomy_tag, evidence, correction, confidence, and per_step_summaries. Per-step summaries for rerollout: For every step inspected with get_step_details, include one entry in per_step_summaries: - intent_summary: infer the agent’s intended action from action code, reasoning, and tool use. - outcome_summary: infer the observable result by comparing the input screenshot 11 against the result screenshot and using execution metadata only as support. - summary_source: always set to " debugger_inspected". After each get_step_details result, write a short plain-text observation before requesting more tools. Older screenshots may be compressed out of history, but the written observation remains. Confidence guidance: - 0.9-1.0: clear single root cause with a strong causal chain. - 0.7-0.89: likely root cause with some ambiguity. - 0.5-0.69: multiple plausible candidates; this is the best guess. - < 0.5: weak evidence; still call finish and note the uncertainty. Initial user prompt and retrieved memories Perform Root Cause Analysis on this failed trajectory: [trajectory summary with task, app, step index, terminal status, and failure step] The trajectory is already loaded. You can directly call get_step_details to inspect steps. Work backwards from the Terminal Failure Step, then call finish() with the root_error_step, taxonomy_tag, evidence, correction, and confidence. If retrieved memories are available: Below are some past memories retrieved as potentially relevant based on semantic similarity. They are not guaranteed to apply; evaluate each one against the current trajectory before using it. If none match the current situation, ignore them and reason from scratch. Example k: [memory title, trigger condition, failed action, corrected action, distinguishing feature, evidence, taxonomy tag, and source episode reference] RCA tool schemas Tool: get_step_details Description: Get full details for a specific step: action code, execution error, agent reasoning, reward, done status, plus two screenshots: the input screenshot before the action and the result screenshot after the action. In RCA mode, these details are used to write intent_summary and outcome_summary. Input schema: "type": "object", "properties": "step_num": "type": "integer", "description": "The step number to inspect." , "required": ["step_num"] Tool: finish Description: Submit the final RCA result after identifying the root error step. Input schema: "type": "object", "properties": "root_error_step": "type": "integer", "description": "Earliest root-cause step N, where N <= terminal step F." , "taxonomy_tag": "type": "string", "enum": [ "P1", "P2", "P3", "P4", "P5", "G1", "G2", "G3", "G4", "R1", "R2", "R3", "R4", "R5", "R6", "R7", "R8", "R9", "R10", "R11", "R12", "R13", "S1", "S2", "S3", "S4", "S5", "S6", "S7", "O1", "P", "G", "R", "S", "O" ], "description": "Prefer a fine-grained subtype code." , "evidence": "type": "string", "description": "Grounded evidence from action code, errors, or screenshots." , "correction": "type": "string", "description": "Concrete action or strategy that should have been used." , "confidence": "type": "number", "minimum": 0.0, "maximum": 1.0, "description": "Confidence that this is the true root-cause step." , "per_step_summaries": "type": "array", "items": "type": "object", "properties": "step_num": "type": "integer", "intent_summary": "type": "string ", "outcome_summary": "type": " string", "summary_source": "type": "string", "enum": ["debugger_inspected"] , "required": [ "step_num", "intent_summary", "outcome_summary", "summary_source" ] , "required": [ "root_error_step", "taxonomy_tag", "evidence", "correction", "confidence", "per_step_summaries" ] When a lesson table is injected into the RCA prompt, the implementation exposes three extra re- trieval tools in addition toget_step_details andfinish. These tools return candidate evi- 12 dence only; the model is instructed to verify each retrieved lesson against the current screenshots and action trace. Additional RCA lesson tools Tool: lookup_lessons_by_taxonomy Purpose: Return additional distilled lessons that share a taxonomy code. Input: taxonomy_tag: subtype or top-level code, e.g. "G1" or "R10" top_k: optional integer in [1,10], default 3 Tool: search_lessons_by_app Purpose: Return lessons from a specific app, optionally filtered by taxonomy. Input: app_id: application identifier, e.g. "chrome " or "vs_code" taxonomy_tag: optional subtype or top-level code top_k: optional integer in [1,10], default 3 Tool: follow_episodic_ref Purpose: Resolve a lesson’s episodic_ref into a compact source-trajectory summary. Input: episodic_ref: UUID string carried by a retrieved lesson A.2 Re-Rollout Prompt Variants and Tools This appendix summarizes the re-rollout prompt variants used in our experiments. All variants re- play the original environment to the cutoff state before continuing the task; they differ only in what diagnostic context is prepended to the acting agent. The direct baseline receives the task and previ- ous action history, but no external RCA, taxonomy label, debugger summary, or correction. History-only re-rollout prompt RE-ROLLOUT CONTEXT Task: [task instruction] Previous result score: [score] (target: 1.0) The agent previously attempted this task for [ N] steps but did not complete it successfully. You are continuing from step [ cutoff+1]. PREVIOUS TRAJECTORY SUMMARY (Steps 1-[N]) Step 1: [OK/ERR/ ] [action_code] Error: [execution error, if any] ... NOTE: No repair recipe available for this trajectory. Review the step history above to identify what went wrong. RE-ROLLOUT INSTRUCTIONS - The environment is now in the state it was in after step [cutoff]. - You are starting from step [cutoff+1]. - Complete the task as efficiently as possible. The self-debug baseline receives the same re- played state and action history as the history-only baseline, but explicitly asks the acting model to di- agnose the previous attempt itself before choosing the next action. Self-debug re-rollout prompt SELF-DEBUG RE-ROLLOUT CONTEXT Task: [task instruction] Previous result score: [score] (target: 1.0) A previous attempt reached step [N] and failed or only partially succeeded. The VM has replayed that attempt through step [cutoff]. You are now continuing from the current screen state. PREVIOUS TRAJECTORY SUMMARY (Steps 1-[N]) Step 1: [OK/ERR/ ] [action_code] Error: [execution error, if any] ... SELF-DEBUG INSTRUCTIONS No expert RCA, taxonomy label, human correction, or debugger diagnosis is available. Do not assume the previous attempt was correct just because it was replayed. At your first inference step, inspect the current screen and briefly self-debug: 1. What was the previous attempt trying to accomplish? 2. What likely went wrong or remains unfinished? 3. Does the current screen contain side effects that need recovery? 4. What is the smallest concrete next action? Then continue the task from the current state. Avoid repeating a failed action pattern unless the screen clearly shows it is now valid. Call DONE only when the original task is visibly satisfied. Call FAIL only when the current environment prevents completing the requested task. Machine RCA, human-oracle RCA, and memory-augmented RCA use the same re-rollout template, with different RCA sources. Machine RCA fills the repair recipe from machine RCA without memory retrieval, the human-oracle row fills it from the human annotation, and our method uses the memory/RAG-augmented RCA package. RCA-guided re-rollout prompt RE-ROLLOUT CONTEXT Task: [task instruction] Previous result score: [score] (target: 1.0) The agent previously attempted this task for [ N] steps but did not complete it. You are continuing from step [cutoff+1]. PREVIOUS TRAJECTORY SUMMARY (Steps 1-[N]) Step 1: [OK/ERR/ ] [action_code] ... DEBUGGER-INSPECTED STEP SUMMARIES Step [k] [debugger_inspected] 13 Intent: [intent_summary from RCA] Outcome: [outcome_summary from RCA] REPAIR RECIPE Historical root error: Step [root_error_step] (error type: [taxonomy_tag]) [confidence: confidence] Rerollout start state: the VM has replayed the original trajectory through step [cutoff]. This is the current state to repair; inspect the screen before acting. Failed pattern to avoid: [RCA evidence] Do instead: [RCA correction] Next action policy: 1. First check whether the current screen still contains side effects from the failed path. 2. If recovery is needed, do the smallest recovery step before applying the correction. 3. Then execute the corrected approach directly; do not repeat the failed pattern above. Completion check: - The final visible/application state must satisfy the original task. - Save or apply changes when the task modifies a file or setting. - Once the success condition is visible, call DONE instead of continuing. All re-rollout variants use the base OSWorld computer-use action interface for mouse, keyboard, screenshot observation, and terminal DONE/FAIL actions. For RCA-guided re-rollout runs, the runner additionally invokes visual helpers after each action and appends their outputs as[SCREEN UPDATE step t] context. These are side-channel obser- vations generated by the runner, not free-form tools that the acting model calls directly. Re-rollout visual side-channel tools Base acting interface (all methods) computer-use actions: mouse clicks, keyboard input, screenshots terminal actions: DONE, FAIL smart_grid_caption_from_bytes(screenshot, query) Summarizes visible screen evidence relevant to the current action intent. local_visual_search_from_bytes(screenshot, query, target) Searches local screen regions for the intended target or related evidence. ground_text_bbox_from_bytes(screenshot, text_query) Returns bounding boxes for visible text matching the query. text_region_ocr_from_bytes(screenshot) Extracts visible text snippets and approximate screen locations. gui_grounding_from_bytes(screenshot, query, model, screen_w, screen_h) Algorithm 1: Retrieval-augmented debug- ging with episodic memory. Input:Training trajectoriesD train , new trajectoryτ, episodic memory M , retrieval size k, mode m∈RCA, re-rollout Output: An RCA report or re-rollout guidance for τ i ∈D train do r i ← RCA output or human annotation for τ i e i ← episode with task, app, steps, root-cause step, local error context, intention, tag, evidence, and correction ℓ i ← DISTILLMEMORY(e i ,r i ) M.ADD(ℓ i ) end q ← BUILDQUERY(τ,m) from the task instruction, app, observed state, suspected failed action, or tentative error type L← memory_query(M,q,k) for ℓ∈ L do inspect the current trajectory and compare ℓ’s trigger condition and distinguishing feature against the current evidence if ℓ matches the current failure pattern then use ℓ’s tag rationale, evidence, and correction as candidate guidance end end return a final RCA report if m = RCA; otherwise return re-rollout guidance that avoids the matched failed pattern Predicts target pixel coordinates for the current semantic action intent. Runner-added context: [SCREEN UPDATE step t] Screen: [caption/evidence] Target: [local visual-search evidence] Text bbox: [text grounding output] Visible text: [OCR snippets] Target pixel: ([x], [y]) A.3 Retrieval-Augmented Debugging Procedure A.4 Human Annotation Details, Instructions, and Consent Six trained researchers annotated the failed CUA trajectories used in CUAErrorBench. Before anno- tation, annotators were briefed on the goal of root- cause debugging, the CUA error taxonomy, and the distinction between a terminal failure symptom and the earliest causal mistake. Annotations were collected with a Streamlit-based front-end tool im- plemented indebugger/vis. The tool loads assigned tasks and linked trajectory artifacts from a trial directory, opens the selected trajectory directly for inspection, and exposes the task instruction, terminal status, debugger RCA proposal, optional recording video, and a step-by-step trajectory view with screenshots, click markers, action code, ex- 14 ecution metadata, accessibility-tree text, and the agent’s LLM reasoning. Annotators entered or updated the root-cause step, taxonomy label, evi- dence, correction, confidence, and notes directly in the interface; the tool saved per-annotator entries into the shared annotation files. Figure 5 shows the annotation platform used in this process. The interface was designed to make trajectory-level causal inspection practical for annotators: it presents the task, trajectory meta- data, agent reasoning, tool-use traces, screenshots, recordings, and annotation fields in one workspace, so annotators can move step by step through a failed agent trajectory and record the root cause without switching between separate files. Annotators were instructed to first verify the final failure symptom and then trace backward through the trajectory to identify the earliest step that intro- duced a new causal error rather than a downstream consequence of an earlier error. For each trajectory, annotators recorded the root-cause step, the most specific taxonomy label available, grounded evi- dence from the trajectory, an actionable correction, and a confidence value. Evidence was required to cite concrete trajectory information, such as visible UI state, selected elements, action code, execution errors, or mismatches between the agent’s stated intention and the observed screen outcome. Cor- rections were required to describe what the agent should have done differently at the root-cause step. Annotation assignments were load-balanced across annotators, with all 204 paper-usable ex- amples assigned to multiple annotators. When an- notations disagreed, annotators discussed the case and resolved it into a final decision used as the ref- erence label. All annotators were members of the research team, were informed that their annotations would be used for research and publication, and consented to this use. The annotation data contains task and trajectory-level labels rather than private information about annotators. A.5 Example Human Annotation Table 5 shows one CUAErrorBench example from an OSWorld LibreOffice Calc task. The exam- ple illustrates why root-cause annotation requires localizing the step where the trajectory becomes causally wrong rather than only describing the final failed spreadsheet. FieldAnnotation content TaskCompute monthly total sales in a new row named “Total” and create a line chart with months on the x-axis. Trajectory source OSWorld, LibreOffice Calc, 50-step CUA trajectory. Human agreement Two human annotators assigned the same root-cause step, taxonomy subtype, and high confidence. Root-cause step Step 21. Taxonomy la- bel P2: Misrecognition / OCR error. EvidenceThe agent reasoned that the “Points and Lines” chart subtype was already selected, but the screenshot showed that “Points Only” was highlighted. It then contin- ued through the chart wizard, producing a chart with points but no connecting lines. CorrectionSelect the “Points and Lines” subtype be- fore continuing in the chart wizard. Why this is root-cause su- pervision The final artifact is a wrong chart, but the root-cause step is the earlier visual recog- nition mistake in the chart-type dialog; re- pairing that step changes the downstream execution. Table 5: Example CUAErrorBench human annotation. A.6 CUA Error Taxonomy Subtypes Table 6 lists the fine-grained subtype definitions used for L2 evaluation and memory indexing. A.7 Additional Debugger Behavior Analysis Figure 6 compares debugger-predicted subtype fre- quencies against human labels on the main Claude split. 15 Figure 5: Annotation platform for CUAErrorBench. Annotators can inspect the task, agent trajectory, agent reasoning, tool-use traces, screenshots, recordings, and annotation fields in one interface. 16 CodeSubtypeDefinition P1Visual hallucinationThe agent perceives objects, text, or UI elements that are not present. P2Misrecognition / OCR errorRelevant content is visible, but the agent identifies or parses it incorrectly. P3Cross-modal misbindingThe agent incorrectly associates information across modalities, screen regions, or UI elements. P4Observation omissionThe agent fails to notice necessary visible information. P5Semantic misunderstandingThe agent sees the content correctly but misinterprets its meaning for the task. G1Coordinate / element ground- ing error The agent targets the wrong coordinates, UI element, DOM node, or spatial region. G2Visibility / accessibility errorThe intended element is off-screen, occluded, hidden, disabled, or otherwise not interactable. G3Interaction mechanics errorThe agent uses the wrong click type, drag behavior, gesture, text-entry method, or input sequence. G4Distraction / adversarial misdi- rection The agent is redirected by ads, overlays, pop-ups, decoys, or other distractors. R1Constraint violationThe agent ignores an explicit task constraint or requirement. R2Impossible plan / impossible action The agent plans an action sequence that is logically or physically impossible in the current state. R3Decomposition failure The agent decomposes the task into incorrect subgoals, missing steps, or the wrong order. R4Inefficient / redundant strategyThe plan is valid in principle but wastes steps or repeatedly pursues low-value actions. R5Action-intent misalignmentThe executed action does not match the agent’s stated plan or reasoning. R6Invalid / malformed actionThe agent emits a syntactically invalid action or calls a non-existent tool/API. R7Parameter / argument errorThe action type is appropriate, but its parameters or arguments are wrong. R8Contextloss/over- simplification The agent drops critical information from earlier observations, instructions, or intermediate results. R9Memory hallucinationThe agent asserts a false memory of a previous observation, result, or action. R10Progress misjudgmentThe agent incorrectly judges task completion, either stopping too early or failing to stop. R11Outcome misinterpretationThe agent misreads feedback from the environment after an action. R12Failed self-correctionThe agent detects a problem but applies an ineffective or incorrect fix. R13Causal misattributionThe agent explains the failure with the wrong cause and therefore chooses the wrong repair. S1Rendering / layout failureThe interface fails to render correctly or places elements in an invalid layout. S2Timing / race conditionEnvironment response timing causes an otherwise valid action to fail. S3Unexpected system behaviorOS dialogs, permission prompts, notifications, or unrelated system events interfere. S4Step / resource limitA viable strategy is blocked by step, token, time, rate, or compute limits. S5Tool / API failureAn external tool or API fails independently of the agent’s decision. S6Environment instabilityThe environment is buggy, non-deterministic, disconnected, or crashes. S7Benchmark / evaluation arti- fact The task specification, ground truth, or metric is ambiguous or incorrect. O1Infeasible-taskThe task is designed to be impossible to complete; the agent fails to recognize this and attempts the task anyway instead of reporting it as infeasible. Table 6: Fine-grained CUA error taxonomy. 17 0.000.020.040.060.080.100.120.140.160.18 Human frequency 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Model frequency G1 G3 O1 P2 P4 P5 R1 R10 R11 R3 R4 R5 R7 S7 Tag Frequency: Model vs Human Perfect match Figure 6: Subtype-frequency agreement with human labels. 18 A.8 Qualitative Debugger Case Studies Figures 7 to 10 show representative debugger case studies. They illustrate both aligned diagnoses, where the debugger matches the human root-cause label and step, and boundary cases, where the de- bugger predicts a plausible but different causal ex- planation. A.9 Generative AI Statement This work utilized generative AI tools to assist with formatting, generating LaTeX templates, and refin- ing word choice. The authors reviewed and verified all AI-assisted content to ensure factual accuracy and academic integrity. 19 Figure 7: Perception case study: human and debugger labels differ at the same root step. 20 Figure 8: Grounding case study: the debugger matches the wrong-toolbar-icon root cause. 21 Figure 9: Reasoning/control case study: the debugger matches premature success with unfinished work visible. 22 Figure 10: External/system case study: human and debugger diagnoses diverge on an evaluator-related failure. 23