Paper deep dive

LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures

Yunfei Zhang, Boyu Feng, Changhua Pei, Zexin Wang, Zhihuang Peng, Xinlong Liu, Hengyue Jiang, Difeng Ma, Jiayi Zhang, Yongzhou Yao, Yanan Zhao, Fei Sun, Yintong Huo, Zhaoyang Liu, Jingjing Li, Gaogang Xie, Dan Pei

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 64

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 8/21/2026, 2:52:52 AM

Summary

The paper introduces LongRCA Bench, a benchmark for diagnosing responsible roles and root causes in long-horizon agent failures, comprising 1,140 failed trajectories across five domains. It presents Root-Cause Trajectory Attribution (RCTA), a training-free method that achieves 51.1% responsible-role accuracy and 24.1% exact root-step accuracy, significantly outperforming baselines.

Entities (9)

LongRCA Bench · benchmark · 98%RCTA · method · 97%SWE-bench Pro · data-source · 95%Terminal Bench 2 · data-source · 95%TravelPlanner · data-source · 95%VitaBench · data-source · 95%WebArena Verified · data-source · 95%Responsible-Role Attribution · task · 92%Root Cause Localization · task · 92%

Relation Signals (9)

RCTA → achievesaccuracyon → Responsible-Role Attribution

confidence 96% · RCTA reaches 51.1% responsible-role accuracy

RCTA → achievesaccuracyon → Root Cause Localization

confidence 96% · RCTA reaches ... 24.1% exact root-step accuracy.

LongRCA Bench → containsdatafrom → TravelPlanner

confidence 95% · The benchmark contains 1,140 observed, non-injected failed trajectories from SWE-bench Pro, Terminal Bench 2, TravelPlanner...

LongRCA Bench → containsdatafrom → VitaBench

confidence 95% · The benchmark contains 1,140 observed, non-injected failed trajectories from SWE-bench Pro, Terminal Bench 2, TravelPlanner, VitaBench...

LongRCA Bench → containsdatafrom → WebArena Verified

confidence 95% · The benchmark contains 1,140 observed, non-injected failed trajectories from SWE-bench Pro, Terminal Bench 2, TravelPlanner, VitaBench, and WebArena Verified

LongRCA Bench → containsdatafrom → SWE-bench Pro

confidence 95% · The benchmark contains 1,140 observed, non-injected failed trajectories from SWE-bench Pro...

LongRCA Bench → containsdatafrom → Terminal Bench 2

confidence 95% · The benchmark contains 1,140 observed, non-injected failed trajectories from SWE-bench Pro, Terminal Bench 2...

LongRCA Bench → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Developers must then inspect the full execution to identify the responsible role and localize the earliest decisive root-cause step. Existing failure-attribution benchmarks largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored. We introduce LongRCA Bench, comprising 1,140 failed trajectories across five domains without injected errors. It provides independently scored human labels for the responsible role and earliest decisive root-cause step. The median trajectory contains 145 steps, and the strongest baseline reaches only 13.2% exact root-step accuracy. We further present Root-Cause Trajectory Attribution (RCTA), a training-free method that retrieves candidate error steps from segment summaries and traces them to available earlier handoff instructions. Using the same backbone, benchmark instances, and scoring protocol, RCTA reaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. These results highlight the need to evaluate responsible-role attribution and exact root-step localization as separate targets in long-trajectory failure diagnosis.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

60,357 characters extracted from source content.

Expand or collapse full text

LONGRCA BENCH: DIAGNOSING RESPONSIBLE ROLES AND ROOT CAUSES IN LONG-HORIZON AGENT FAILURES Yunfei Zhang 1,* , Boyu Feng 3,* , Changhua Pei 1,† , Zexin Wang 1 , Zhihuang Peng 1 , Xinlong Liu 2 , Hengyue Jiang 2 , Difeng Ma 1 , Jiayi Zhang 1 , Yongzhou Yao 4 , Yanan Zhao 3 , Fei Sun 4 , Yintong Huo 5 , Zhaoyang Liu 6 , Jingjing Li 1 , Gaogang Xie 1 , Dan Pei 7 1 Computer Network Information Center, Chinese Academy of Sciences 2 Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences 3 Chongqing University 4 Institute of Computing Technology, Chinese Academy of Sciences 5 Singapore Management University 6 Tongyi Lab, Alibaba Group 7 Tsinghua University * Equal contribution. † Corresponding author. ABSTRACT When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Devel- opers must then inspect the full execution to identify the responsible role and local- ize the earliest decisive root-cause step. Existing failure-attribution benchmarks largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored. We introduce LongRCA Bench, comprising 1,140 failed trajectories across five domains without injected errors. It provides independently scored human labels for the responsible role and earliest decisive root-cause step. The median trajectory contains 145 steps, and the strongest baseline reaches only 13.2% exact root-step accuracy. We further present Root-Cause Trajectory At- tribution (RCTA), a training-free method that retrieves candidate error steps from segment summaries and traces them to available earlier handoff instructions. Us- ing the same backbone, benchmark instances, and scoring protocol, RCTA reaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. These re- sults highlight the need to evaluate responsible-role attribution and exact root-step localization as separate targets in long-trajectory failure diagnosis. 1INTRODUCTION LLM-powered agents increasingly complete complex tasks through long sequences of planning, tool use, coordination, execution, and verification. When these executions fail, outcome-level evaluators identify the unsuccessful result but rarely reveal where the decisive error entered the trajectory. They also do not identify which workflow role was responsible. This diagnostic gap becomes especially consequential when hundreds of recorded steps separate the initial error from the failed outcome. We therefore formulate failure attribution as two independent prediction tasks. Root-cause localiza- tion identifies the earliest recorded step that introduced the decisive error relevant to the evaluator- confirmed failure. An error that was successfully repaired before the final failure is not selected. Responsible-role attribution identifies the workflow role responsible for that failure. A valid role la- bel must match a role name recorded in the source trajectory, which may include a human role. The role and root step are predicted and scored independently; the role is not derived from the emitter of the selected step. Later steps may execute, propagate, or expose an existing error, but they do not replace the step that introduced it. The SWE-bench Pro trajectory in Figure 1 illustrates this distinction (Deng et al., 2025). At step 37, a diagnostic agent issues a repair plan that uses an is sequence API inconsistent with the task requirement. The execution agent follows this plan, and the system reports completion at step 163. Only the post-run evaluator reveals the failure: none of the 47 required tests passed. The human annotation labels DiagnostAgent as the responsible role and marks step 37 as the earliest decisive root cause. It neither assigns responsibility to the later executor nor selects the terminal completion step as the root. The 126 subsequent steps constitute the root-to-end distance, which counts the recorded steps after the labeled root. 1 arXiv:2608.15242v2 [cs.AI] 20 Aug 2026 020406080100120140160 Trajectory step index Root cause (step 37): planner's repair plan violates a task constraint (wrong is_sequence API in the handoff) Final failure (step 163): hidden tests 0/47 pass 126 steps of downstream propagation baselines are attracted to the late executor / final error Case study: a delayed root cause with a planner → executor handoff (SWE Bench Pro, ansible) Figure 1: This example shows a LongRCA Bench trajectory from SWE-bench Pro. Each logged record contains a step index, name, message role, and content. The reference assigns the root to the DiagnostAgent handoff at step 37. Later records implement and verify the plan before the agent reports completion at step 163. The source evaluator reports that 0/47 required tests passed. The benchmark labels only the responsible role and root-cause step; later records illustrate the subse- quent execution. This paper studies offline root-cause attribution for completed, failed long-horizon agent trajectories. This setting creates two related challenges for reliable diagnostic methods. First, excessive trajectory length makes the few fault-relevant steps difficult to find. Second, the reference root may occur far before the recorded execution ends. A diagnostic method must therefore distinguish the decisive root from many subsequent steps. We introduce LongRCA Bench 1 to evaluate responsible-role attribution and exact root-cause lo- calization in long execution histories. The benchmark contains 1,140 observed, non-injected failed trajectories from SWE-bench Pro, Terminal Bench 2, TravelPlanner, VitaBench, and WebArena Verified (Deng et al., 2025; Merrill et al., 2026; Xie et al., 2024; He et al., 2026; El Hattami et al., 2025). These sources span five agentic domains and contain 178,137 recorded steps in total. Trajec- tory length is 156.3 steps on average, 145 at the median, and 728 at the maximum. For a trajectory indexed from 0 through T − 1, we define the root-to-end distance as (T − 1)− r, where r is the reference root step. This distance is 48 steps at the median, 183 at the 90th percentile, and 605 at the maximum. These statistics quantify how much recorded execution may follow the root before a failed trajectory ends. We further present Root-Cause Trajectory Attribution (RCTA), a training-free method for long- trajectory diagnosis. RCTA uses segment-level analysis to recall a small set of candidate error steps. It then traces each candidate to earlier handoff instructions that may contain the same error. This design separates broad trajectory search from focused comparison of the original logged text. We compare RCTA with five training-free baselines using the same inference backbone, 1,140 tra- jectories, and scoring protocol. RCTA achieves 51.1% responsible-role accuracy, 24.1% root-cause exact accuracy, and 37.4% root-cause±5 accuracy. These results exceed the strongest baseline by 23.6, 10.9, and 12.7 percentage points, respectively. Our contributions are as follows. • We introduce LongRCA Bench, which contains 1,140 observed, non-injected failed tra- jectories. Each trajectory has human labels for the responsible role and earliest decisive root-cause step. • We characterize the benchmark along three measurable dimensions: source diversity, tra- jectory length, and root-to-end distance. Together, these dimensions define a diagnostic regime in which the reference root step can be followed by hundreds of recorded steps before the failed execution ends. • We present RCTA, which combines segment-level candidate recall with backward tracing to earlier handoff instructions. Among the evaluated methods, RCTA achieves the strongest responsible-role attribution and root-cause localization results on LongRCA Bench. 1 https://huggingface.co/datasets/CLoud5-real/longrca-bench 2 BenchmarkFailure source NHuman coverage Prediction targetMean length HORIZONControlled > 3,100Sample (40)Failure mode– Who&WhenNatural184Full (184)Agent and decisive step 22.2 steps Who&When ProInjected12,326Sample (100)Agent, step, and mode 7.5 steps RootSENatural102Full (102)Earliest root step50.9 steps TraceElephantNatural220Full (220)Component and origin step 20.5–29.3 calls Failure as a ProcessNatural1,184Validated subset Four failure stages42 steps LongRCA BenchNatural1,140Full (1,140)Role and earliest root step 156.3 steps Table 1: Comparison of representative benchmarks for trajectory-level failure diagnosis. N denotes the released or analyzed trajectory count. Failure source describes how failed trajectories are ob- tained: Natural denotes failures arising during ordinary task execution, Injected denotes failures produced through explicit error injection, and Controlled denotes failures generated under prede- fined controlled conditions. Human coverage is reported as Full when all trajectories receive human review, Sample when only a reported sample is reviewed, and Validated subset when human vali- dation is reported for a subset. Mean length reports the average trajectory length in steps, or calls when specified. LongRCA Bench contains 1,140 fully annotated trajectories labeled by 22 CS grad- uate annotators. References: HORIZON (Wang et al., 2026b); Who&When (Zhang et al., 2025); Who&When Pro (Liu et al., 2026); RootSE (Wang et al., 2026a); TraceElephant (Chen et al., 2026); and Failure as a Process (Zhao et al., 2026). 2RELATED WORK 2.1BENCHMARKS AND EMPIRICAL RESOURCES FOR AGENT-FAILURE ANALYSIS Benchmarks for trajectory-level failure analysis differ in their target outputs and level of supervision. MAST categorizes recurring failure modes in multi-agent traces, whereas TRAIL labels erroneous spans and their effects (Cemri et al., 2025; Deshpande et al., 2025). AgentRx identifies the first step after which recovery is no longer possible and assigns a root-cause category (Barke et al., 2026). Failure as a Process records a coding agent’s initial deviation, failure inevitability, failure observation, and eventual recovery (Zhao et al., 2026). HORIZON uses controlled tasks to study how task horizon affects success and aggregate failure patterns (Wang et al., 2026b). Together, these resources characterize failure through modes, spans, stages, or horizon-level patterns. They do not share a common target for responsible-role attribution and earliest-step localization. Other resources supervise the responsible entity, causal step, or both. Who&When directly evaluates responsible-agent and decisive-step attribution (Zhang et al., 2025). Who&When Pro extends this interface through controlled error injection and audits a stratified subset of the constructed labels (Liu et al., 2026). MP-Bench applies an ambiguity-aware annotation protocol to failure-inducing steps (In et al., 2026). TraceElephant pairs responsible-component labels with origin-step labels across traces from three agent systems (Chen et al., 2026). TrajAudit introduces RootSE, which provides human labels for the earliest decisive error step and a diagnostic justification in repository- level coding trajectories (Wang et al., 2026a). These resources more closely match the prediction target of LongRCA Bench but differ in failure source, supervision, scale, or history length. Table 1 compares these resources along five dimensions: failure source, size, human review, pre- dicted output, and history length. LongRCA Bench contains 1,140 observed failures with retained human annotations for the responsible role and earliest decisive root-cause step. It places these labels within substantially longer recorded histories. 3 2.2METHODS FOR TRAJECTORY DIAGNOSIS Methods for trajectory diagnosis differ in how they search recorded histories and validate candidate error steps. Prompting-based approaches operate directly on logged text or compact trajectory repre- sentations. Who&When introduced all-at-once, step-by-step, and binary-search prompting for agent and step attribution (Zhang et al., 2025). ECHO combines hierarchical context representations with consensus voting (Banerjee et al., 2025). RAFFLES alternates between a central judge and special- ized evaluators to refine step-level fault hypotheses (Zhu et al., 2026a). AgentRx synthesizes task constraints and checks individual steps before predicting a critical step and category (Barke et al., 2026). For traces that exceed a model’s effective context budget, SAFARI uses selective trace access and persistent short-term memory instead of full-context ingestion (Zhu et al., 2026b). Structure-aware methods exploit execution dependencies or evidence from previous runs. FAMAS replays failed tasks and applies spectrum-based localization over agent–action–state triples (Ge et al., 2025). CDC-MAS combines causal and Shapley-based analysis to attribute roles and steps (Ma et al., 2025). CHIEF constructs a hierarchical causal graph and applies backtracking with counterfactual screening (Wang et al., 2026c). FALAT searches dependencies among decisions, tool outputs, and messages (Rafi et al., 2026). CORRECT uses a different source of evidence by retrieving compact error schemata distilled from earlier failures (Yu et al., 2025). These methods address different diagnostic bottlenecks and produce different outputs. Some return only a critical step or failure category, whereas others return both a role and a step. LongRCA Bench independently evaluates a responsible role and a root-cause step on long trajectories. Our experiments compare representative strategies based on direct prompting, hierarchical context, and dependency search. 3LONGRCA BENCH LongRCA Bench is constructed through four stages: selecting evaluator-confirmed failed execu- tions, mapping heterogeneous logs to a stable trajectory representation, collecting human root-cause annotations, and applying validity checks and evidence-based label resolution. This section defines the resulting task and documents the construction decisions that make exact cross-source attribution possible. 3.1TASK DEFINITION Each benchmark instance contains a task instruction, a failed trajectory H = (h 0 ,...,h T−1 ), and the outcome supplied by the source benchmark evaluator. Each recorded step has a stable 0-based index and identifies its source role. We use responsible role for the recorded workflow role to which the failure is attributed. The reference role must appear in the trajectory, but it need not emit the selected root-cause step. A method returns a responsible role ˆρ and a root-cause step ˆr ∈ 0,...,T − 1. These outputs are predicted and scored independently. We evaluate ˆρ as the method’s explicit role prediction and never infer it from ˆr. The reference labels (ρ ∗ ,r ∗ ) follow the same separation. External subjects that are not recorded workflow roles fall outside the task’s output space. A trajectory-grounded rationale is retained with each reference for audit and review, but it is not scored. Operationally, r ∗ is the earliest recorded step that introduces the decisive error relevant to the final evaluator-confirmed failure. The error must remain unrepaired before that failure. Earlier errors that are successfully repaired are excluded, as are later steps that merely execute, propagate, or expose an existing error. Under a handoff, the instruction step is selected when it already contains the decisive error; a later recipient step is selected only when it departs from the instruction or introduces a new decisive error. The complete decision rules are provided in Appendix D. LongRCA Bench evaluates both outputs after the failed trajectory is complete; it does not evaluate online failure detection. 4 3.2SOURCE CURATION AND TRAJECTORY NORMALIZATION We collect completed executions from SWE-bench Pro, Terminal Bench 2, TravelPlanner, VitaBench, and WebArena Verified (Deng et al., 2025; Merrill et al., 2026; Xie et al., 2024; He et al., 2026; El Hattami et al., 2025). An execution is retained only when the original benchmark evaluator marks the task outcome as failed. We do not inject synthetic errors. Infrastructure, smoke- test, and debug runs are excluded because they do not represent completed task attempts. After validity filtering and cross-stage deduplication, the release contains 1,140 unique failed trajectories. The five sources cover software repair, terminal tasks, travel planning, service-oriented tool use, and web interaction. They also span fixed-role teams, group-chat coordination, and sequential planner– executor–critic workflows. This diversity matters for attribution: the eligible role set, handoff pat- tern, and placement of plans, tool results, and verifier messages all depend on the agent organiza- tion. Table 2 summarizes the retained coverage. The trajectories were generated by MiniMax-M2.5, Kimi-K2.5, and Qwen3.5-Plus; framework-level provenance and generator counts are reported in Appendix B. SourceNTask domainAgent organization SWE-bench Pro128Software repairDiagnosis–execution–verification team Terminal Bench 242Terminal tasksDiagnosis–execution–verification team TravelPlanner685Travel planningMagentic-One group chat VitaBench108Service-oriented tool use Sequential agent or planner–critic–action team WebArena Verified177Web interactionPlanner–web-surfer–critic workflow Total1,1405 task domains Table 2: Task and execution coverage of LongRCA Bench. The five sources span distinct task domains and include fixed-role teams, group-chat coordination, and sequential agent organizations. The source frameworks expose incompatible log structures: some organize a run as a multi-agent dialogue, whereas others interleave planning messages, tool calls, observations, browser or terminal interactions, and evaluator feedback. Annotating these logs directly would make an exact step label depend on source-specific logging conventions. We therefore convert every retained run to the same ordered history. Each record contains a 0-based step index, its recorded role name, and its original content; tool and verifier metadata are retained when available. Normalization preserves the original temporal order and textual evidence. It does not invent inter- mediate actions or split an indivisible source event into artificial steps. The resulting history is the shared coordinate system used by annotators, released references, and evaluated methods. Each in- stance also stores the task instruction and evaluator outcome, while the responsible-role, root-step, and rationale fields are withheld during prediction. The field-level schema appears in Appendix A. 3.3HUMAN ANNOTATION AND QUALITY CONTROL Root-cause attribution over long trajectories requires more than locating the last visible symptom. Annotators must distinguish the first decisive error from its later execution, propagation, or detec- tion. We recruited 22 master’s and doctoral students in computer science, and the same annotator pool participated in both stages of the campaign. The first stage calibrated the protocol on 100 trajectories. Annotators applied the preliminary in- structions, discussed recurrent boundary cases, and used the results to refine the operational rules for repaired errors, erroneous handoffs, downstream execution, and missed verification. This stage also verified that the required evidence was available in the normalized histories and established the expected annotation cost. During the full-scale stage, annotators independently reviewed the task instruction, complete step- indexed history, and source evaluator outcome. They returned three fields: the responsible role, the earliest decisive root-cause step, and a written rationale tied to trajectory evidence. Auxiliary diagnostic summaries could be used to navigate a long history, but they were kept separate from the 5 SourcesConstruction PiplineFinal Dataset Source failed runs SWE-bench Pro Terminal Bench 2 TravelPlanner ViaBench WebArea Verified Filtering Gates Failure verification Valid failure confirmation Remove invalid /non- actionable cases History Standar- dization Trace normalization De-duplicate noise Canonicalize events & timestamps Instance Packaging Step-indexed packages Segment into logical steps Index steps (1...N) Attach metadata & artifacts Multi-round Annotation, Adjudication & Quality Control Independent annotation × 3 Annotator 1 Responsible actor Root-cause step Rationale Agreement / consistency check Majority agreement Consistency metrics Disagreement or ambiguity GPT-5.5-assisted adjudication Resolve disagreements Propose consolidated labels & rationale Annotator 2 Responsible actor Root-cause step Rationale Annotator 3 Responsible actor Root-cause step Rationale re-annotate Human expert review (senior annotator) Validate correctness Ensure rationale sufficiency & plausibility Final quality control Spot audits & guideline compliance Outlier and leakage checks LongRCA Bench 1,140 instances High-quality adjudicated root-cause annotations Standardized histories Cleaned, deduplicated, canonicalized traces Step-indexed packages Structured steps with artifacts & metadata Responsible actor label Who is ultimately responsible Root-cause step + rationale Precise step index and human-validated rationale Quality-Controlled Trace Curation Figure 2: Construction and annotation workflow for LongRCA Bench. Source runs are filtered and normalized into step-indexed histories before annotation, label resolution, and final quality control. human fields and were never promoted automatically to reference labels; the original trajectory and evaluator outcome remained the evidence of record. The full-scale campaign collected 1,444 annotations over 1,100 trajectories, for an average of 1.31 annotations per trajectory. A complete annotation typically required 30–40 minutes, corresponding to approximately 722–963 annotator-hours for the full-scale submissions alone, excluding calibra- tion and label-resolution effort. Quality control combines structural validation with evidence review. For every submitted annota- tion, we check that the selected step is an integer within the trajectory range and that the selected role occurs in the recorded role set. The accompanying rationale must identify evidence sufficient to support the proposed causal interpretation. Records with invalid references, evident annotation defects, or insufficient evidence are excluded rather than repaired by changing the submitted step automatically. The redundantly annotated subset provides a direct measure of task ambiguity. Pairwise exact agree- ment is 65.9% for the responsible role, 39.5% for the root-cause step, and 38.4% for the joint role– step label. Exact step agreement is deliberately strict: two annotations disagree whenever they iden- tify different recorded indices, even when the steps occur near each other. We use these agreement values as diagnostics for where human review is required, not as an automatic rule for accepting a label. Table 3 summarizes the scale and reliability evidence of the campaign. Annotation statisticValue Graduate annotators22 Calibration trajectories100 Full-scale annotations1,444 Full-scale trajectories1,100 Average annotations per trajectory1.31 Exact agreement: role / step / joint65.9% / 39.5% / 38.4% Typical time per annotation30–40 min Final released trajectories1,140 Table 3: Annotation scale and quality-control evidence. Agreement is computed on the redundantly annotated subset using exact role and exact step matches. For multiply annotated trajectories, valid submissions are compared field by field. Exact matches are retained as consensus references. Disagreements are reviewed against the original task, complete history, source evaluator outcome, and written rationales under the same earliest-decisive rules used during annotation. Resolution selects an evidence-supported human annotation; it does not average 6 step indices, infer the role from the selected step’s emitter, or replace a human label with an auxiliary model suggestion. Finally, we rerun the role and step validity checks, remove duplicate trajectories across the two annotation stages, and retain a single reference record per trajectory. This produces 1,140 human- labeled and quality-controlled failed trajectories. 4DATASET ANALYSIS We characterize LongRCA Bench by source, agent organization, generator model, trajectory length, and root-to-end distance. 4.1SOURCE, AGENT ORGANIZATION, AND GENERATOR MODEL COVERAGE LongRCA Bench contains 1,140 failed trajectories from five domains: travel planning, web inter- action, software repair, service-oriented tool use, and terminal tasks. TravelPlanner contributes 685 trajectories (60.1%), while the other four sources contribute 455 (39.9%). The associated agent or- ganizations include fixed-role teams, group-chat coordination, and sequential agent organizations (Table 2). The trajectories were generated by three models: MiniMax-M2.5 (681), Kimi-K2.5 (276), and Qwen3.5-Plus (183). Source-level profiles also vary in trajectory length and root-to-end distance. Median trajectory length ranges from 51 steps in VitaBench to 257.5 in SWE-bench Pro. Median root-to-end distance ranges from 24 to 186 steps across sources. LongRCA Bench therefore spans five task domains, several agent organizations, three generator models, and a broad range of execu- tion lengths. 4.2TRAJECTORY LENGTH AND ROOT-TO-END DISTANCE Across all trajectories, the benchmark contains 178,137 recorded steps. Trajectory length averages 156.3 steps, with a median of 145 and a maximum of 728. The reference root occurs at median step 55, while root-to-end distance has a median of 48 steps, a 90th percentile of 183, and a maximum of 605. In 559 trajectories (49.0%), more than 50 steps follow the reference root. In 324 trajectories (28.4%), more than 100 steps follow it; in 88 trajectories (7.7%), more than 200 steps follow it. Together, these distributions show that a method may need to distinguish the decisive root from hundreds of subsequent recorded steps (Figure 4). Root-to-end distance measures the number of recorded steps after the annotated root; it is not a labeled propagation-chain length. The stratified analysis in Section 6 is descriptive and does not establish a causal effect of distance on model accuracy. 5ROOT-CAUSE TRAJECTORY ATTRIBUTION Long trajectories complicate root-cause attribution because fault-relevant evidence is dispersed across many steps. Moreover, an error observed late in the trajectory may originate in an earlier instruction. RCTA therefore separates broad candidate recall from evidence-based final attribution. A segment is a consecutive block of recorded trajectory steps. A handoff instruction is a logged message, marked as X (-> Y), in which role X instructs role Y . Given a failed trajectory, RCTA applies three stages: (1) partitioning the trajectory, (2) recalling candidate error steps from segment summaries, and (3) retrieving relevant earlier handoff instructions. For executor and verifier candidates, the retrieved instruction must be addressed to the candidate’s role. Other candidates receive the nearest earlier handoff as plan context when one is available. Segment summaries support broad search, whereas final attribution uses the original logged text. Appendix E reports the prompt templates, implementation thresholds, and output checks. 7 Source composition of LongRCA (N = 1,140) TravelPlanner685 (60.1%) WebArena Verified177 (15.5%) SWE Bench Pro128 (11.2%) VitaBench108 (9.5%) Terminal Bench 242 (3.7%) 0200400600800 Failed trajectories (N) Figure 3: Source composition of LongRCA Bench, with counts and percentages summing to 1,140 trajectories and 100%, respectively. Total trajectory steps Adjudicated root-cause step Distance from root cause to end 0 100 200 300 400 500 600 700 Steps P90 268 med 145 max 728 P90 179 med 55 max 521 P90 183 med 48 max 605 a Trajectory length and root-cause delay Median Mean Min–P90 >50 steps>100 steps>200 steps Root cause precedes final failure by 0 20 40 60 Share of trajectories 49.0% (n=559) 28.4% (n=324) 7.7% (n=88) b Temporal separation of cause and failure Figure 4: Long-horizon characteristics of LongRCA Bench: Panel (a) summarizes trajectory length, reference root step, and root-to-end distance. Circles and diamonds mark medians and means, thick ranges span the minimum to the 90th percentile, and a shared note lists the three maxima. Panel (b) reports the fractions of trajectories with more than 50, 100, or 200 steps after the reference root. 5.1SEGMENTING THE TRAJECTORY RCTA partitions the trajectory into consecutive segments while preserving the original step IDs. Rule-based boundaries enforce character and step limits and may align with logged completion or handoff transitions. Each segment includes up to five preceding steps as overlap context, preserving information near the boundary. Segmentation uses no language model; Appendix E provides the exact boundary rules. 5.2RECALLING CANDIDATE ERROR STEPS One LLM call per segment produces a summary and proposes candidate error steps with their recorded IDs. A second call combines adjacent segment summaries into a trajectory outline or- ganized by subgoal. The resulting outline summarizes the segments rather than repartitioning the raw trajectory. RCTA pools and prioritizes the cited candidate IDs, then retrieves the original text of every retained step. The outline provides global progress information, while the original step text supplies evidence for final attribution. 8 Trajectory Representations Task Brief Step Sequence Action State Hierarchical Evidence Abstraction Step-level Segment-level Phase-level Whitelist inputs Allowed agents Schema constraints Step 1Step 2Step 3Step n 123n... ... Agent / Action State Planner ExecutorVerfier agent: action: signal: ... ... Candidate Evidence Retrieval Delegate Linkage Analysis Root-vs-Manifestation Localization root propagation manifestation earliertime later Leakage & Schema Guardrails Evidence-Grounded Attribution Engine ... Planner ExecutorVerfier delegates ... Target Diagnostics Root-Cause Agent / Step Failure Chain root propagation manifestation Review Summary FormalizationAbstraction Evidence-Grounded Attribution Diagnosis Figure 5: RCTA partitions a long trajectory into consecutive segments and uses their summaries to recall candidate error steps. It then compares the original candidate text with relevant earlier handoff context before independently predicting the responsible role and earliest decisive root-cause step. 5.3TRACING CANDIDATES TO HANDOFF INSTRUCTIONS For a candidate produced by an executor or verifier role Y , RCTA retrieves the nearest preceding handoff instruction addressed to Y when available. For other candidates, it retrieves the nearest earlier handoff as plan context. The final LLM call compares the original text of each retained candidate with the retrieved handoff context. If an instruction already contains the decisive error and the later step carries it out without repair, RCTA selects the instruction step as the root. If the later step departs from the instruction or in- troduces a new decisive error, the candidate step remains the root. A successfully repaired error is excluded as the root of the evaluator-confirmed failure. The final decision relies only on evidence recorded in the trajectory. RCTA returns the responsible role and root-cause step in separate output fields, so root selection does not determine the role prediction. Before scoring, a programmatic validator checks that the predicted role occurs in the trajectory. It also verifies that the root and supporting IDs refer to recorded steps and that quoted handoff text occurs in the referenced instruction. These checks establish output validity and textual provenance, not the semantic correctness of the diagnosis. Each invalid response receives one retry with programmatic validation feedback. If the retry remains invalid, unsupported fields are removed and the output is flagged for review or marked as an abstention. 6EVALUATION 6.1EXPERIMENTAL SETTINGS We evaluate responsible-role attribution and root-cause-step localization independently on all 1,140 failed trajectories. Responsible-role accuracy compares each method’s explicit role output ˆρ with the reference role ρ ∗ . Before comparison, we normalize case, whitespace, and explicit handoff suffixes. The metric never infers a role from the name at the predicted root step. A missing role or one absent from the trajectory’s recorded role set is counted as incorrect. This outcome does not affect either root-step metric. Root-cause exact accuracy requires ˆr to equal the 0-based reference step r ∗ . Root-cause ±5 accuracy allows an absolute error of at most five steps. A non-numeric root or one outside the recorded trajectory is incorrect for both step metrics. This outcome does not affect responsible-role accuracy. 9 MethodRole Acc.↑Root Exact↑Root±5↑Root MAE↓ All-at-once26.2%7.6%19.9%55.9 Step-by-step22.2%5.3%16.9%52.3 Binary search23.0%3.4%13.3%61.7 ECHO27.5%13.2%24.7%50.4 FALAT19.0%2.8%12.5%66.6 RCTA (ours)51.1%24.1%37.4%38.6 Table 4: Full-benchmark results on all 1,140 trajectories. All methods use DeepSeek-V4-Flash. Role accuracy scores each method’s explicit role prediction independently of its predicted root step. Root MAE is the source-weighted valid-output mean absolute step error. Higher values are better for the three accuracy metrics, whereas lower values are better for MAE. We also report source-weighted valid-output root MAE. Let V b be the trajectories in source b with a valid numeric step prediction, N b the total number of trajectories from that source, and N = P b N b . The metric is 1 N X b N b 1 |V b | X i∈V b |ˆr i − r ∗ i | ! . Missing or invalid step outputs are incorrect for both root-step accuracy metrics, but they do not contribute a direct error term to MAE. Exact matching is the primary localization metric. A fixed tolerance is difficult to interpret across trajectories whose lengths differ by an order of magnitude. We compare RCTA with representative training-free baselines under matched conditions. All-at- once presents the full trajectory to the inference backbone and requests a responsible role and de- cisive step in one call. Step-by-step scans the trajectory sequentially and asks whether the decisive error has occurred. Binary search recursively bisects the trajectory to locate the root-cause region. ECHO uses hierarchical context and consensus voting (Banerjee et al., 2025). FALAT applies dependency-guided search to decisions, tool outputs, and messages (Rafi et al., 2026). All six meth- ods use DeepSeek-V4-Flash, the same 1,140 trajectories, and the same LongRCA Bench scoring rules. 6.2FULL-BENCHMARK RESULTS Exact root-step localization remains difficult. Even the strongest method reaches only 24.1% exact accuracy, compared with 51.1% responsible-role accuracy. RCTA therefore identifies the responsible workflow role more often than it localizes the exact earliest decisive step. RCTA improves both role attribution and root localization. RCTA performs best on all four reported metrics. ECHO is the strongest baseline, with 27.5% role accuracy, 13.2% exact root accuracy, 24.7% within-five accuracy, and a root MAE of 50.4. Relative to ECHO, RCTA gains 23.6, 10.9, and 12.7 percentage points on the three accuracy metrics. It also reduces MAE by 11.8 steps. Dependency-guided search alone is insufficient in this setting. Under the matched backbone and benchmark protocol, FALAT attains 2.8% exact root accuracy and 12.5% within-five accuracy. Both results are below those of all-at-once prompting. This comparison is limited to the evaluated implementation and does not characterize dependency-based diagnosis methods in general. 6.3PERFORMANCE BY TRAJECTORY LENGTH AND ROOT-TO-END DISTANCE We next stratify root-cause exact accuracy by two benchmark characteristics introduced in Section 3. The trajectory-length bins are≤ 100, 101–200, 201–400, and > 400 steps. The root-to-end-distance bins are≤ 10, 11–50, 51–100, and > 100 steps. For a trajectory indexed from 0 through T − 1 with reference root r ∗ , root-to-end distance equals (T − 1)− r ∗ . Source domain, workflow, trajectory length, and root position are correlated. We therefore interpret this stratified analysis descriptively rather than causally. 10 <=100 (n=393) 101--200 (n=467) 201--400 (n=238) >400 (n=42) Difficulty bin 0 5 10 15 20 25 30 35 40 45 Root-cause exact accuracy (%) (a) Trajectory length <=10 (n=256) 11--50 (n=325) 51--100 (n=235) >100 (n=324) Difficulty bin (b) Root-to-end distance All-at-onceStep-by-stepBinary searchECHOFALATRCTA Figure 6: Root-cause exact accuracy by (a) trajectory length and (b) root-to-end distance. Error bars show instance-level 95% Wilson intervals. Each x-axis label reports the number of trajectories in the corresponding bin. The pooled patterns are descriptive because source composition differs across bins. Figure 6 shows that exact accuracy drops after the shortest trajectory bin for most methods. RCTA decreases from 30.3% for trajectories of at most 100 steps to 20.3% for 101–200 steps and 20.2% for 201–400 steps. The > 400 bin has 31.0% accuracy, but it contains only 42 trajectories and has a different source composition. Results across root-to-end-distance bins are also non-monotonic. RCTA obtains 21.5%, 27.1%, 20.9%, and 25.6% across the four bins. These pooled results do not identify either trajectory length or root-to-end distance as an isolated causal factor. 7DISCUSSION AND LIMITATIONS LongRCA Bench extends agent-failure attribution to long execution histories.By scoring responsible-role attribution and root-cause-step localization independently, the benchmark separates two diagnostic capabilities that can otherwise be conflated. The evaluation shows that RCTA identi- fies the responsible workflow role more often than it localizes the exact earliest decisive event. This separation prevents success on role attribution from obscuring weaknesses in event localization. Several limitations define how these conclusions should be interpreted and applied. Annotation. Root-cause labeling requires careful human judgment across long execution trajectories. Lon- gRCA Bench provides one finalized responsible-role label, one finalized root-cause-step label, and a human-written rationale for every trajectory. The two labels are scored independently, whereas the rationale documents supporting evidence but is not scored. Label scope. The benchmark scores only the responsible role and earliest decisive root-cause step. Any intermediate causal chain generated by RCTA remains an unscored explanation rather than a benchmark target. Accordingly, the reported results do not measure full causal-chain reconstruction. Evaluation setting. LongRCA Bench evaluates diagnosis from the full log after a failed trajectory has ended. It therefore does not test either early warning or online intervention. These results should not be compared directly with protocols that reward detection before a failure has fully unfolded. Baselines. All methods use the same inference backbone, which controls this source of varia- tion. Absolute performance may nevertheless change with a stronger inference backbone. The current comparison therefore supports conclusions under a matched backbone rather than model- independent performance claims. Difficulty stratification. The length and root-to-end bins combine source domains with differ- ent workflows, role structures, and failure distributions. The observed trends establish associations 11 within LongRCA Bench, not isolated effects of either variable. Controlled matched-source experi- ments are required before causal conclusions can be drawn. These boundaries also motivate several targeted follow-up experiments for RCTA. Component ab- lations could isolate the contributions of candidate recall, candidate-to-handoff tracing, backward instruction checks, and validation. Matched-backbone evaluations of additional attribution methods would provide broader evidence about relative performance. 8CONCLUSION We introduced LongRCA Bench to evaluate responsible-role attribution and root-cause-step local- ization in long-horizon agent failures. The benchmark contains 1,140 failed trajectories with retained human annotations from five heterogeneous source benchmarks. In many trajectories, the reference root cause is followed by a long sequence of downstream steps before execution ends. We also presented RCTA, a diagnostic pipeline that narrows long trajectories to candidate error steps. It compares each candidate with relevant earlier handoff instructions and returns separately validated responsible-role and root-cause-step predictions. Under the same backbone, benchmark instances, and scoring protocol, RCTA achieves the strongest performance on both targets among the evaluated methods. Specifically, it reaches 51.1% role accuracy and 24.1% exact root-step accuracy. Together, the benchmark and results support treating responsibility and root-cause localization as explicit targets when diagnosing completed agent failures. Final-outcome evaluation alone cannot provide either form of diagnostic evidence. REFERENCES Adi Banerjee, Anirudh Nair, and Tarik Borogovac. Where did it all go wrong? a hierarchical look into multi-agent error attribution. arXiv preprint arXiv:2510.04886, 2025. URL https: //arxiv.org/abs/2510.04886. Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal. Agen- tRx: Diagnosing AI agent failures from execution trajectories. arXiv preprint arXiv:2602.02475, 2026. URL https://arxiv.org/abs/2602.02475. Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail?In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2503.13657. Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang, Zhe Liu, Huanxiang Feng, and Qing Wang. Seeing the whole elephant: A benchmark for failure attribution in LLM-based multi-agent systems. In Proceedings of the 64th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), p. 19888–19905. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.acl-long.912. URL https://aclanthology.org/2026. acl-long.912/. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. URL https://arxiv.org/abs/2509.16941. Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. TRAIL: Trace reasoning and agentic issue localization. arXiv preprint arXiv:2505.08638, 2025. URL https://arxiv.org/abs/2505.08638. Amine El Hattami, Megh Thakkar, Nicolas Chapados, and Christopher Pal. WebArena Verified: Reliable evaluation for web agents. In Workshop on Scaling Environments for Agents, 2025. URL https://openreview.net/forum?id=94tlGxmqkN. 12 Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang. Who is introducing the failure? au- tomatically attributing failures of multi-agent systems via spectrum analysis. arXiv preprint arXiv:2509.13782, 2025. URL https://arxiv.org/abs/2509.13782. Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Dunwei Tu, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, Man Gao, Xi Su, Xiaodong Cai, Xunliang Cai, Yu Yang, and Yunke Zhao. VitaBench: Benchmarking LLM agents with versatile interactive tasks in real-world applications. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2509.26490. Yeonjun In, Md Mehrab Tanjim, Jayakumar Subramanian, Sungchul Kim, Uttaran Bhattacharya, Wonjoong Kim, Sangwu Park, Somdeb Sarkhel, and Chanyoung Park. Rethinking failure attri- bution in multi-agent systems: A multi-perspective benchmark and evaluation. arXiv preprint arXiv:2603.25001, 2026. URL https://arxiv.org/abs/2603.25001. Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu, and Huazheng Wang. Who&when pro: Can LLMs really attribute failures in AI agents? arXiv preprint arXiv:2607.09996, 2026. URL https://arxiv.org/abs/2607.09996. Guoqing Ma, Jia Zhu, Hanghui Guo, Weijie Shi, Jiawei Shen, Jingjiang Liu, and Yidan Liang. Automatic failure attribution and critical step prediction method for multi-agent systems based on causal inference. arXiv preprint arXiv:2509.08682, 2025. URL https://arxiv.org/abs/ 2509.08682. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. Terminal- Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2601. 11868. Md Nakhla Rafi, Md Ahasanuzzaman, Dong Jae Kim, Zhijie Wang, and Tse-Hsun Chen. Falat: Tracing failures in llm agent trajectories via dependency-guided search. arXiv preprint arXiv:2606.00765, 2026. URL https://arxiv.org/abs/2606.00765. Minxing Wang, Xiaofei Xie, and Yintong Huo. Trajaudit: Automated failure diagnosis for agentic coding systems. arXiv preprint arXiv:2605.26563, 2026a. Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun, Haorui Wang, Shuibai Zhang, Wenjie Hu, Mya Schroder, Bilge Mutlu, Dawn Song, and Robert D. Nowak. The long-horizon task mirage? di- agnosing where and why agentic systems break. arXiv preprint arXiv:2604.11978, 2026b. URL https://arxiv.org/abs/2604.11978. Yawen Wang, Wenjie Wu, Junjie Wang, and Qing Wang. From flat logs to causal graphs: Hierar- chical failure attribution for LLM-based multi-agent systems. arXiv preprint arXiv:2602.23701, 2026c. URL https://arxiv.org/abs/2602.23701. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. TravelPlanner: A benchmark for real-world planning with language agents. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, p. 54590–54613. PMLR, 2024.URL https: //proceedings.mlr.press/v235/xie24j.html. Yifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, and Bryan Wang. Cor- rect: Condensed error recognition via knowledge transfer in multi-agent systems. arXiv preprint arXiv:2509.24088, 2025. URL https://arxiv.org/abs/2509.24088. Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 76583–76599. PMLR, 2025. URL https://proceedings.mlr. press/v267/zhang25cq.html. 13 Xiangxin Zhao, Han Li, Shuaiting Li, Tianyi Zhao, Earl T. Barr, Federica Sarro, and He Ye. Failure as a process: An anatomy of CLI coding agent trajectories. arXiv preprint arXiv:2607.09510, 2026. URL https://arxiv.org/abs/2607.09510. Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Yuhui Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, and Daben Liu. RAFFLES: Reasoning-based attribution of faults for LLM systems. In Proceedings of the 19th Conference of the European Chapter of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), p. 7659–7688. Association for Computational Linguistics, 2026a. doi: 10.18653/v1/2026.eacl-long.359. URL https: //aclanthology.org/2026.eacl-long.359/. Chenyang Zhu, Jiayu Yao, Kushal Chawla, Youbing Yin, Nathan Wolfe, Pengshan Cai, Jingyu Wu, Spencer Hong, Sangwoo Cho, Shi-Xiong Zhang, Daben Liu, Sambit Sahu, and Erin Babinsky. SAFARI: Scaling long horizon agentic fault attribution via active investigation. arXiv preprint arXiv:2606.24626, 2026b. URL https://arxiv.org/abs/2606.24626. ARELEASED DATA SCHEMA Each released record separates model-visible fields from reference annotations that are withheld during prediction. The schema below defines these fields and clarifies their roles in scoring. FieldMeaning questionIDSource-prefixed trajectory identifier historyOrdered trajectory records containing step, name, role, and content; includes the task instruction mistake agentReference responsible role, scored independently of the root-cause step mistake stepReference 0-based root-cause step, scored independently of the responsible role mistakereasonTrajectory-grounded annotation rationale, withheld during prediction and not scored Table 5: Released LongRCA Bench data schema. Evaluated methods receive questionID and history, which includes the task instruction. The three reference annotation fields are withheld during prediction. BSOURCE EXECUTION PROVENANCE This section records implementation-level workflow names and generator counts for each source. These names complement the broader agent-organization descriptions in the main benchmark sum- mary. CDIFFICULTY-STRATIFIED NUMERIC RESULTS This section reports the numeric values underlying the difficulty analysis in Figure 6. Both tables use root-cause exact accuracy, and each caption reports the number of trajectories in every bin. As in the main analysis, source domains and workflows differ across bins. The reported values therefore support descriptive comparisons rather than causal conclusions. DROOT-CAUSE DECISION RULES The final-attribution stage applies the operational definition in Section 3 to the recorded trajectory. The rules below specify how it resolves common boundary cases and selects a single root-cause step. • Error already present in a handoff instruction. If a later step implements an erroneous decision already stated in an earlier handoff instruction, the handoff step is selected as the root-cause step. 14 SourceN WorkflowM2.5 K2.5 Q3.5+ SWE-bench Pro128 Custom DiagnostAgent–ActionAgent–JudgeAgent team 128– Terminal Bench 242 Custom DiagnostAgent–ActionAgent–JudgeAgent team 42– TravelPlanner685 AutoGen MagenticOneGroupChat with specialist roles 265260160 VitaBench108 AutoGen Planner–Critic–ActionHead round-robin team 691623 WebArena Verified177 AutoGen Planner–WebSurfer–Critic controller177– Total1,140681276183 Table 6: Execution provenance and generator counts for LongRCA Bench. The SWE-bench Pro and Terminal Bench 2 archives record their custom role workflow and generator, but no separate third-party scaffold package. Generator columns correspond to MiniMax-M2.5 (M2.5), Kimi-K2.5 (K2.5), and Qwen3.5-Plus (Q3.5+). A dash indicates that the source contains no trajectory from that generator. Method ≤ 100101–200201–400 > 400 All-at-once18.12.60.84.8 Step-by-step10.43.41.30.0 Binary search6.41.72.50.0 ECHO23.78.85.09.5 FALAT4.61.52.14.8 RCTA30.320.320.231.0 Table 7: Root-cause exact accuracy (%) by trajectory-length bin. The four bins contain 393, 467, 238, and 42 trajectories, respectively. • Error introduced after a handoff. If a later step departs from the handoff instruction or introduces a decisive error not present in it, that later step is selected as the root-cause step. • Successfully repaired error. An earlier error is excluded if it is corrected before the evaluator-confirmed failure. • Error missed during later verification. A later verifier may fail to detect an earlier error. This omission may be retained as unscored explanatory context, but it does not replace the step that introduced the error. • Contradictory task and evaluation requirements. An input step may be selected only if the task instruction and the official evaluator or test requirement are explicitly incompatible. Both texts must be cited from the recorded trajectory. • Earliest supported introduction. Among candidates supported by the original log, RCTA selects the earliest recorded step that introduced the decisive error relevant to the fi- nal evaluator-confirmed failure. The selected error must remain unrepaired through the evaluator-confirmed failure. These rules govern only the root-cause-step output. RCTA predicts the responsible role in a separate output field. The emitter of the selected step therefore does not determine the predicted role. ERCTA PROMPTS AND IMPLEMENTATION DETAILS RCTA comprises three prompt stages: local summarization, trajectory-outline aggregation, and final attribution. A segment is a consecutive block of original trajectory steps. A phase groups one or more adjacent segment summaries that serve the same subgoal. Phases appear only in the trajectory outline and do not repartition the raw steps. A handoff instruction is a logged X (-> Y) message whose content instructs role Y . 15 Method ≤ 1011–5051–100 > 100 All-at-once10.212.34.33.4 Step-by-step5.98.34.32.5 Binary search0.45.53.83.4 ECHO11.721.29.88.6 FALAT0.83.73.03.4 RCTA21.527.120.925.6 Table 8: Root-cause exact accuracy (%) by root-to-end-distance bin. The four bins contain 256, 325, 235, and 324 trajectories, respectively. For segmentation, a verifier signal is a rule-extracted PASS or FAIL marker from explicit language in a verifier or terminal record. The signal serves only as an optional boundary cue, not as a semantic judgment, benchmark label, or the source evaluator’s final outcome. RCTA applies the local-summary prompt once per segment, followed by one trajectory-outline call and one final-attribution call. A trajectory with m segments therefore requires m + 2 initial LLM calls before any validation-triggered retry. All reported predictions use the frozen system prompts included with the run artifacts. These artifacts contain separate templates for local summarization, phase aggregation, and final attribution, together with the source files used to construct user mes- sages. Each run snapshot stores byte-identical copies of all three templates and their SHA-256 hashes. It also records the schema and prompt versions, model identifier, inference settings, and retry policy. All 20 snapshots used for the final 1,140 predictions contain identical prompt hashes. The frozen prompts contain task rules and output-schema guidance, but no LongRCA Bench in- stances or input–output demonstrations. User messages are constructed from the released identifier and trajectory history. The reference role, root step, and rationale fields are not provided during prediction. All three stages use DeepSeek-V4-Flash, but their reasoning configurations differ. Local summa- rization disables model thinking, whereas outline aggregation and final attribution enable it with high reasoning effort. The default output limits are 4,000, 6,000, and 8,000 tokens for the three stages, respectively. A length-truncated final response is retried with a doubled limit. Eleven tra- jectories requiring longer structured responses use a recorded 16,000-token final-attribution limit with a 32,000-token length retry. Table 9 separates prompt requirements from the programmatic membership checks for each stage. StageInputRequired outputValidation LocalSegment, task, and overlap Purpose and candidate steps with evidence IDs All IDs occur in the segment OutlineSegment summaries and verifier-signal summary Contiguous subgoal phases with step IDsAll IDs occur in the trajectory Attribution Outline, summaries, candidate text, and handoffs Independent role and root fields, evidence IDs, and any cited handoff Role and IDs occur in the trajectory; cited text is grounded Table 9: Prompt requirements and programmatic membership checks for the three RCTA prompt templates. The local-summary template is applied once per segment; the final role and root fields are validated independently. If the phase-construction response cannot be parsed, the implementation assigns one phase to each local summary. The validator checks referenced step identifiers but does not enforce the requested phase count or ordering. Consequently, 308 of the 1,140 reported outputs contain phase counts outside the requested 3–8 range. Their serialized form is consistent with the fallback behavior. However, the final artifacts do not retain a fallback flag, so this total should not be interpreted as an exact count of parse failures. The final-attribution validator checks the role and root fields independently. It accepts only a role label recorded in the trajectory and a root step drawn from the same trajectory. An invalid field does 16 not overwrite a valid prediction in the other field. External subjects such as ENVIRONMENT are not benchmark labels and cannot be scored outputs. The marker system evaluation may appear only as an unscored terminal node in the explanatory failure chain; it cannot be the predicted root step. Every supporting step must also belong to the recorded trajectory. Segmentation uses rule-based implementation settings, not learned parameters. When adding the next step would exceed 32,000 characters, the implementation starts a new segment when possible; an individual overlong step may exceed this limit. Each segment contains at most 80 non-overlap steps. A boundary after a finish step, or before a role-changing handoff instruction or completion step, requires at least eight steps and 10,000 characters in the current segment. A boundary before a change in the most recent explicit verifier signal requires 10,000 characters but no eight-step mini- mum. After the 24,000-character target, an ordinary source-role change can form a soft boundary. Except for the first, each segment receives up to five preceding steps from the previous segment as overlap context. The base candidate list is capped at 80 before preceding handoff instructions are added, so the final excerpt set can exceed 80. For an upstream attribution, the checker accepts a normalized substring match or sufficient word overlap between the cited text and the referenced handoff instruction. This check establishes textual provenance but does not independently verify that the cited instruction contains the same causal error. The implementation also contains role-name-triggered recall rules for a configured family of planner–executor logs. These rules can add visible executor errors, planner decisions, recovered- error context, and terminal context before candidate prioritization. None of the 1,140 LongRCA Bench trajectories matched the configured role-name signature, so these rules did not affect the reported results. FPIPELINE PSEUDOCODE The pseudocode below summarizes the inference sequence described in Section E, including phase fallback, candidate retrieval, independent role and root validation, and abstention. Input: failed trajectory T with a normalized step-indexed history steps <- enrich_steps(T.history) # role, action type, recorded verifier # signal, and exit status segments <- segment(steps) # rule-based boundaries, size limits, # and five-step overlap local <- [] for s in segments: local.append(local_summary(s)) # one LLM call per segment phases <- organize_phases(local) # prompt requests 3--8 phases if phase-construction JSON cannot be parsed: phases <- one phase per local summary base <- retrieve_source_steps(local, phases) if configured planner--executor signature matches: base <- add rule-based recall candidates(base, steps) base <- prioritize(base)[:80] candidates <- base for each candidate in base: if candidate is from an executor or verifier: add nearest earlier handoff instruction addressed to that role, if present else: add nearest earlier handoff instruction, if present prediction <- select_attribution(phases, candidates) # compare instruction with action; # predict role and root independently validate(prediction, T) # recorded role label and root/support # IDs occur in the trajectory; # system_evaluation only as an unscored # chain terminal; cited handoff text is 17 # grounded in its logged instruction; # retry once with validation feedback if a field is still invalid: preserve valid fields; remove unsupported fields flag for review; abstain if essential fields are missing 18

Instant research discovery