Paper deep dive
TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces
Shu-Xun Yang, Cunxiang Wang, Haoke Zhang, Wenbo Yu, Lindong Wu, Jiayi Gui, Dayong Yang, Yukuo Cen, Zhuoer Feng, Bosi Wen, Yidong Wang, Lucen Zhong, Jiamin Ren, Linfeng Zhang, Jie Tang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 4:59:19 AM
Summary
The paper introduces TraceSIR, a multi-agent framework designed to analyze and report on agentic execution traces. It addresses the challenges of long, complex traces by using three specialized agents: StructureAgent for trace compression via TraceFormat, InsightAgent for fine-grained diagnosis (issue localization, root cause analysis), and ReportAgent for generating comprehensive analysis reports. The framework is evaluated on TraceBench, a new benchmark covering deep research, function calling, and coding scenarios, using the ReportEval protocol. Results show TraceSIR significantly outperforms baselines like ClaudeCode in producing coherent, actionable reports.
Entities (12)
Relation Signals (10)
TraceSIR → containsagent → InsightAgent
confidence 95% · TraceSIR coordinates three specialized agents: ... (2) InsightAgent...
TraceSIR → containsagent → ReportAgent
confidence 95% · TraceSIR coordinates three specialized agents: ... (3) ReportAgent...
TraceSIR → containsagent → StructureAgent
confidence 95% · TraceSIR coordinates three specialized agents: (1) StructureAgent...
TraceSIR → evaluatedon → TraceBench
confidence 94% · To evaluate TraceSIR, we construct TraceBench
TraceSIR → evaluatedwith → ReportEval
confidence 93% · introduce ReportEval, an evaluation protocol for assessing the quality and usability of analysis reports
StructureAgent → usesformat → TraceFormat
confidence 92% · StructureAgent, which introduces a novel abstraction format, TraceFormat
TraceBench → includesscenario → BrowseComp
confidence 90% · TraceBench, covering three real-world agentic scenarios... BrowseComp
TraceBench → includesscenario → Tau2Bench
confidence 90% · TraceBench... Tau2Bench
TraceBench → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic systems augment large language models with external tools and iterative decision making, enabling complex tasks such as deep research, function calling, and coding. However, their long and intricate execution traces make failure diagnosis and root cause analysis extremely challenging. Manual inspection does not scale, while directly applying LLMs to raw traces is hindered by input length limits and unreliable reasoning. Focusing solely on final task outcomes further discards critical behavioral information required for accurate issue localization. To address these issues, we propose TraceSIR, a multi-agent framework for structured analysis and reporting of agentic execution traces. TraceSIR coordinates three specialized agents: (1) StructureAgent, which introduces a novel abstraction format, TraceFormat, to compress execution traces while preserving essential behavioral information; (2) InsightAgent, which performs fine-grained diagnosis including issue localization, root cause analysis, and optimization suggestions; (3) ReportAgent, which aggregates insights across task instances and generates comprehensive analysis reports. To evaluate TraceSIR, we construct TraceBench, covering three real-world agentic scenarios, and introduce ReportEval, an evaluation protocol for assessing the quality and usability of analysis reports aligned with industry needs. Experiments show that TraceSIR consistently produces coherent, informative, and actionable reports, significantly outperforming existing approaches across all evaluation dimensions. Our project and video are publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.00623v1
- Canonical: https://arxiv.org/abs/2603.00623v1
Trouble viewing inline? Open PDF directly →
Full Text
45,682 characters extracted from source content.
Expand or collapse full text
TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces Shu-Xun Yang 1,2 * Cunxiang Wang 2,3∗† Haoke Zhang 2 Wenbo Yu 2 Lindong Wu 2 Jiayi Gui 2 Dayong Yang 2 Yukuo Cen 2 Zhuoer Feng 2,3 Bosi Wen 2,3 Yidong Wang 2 Lucen Zhong 2 Jiamin Ren 2 Linfeng Zhang 4 Jie Tang 3 1 Beijing Institute of Technology, Beijing, China 2 Zhipu AI, Beijing, China 3 Tsinghua University, Beijing, China 4 Shanghai Jiao Tong University, Shanghai, China sheryl.xun@bit.edu.cn; wangcunxiang303@gmail.com Abstract Agentic systems augment large language mod- els with external tools and iterative decision making, enabling complex tasks such as deep research, function calling, and coding. How- ever, their long and intricate execution traces make failure diagnosis and root cause analy- sis extremely challenging. Manual inspection does not scale, while directly applying LLMs to raw traces is hindered by input length lim- its and unreliable reasoning. Focusing solely on final task outcomes further discards crit- ical behavioral information required for ac- curate issue localization. To address these issues, we propose TraceSIR, a multi-agent framework for structured analysis and report- ing of agentic execution traces. TraceSIR co- ordinates three specialized agents: (1) Struc- tureAgent, which introduces a novel abstraction format,TraceFormat, to compress execution traces while preserving essential behavioral in- formation; (2) InsightAgent, which performs fine-grained diagnosis including issue localiza- tion, root cause analysis, and optimization sug- gestions; (3) ReportAgent, which aggregates insights across task instances and generates comprehensive analysis reports. To evaluate TraceSIR, we construct TraceBench, covering three real-world agentic scenarios, and intro- duceReportEval, an evaluation protocol for assessing the quality and usability of analysis reports aligned with industry needs. Exper- iments show that TraceSIR consistently pro- duces coherent, informative, and actionable reports, significantly outperforming existing approaches across all evaluation dimensions. Our project and video are publicly available at https://github.com/SHU-XUN/TraceSIR. 1 Introduction Agentic systems, in which large language mod- els (LLMs) are augmented with external tools and * Equal contribution. Work done at Zhipu AI. † The corresponding author TASK: I N S T A N C E 1 I N S T A N C E 2 I N S T A N C E . . . I N S T A N C E * This series, released prior to 2015, is known by four other titles. The story follows a protagonist who, after losing ...... Based on these details, please identify the English name of this series. TRACES: ANSWER: ResetROUNDS: 33TOKENS: 89,616CHARACTERS: 319,254 We have a puzzle: identify a series (TV series? drama?) released prior to 2015 ...... search"query":"`lost his beloved to a crime` drama series protagonist dedicates ......" 【0† Tribute to late actor in Murder in a Small Town episode†https://w.facebook.c ...... finish"answer": "Tell Me What You Saw" Explanation: - The series ...... Exact Answer: Tell Me What You Saw Confidence: 95% ........................................................................................................................................................ Existing Methods TraceSIR > Outcome-Only Evaluation SCORE: ERROR: ADVANTAGE: DISADVANTAGE: INSIGHT: OPTIMIZATION: 5 LABEL: Logical Gap Main: Year mismatch; Other: ... Systematic analysis ... Inefficient searching ... Cognitive architecture flaws ... Fine-tuned samples ... I N S T A N C E 1 I N S T A N C E . . . I N S T A N C E * Single Instance Multiple Instances > Execution Trace Evaluation with Final Answer without Behavioral Signal Issue Localization Root Cause Analysis Length Limit Unreliable Reasoning Long Execution Traces Meaningless Content > Clustering-Based Aggregation > Direct Summarization Instance1Instance2Instance* Instance1Instance2Instance* ...... Summary ...... Clustering Surface Pattern Coarse Report Lack of Pattern Generic Report # Overall Overview # Error Analysis # Score Distribution # Weakness Patterns # Root Cause Analysis # Optimization Suggestions # Appendix - TraceBench Single Instance Multiple Instances Figure 1: Comparison of existing methods and Trace- SIR for agentic execution trace analysis and reporting. iterative decision making, have emerged as a pow- erful paradigm for real-world applications such as ClaudeCode and OpenClaw. As a result, improving the effectiveness of agentic systems has become an important research topic. This requires the identi- fication of the limitations of existing agentic mod- els and the root causes of the issues they exhibit. However, the dominant practice of manually in- specting agent outputs does not scale to the long and complex execution traces of agentic systems, where a single task may involve thousands of tool invocations and sub-agent interactions spanning large numbers of tokens, making comprehensive understanding and root cause analysis prohibitively difficult for humans (Deshpande et al., 2025). This difficulty is further compounded by the fact that valuable issue localization and root cause analysis often require coordinated comparison across mul- tiple task instances. Such cross-instance analysis substantially increases the analytical burden and 1 arXiv:2603.00623v1 [cs.AI] 28 Feb 2026 underscores the urgent need for automated analysis and reporting frameworks. However, automated analysis of agentic systems faces a fundamental bottleneck, as long execution traces can easily exceed the input length limits of LLMs and substantially interfere with their ability to reliably analyze agent behavior, often leading to meaningless or hallucinatory content (Chen et al., 2026; Jia et al., 2026), as shown in Figure 1. One possible alternative is to focus solely on final task outcomes, but relying only on outcomes discards large amounts of valuable information embedded in the execution traces, which directly hinders ac- curate issue localization and prevents root cause analysis that depends on behavioral context. Ac- cordingly, a central challenge is how to abstract agentic execution traces in a way that preserves core information necessary for accurate issue local- ization and root cause analysis. In addition, present- ing researchers with comprehensive and coherent analysis reports poses another important challenge. To date, little prior work has focused on generat- ing and evaluating analysis reports that align with practical industry needs. To address these challenges, we propose Trace- SIR, a multi-agent framework for structured anal- ysis and reporting of agentic execution traces. Specifically, TraceSIR coordinates three special- ized agents that operate collaboratively, namely StructureAgent, InsightAgent, and ReportAgent. StructureAgent introduces a novel abstraction repre- sentation, referred to asTraceFormat, which struc- turally abstracts execution traces to substantially reduce redundancy while preserving core informa- tion from the original traces. Built on the structured traces, InsightAgent performs fine grained analy- sis, including overall assessment, issue localization, weakness identification, root cause analysis, and optimization suggestions. Finally, ReportAgent summarizes critical errors observed in InsightA- gent and determines whether report generation is warranted based on the number of task instances. When triggered, it conducts targeted statistical anal- ysis and produces comprehensive analysis reports that provide researchers and engineers with system- atic and detailed insights to support decisions. To evaluate the effectiveness of TraceSIR, we collect task instances from three representative real- world agentic benchmarks, including BrowseComp (Deep Research) (Wei et al., 2025), Tau2Bench (Function Calling) (Barres et al., 2025), and SWE- bench (Agentic Coding) (Jimenez et al., 2024), and construct a unified benchmark named TraceBench for systematic experimental analysis. Specifically, we select 50 failure cases from each benchmark us- ing GLM-4.6 as the target model, and retain execu- tion traces in the standard OpenAI message format, which serve as inputs to our TraceSIR for struc- tured analysis and reporting of agentic execution traces. To reflect the practical needs of researchers and engineers in real industry settings, we further introduce a report evaluation protocol, referred to asReportEval. ReportEval defines a set of gold standard principles for assessing analysis reports and adopts LLM-as-a-judge to assist human annota- tion, enabling end-to-end evaluation of the quality and usability of reports generated by TraceSIR. Experiments under theReportEvalprotocol show that the analysis reports generated by Trace- SIR on TraceBench consistently outperform those produced by the strong baseline ClaudeCode. On average, TraceSIR improves overall report quality by 9.7% under human evaluation by expert LLM Agent researchers and 7.5% under LLM-as-a-judge evaluation, with the largest relative gain reaching 26.0%. These results indicate that TraceSIR pro- duces analysis reports that better align with prac- tical industry needs and deliver more reliable and informative diagnostic insights. More importantly, TraceSIR yields clear and robust advantages in is- sue localization, error interpretation, root cause analysis, and optimization suggestions, demonstrat- ing its effectiveness in trace structuring and analy- sis across different underlying LLMs. 2 Related Work Most existing approaches (Wei et al., 2025; Bar- res et al., 2025) for evaluating agentic systems re- main outcome-oriented, which limits their ability to support issue localization and root cause analy- sis that depend on fine-grained behavioral context. Although some recent work attempts to incorpo- rate partial intermediate signals, such as selected states (Avraham et al., 2026; Chuang et al., 2026) or summarized steps (Chen et al., 2026; Deshpande et al., 2025), these abstractions inevitably discard substantial contextual information embedded in long execution traces and fail to support in-depth, insight-driven root cause analysis, making system- atic diagnosis of agent behavior difficult. In practice, understanding agentic systems often requires analyzing multiple task instances and pro- ducing consolidated analysis reports, yet such set- 2 ReportEvalTraceFormat Input File Format Single Trace Instance in JSON Format Multiple Trace Instances in ZIP Format Single Trace Instance Format Optional Additional Requirements for Report Generation Language, Template, Structure, Format, Scope ...... [Required] oid messages (OpenAI) [Optional] task gold_score gold_judge other StructureAgent InsightAgent ReportAgent ThoughtActionObservation Round1 We have a puzzle: ...... search"quer y":"......" 【0† Tribute to late actor ...... Round... .................. Round* Explanation: - The ...... finish"answ er":"......" / TraceFormat messages (OpenAI) id oid thought action observation task length gold_score gold_judge other TraceBench-1, 2, ... , * len(thought)=len (action)=len(obs ervation)=* Abstract length threshold per field per step > Overall Assessment 0~100 > Error Detection Main Error: Core fatal error in the trace Other Errors: Other minor errors in the trace > Weakness Identification Disadvantages: Shortcomings revealed in the trace Advantages: Commendable aspects in the trace > Root Cause Analysis Insight > Optimization Suggestions Suggestions: Concrete and actionable suggestions Fine-tuned Samples: Targeted SFT samples > Error Label> Ready Time > Error Frequency Estimation > Score Distribution Modeling > Report Generation > Report Refinement Markdown Requirement Appendix TraceBench ReportEval gold_score gold_judge Overall StructureOverall Impact Error Analysis Root Cause Analysis Optimization Analysis Figure 2: The overall architecture of TraceSIR. tings have received limited attention in prior work. Existing approaches typically evaluate cases in iso- lation and aggregate results only at a superficial level (Chen et al., 2026; Deshpande et al., 2025; Avraham et al., 2026), which obscures recurring failure patterns across execution traces. Moreover, techniques based on naive summarization (Aranya and Desai, 2026; Aubreville et al., 2025; Gan et al., 2025) or clustering-style aggregation (Rivera et al., 2026; Salmè et al., 2026; Zhao et al., 2026) strug- gle to preserve trace-level evidence, often resulting in overly coarse or weakly grounded reports. In contrast, TraceSIR is designed specifically for structured analysis and reporting of agent execution traces. By abstracting long traces into structured representations and coordinating specialized agents for diagnosis and aggregation, TraceSIR enables scalable issue localization, root cause analysis, and actionable reporting across multiple cases. 3 Method We propose TraceSIR, a multi-agent framework for structured analysis and reporting of agentic ex- ecution traces. As illustrated in Figure 2, Trace- SIR coordinates three specialized components, i.e., StructureAgent, InsightAgent, and ReportAgent. Each component operates autonomously with tool invocation over structured traces, while the overall framework is designed to support both fine-grained per-case analysis and scalable cross-case reporting. 3.1 TraceFormat The input to TraceSIR can be either a single execu- tion trace in JSON format or multiple trace cases in ZIP format. Each trace case is represented as an object with a requiredoidfield for trace identifi- cation, amessagesfield containing the execution trace in the standard OpenAI format, and several optional auxiliary fields. Formally, for each trace case, the core execution trace stored in themessagesfield is represented as a sequence of messagesM =m 1 ,m 2 ,...,m K . Each messagem k consists of a role (e.g.,user, assistant,tool) and the corresponding content. This representation is widely adopted by modern agentic systems and naturally captures iterative rea- soning, tool invocation, and environment feedback. In addition to themessagesfield and an arbitrary trace identifieroid, a trace case may optionally include the following fields: • task: a textual description of the task; • gold_score: an automated evaluation score provided by external evaluators; • gold_judge : textual feedback or error mes- sages from automated assessment; • other: auxiliary or miscellaneous metadata. TraceSIR also supports an optional additional requirementspecification that constrains or guides the report generation process, such as pref- erences over language. In the absence of explicit specifications, reports are generated in Chinese by default, considering practical deployment settings. All optional fields are treated as auxiliary signals and are not required for TraceSIR to function. To enable scalable and reliable analysis, Trace- SIR introducesTraceFormat, a structured abstrac- tion of OpenAI messages that preserves essential behavioral information while providing a compact and analyzable representation. Given an execution traceM, we define a deterministic parsing function Φ :M→T =(t i ,a i ,o i ) N i=1 (1) which transforms the original message sequence 3 into a structured traceTconsisting ofNinterac- tion rounds. In each round,t i means the agent’s Thought, capturing its intermediate reasoning ex- pressed in assistant messages;a i denotes the Ac- tion selected by the agent, such as a tool invocation or explicit operation; ando i represents the resulting Observation returned by the environment or tool. TraceFormatstrictly preserves the temporal or- der and causal alignment among reasoning, action, and observation, and can be rendered as a three- column table to support both human inspection and automated analysis. Each trace case is assigned a standardized identifier with aTraceBenchprefix through a newly introducedidfield, ensuring con- sistent indexing and tool-based cross-case compar- ison throughout the analysis and reporting process. 3.2 StructureAgent WhileTraceFormatstandardizes execution traces, long-horizon agent behaviors often contain verbose reasoning, long code snippets, or extensive tool out- puts that exceed practical context limits for down- stream analysis. StructureAgent addresses this bot- tleneck by performing trace-level abstraction. Formally, given a structured traceT, Struc- tureAgent applies a length-aware abstraction oper- ator parameterized by a thresholdθto produce a compressed traceT ′ = A θ (T ), whereA θ selec- tively abstracts overlong elements while preserving essential behavioral information through automatic tool invocation. The abstraction is performed inde- pendently at the level of individual steps and across different fields, i.e., Thought, Action, and Obser- vation, ensuring that only redundant or excessively verbose content is compressed, without obscuring critical signals required for subsequent analysis. When necessary, StructureAgent also generates a compact abstraction of thetaskfield. The re- sulting structured traceT ′ substantially reduces redundancy while remaining faithful to the original execution semantics, thereby enabling reliable and scalable downstream diagnosis. 3.3 InsightAgent Built on the structured traces produced by Struc- tureAgent, InsightAgent performs fine-grained, instance-level diagnostic analysis. Rather than re- lying solely on final task outcomes, it reasons over the entire structured execution trace to uncover be- havioral issues and performance limitations. Formally, given a structured traceT ′ and the associatedtaskfieldq, InsightAgent produces a set of structured diagnostic outputs, denoted as D =s,E,W,R,O, by autonomously invoking analysis tools over the structured trace. Here,s represents an overall task completion assessment ranging from 0 to 100.Ecaptures detected er- rors in the execution trace and distinguishes be- tween the primary, task-critical error and other sec- ondary errors.Wdenotes identified weaknesses revealed by the trace, while also optionally noting notable strengths when relevant.Rcorresponds to root cause analysis, providing in-depth and insight- driven explanations of why the observed failures or limitations occur, and serves as a central diagnostic signal in our framework. Finally,Oconsists of optimization suggestions, including concrete tex- tual recommendations and fine-tuned samples to address the identified issues. When available,gold_scoreandgold_judge are incorporated as auxiliary reference signals to support task completion assessment and error iden- tification and interpretation. In their absence, In- sightAgent derives all diagnostic outputs solely from trace-level behavioral evidence. All diagnos- tic outputs for each trace case are returned to the user in a structured format, supporting fine-grained, per-case analysis and enabling reliable aggregation and comparison across multiple cases. 3.4 ReportAgent While InsightAgent focuses on individual task in- stances, practical diagnosis of agentic systems of- ten requires coordinated analysis across multiple cases. ReportAgent operates at this higher level. Formally, given a collection of diagnostic out- putsD 1 ,...,D M produced by InsightAgent for Mtrace cases, ReportAgent autonomously invokes analysis tools to determine whether report gener- ation is warranted based on the number of avail- able cases and predefined triggering criteria. When activated, it performs targeted statistical analysis over the aggregated diagnostics, including error fre- quency estimation and score distribution modeling, to uncover recurring patterns and systematic issues. For error analysis, ReportAgent summarizes the detected errorsE i of each case into a canonical er- ror labelℓ i , and estimates the frequency of each er- ror type acrossMcases asP (ℓ) = 1 M P M i=1 I(ℓ i = ℓ) , whereℓ i denotes the error label derived from E i for casei. For performance analysis, ReportA- gent models the distribution of task completion scores over a set of predefined, disjoint score in- tervalsB = b 1 ,...,b L . The score distribution 4 ScenarioMethodBackboneOverall ScoreOSEARCAOAOIRanking Deep Research ClaudeCodeGLM-555.05.54.55.56.06.04 ClaudeCodeClaude-4.666.06.57.55.57.06.52 TraceSIRGLM-565.07.56.55.56.07.03 TraceSIRClaude-4.681.08.58.58.07.08.51 Function Calling ClaudeCodeGLM-5 40.05.03.54.53.04.04 ClaudeCodeClaude-4.6 74.07.57.58.56.57.02 TraceSIRGLM-553.06.06.05.05.04.53 TraceSIRClaude-4.677.07.57.58.57.57.51 Agentic Coding ClaudeCodeGLM-557.06.55.55.56.05.04 ClaudeCodeClaude-4.677.07.58.07.07.58.52 TraceSIRGLM-562.05.56.56.56.56.03 TraceSIRClaude-4.689.09.09.09.08.59.01 Table 1: Comparison of analysis report quality under theReportEvalprotocol usinghuman evaluation. TraceSIR is our system while ClaudeCode is the baseline. is estimated asP (b) = 1 M P M i=1 I(ˆs i ∈ b), where ˆs i denotes the score used for aggregation, taking gold_scorewhen available and otherwise falling back to the score s i produced by InsightAgent. ReportAgent integrates quantitative statistics with qualitative insights to generate comprehen- sive analysis reports in Markdown format by au- tonomously invoking reporting and analysis tools. When user-specifiedrequirementfields are pro- vided, they are incorporated during report genera- tion; otherwise, reports are produced according to a default analysis schema. After report generation, ReportAgent further refines the report by automat- ically identifying referenced trace cases through standardizedidfields with theTraceBenchpre- fix using a matching tool, and appending the cor- responding structured trace data to the appendix. This process ensures transparency and traceability of the reported analyses. By aggregating trace-level evidence across cases, TraceSIR produces coherent, evidence-grounded reports that support practical research and engineering decision-making. 4 Experiments 4.1 TraceBench To evaluate the effectiveness of TraceSIR, we con- struct TraceBench, a unified benchmark of agentic execution traces collected from three representa- tive real-world agentic benchmarks: BrowseComp (Deep Research) (Wei et al., 2025), Tau2Bench (Function Calling) (Barres et al., 2025), and SWE- bench (Agentic Coding) (Jimenez et al., 2024). TraceBench includes 150 failed task instances of GLM-4.6 1 , distributed across three scenarios. Con- struction details are provided in Appendix A. 1 https://huggingface.co/zai-org/GLM-4.6 4.2 ReportEval To evaluate the quality and practical usefulness of analysis reports generated by TraceSIR, we proposeReportEval, a report-centric evaluation protocol for agentic execution trace analysis. ReportEvalassesses each report along five dimen- sions with equal weight, producing both dimension- level scores and an overall quality score. The five evaluation dimensions are defined as fol- lows. Overall Structure (OS) evaluates whether the report is well organized, coherent, and clearly grounded in the analyzed execution traces. Er- ror Analysis (EA) measures the correctness and trace support of the identified agent errors. Root Cause Analysis (RCA) assesses whether the report provides insightful and well-justified explanations of the underlying causes of the observed errors. Optimization Analysis (OA) evaluates the rele- vance, feasibility, and actionability of the proposed optimization suggestions. Overall Impact (OI) provides a holistic assessment of the report’s use- fulness from a practitioner’s perspective, reflecting its value for understanding agent behavior and sup- porting decision making. Each dimension is scored on a scale ranging from 0 to 10. The overall report score is computed as the equally weighted sum of the five evaluation dimension scores and normal- ized to a range from 0 to 100. 4.3 Settings TraceSIR is instantiated with two backbone LLMs, GLM-5 (GLM-5-Team et al., 2026) and Claude- 4.6 2 , and we adopt ClaudeCode 3 as the base- line model. For report evaluation, we adopt a hy- 2 https://w.anthropic.com/news/claude-opus-4-6 3 https://github.com/anthropics/claude-code 5 ScenarioMethodBackboneOverall ScoreOSEARCAOAOIRanking Deep Research ClaudeCodeGLM-582.78.78.07.39.08.34 ClaudeCodeClaude-4.690.09.38.79.09.09.02 TraceSIRGLM-588.09.08.09.09.09.03 TraceSIRClaude-4.691.39.39.09.39.09.01 Function Calling ClaudeCodeGLM-5 80.78.37.37.79.08.04 ClaudeCodeClaude-4.6 88.08.78.39.09.09.03 TraceSIRGLM-589.39.08.79.09.09.02 TraceSIRClaude-4.691.39.09.09.09.79.01 Agentic Coding ClaudeCodeGLM-558.76.04.05.78.05.74 ClaudeCodeClaude-4.690.09.09.09.09.09.02 TraceSIRGLM-584.78.37.78.79.08.73 TraceSIRClaude-4.690.79.39.09.09.09.01 Table 2: Comparison of analysis report quality under theReportEvalprotocol usingLLM-as-a-judge. TraceSIR is our system while ClaudeCode is the baseline. brid setting that combines human assessment (6 expert LLM agent engineers/researchers) and au- tomated LLM-based judging. Human evaluators were blinded to the underlying methods and model identities. Further experimental settings and evalu- ation details are provided in Appendix B. 4.4 Results From Table 1 and Table 2, it can be observed that TraceSIR consistently and substantially out- performs ClaudeCode across all three agentic sce- narios, demonstrating clear advantages in overall report quality and fine-grained diagnostic capabil- ity. From the perspective of human evaluation, TraceSIR yields consistent and sizable improve- ments over ClaudeCode with both backbone mod- els. When using the weaker GLM-5 backbone, TraceSIR improves the overall score by 10.0%, 13.0%, and 5.0% in Deep Research, Function Call- ing, and Agentic Coding, respectively. With the stronger Claude-4.6 backbone, TraceSIR further improves performance by 15.0%, 3.0%, and 12.0% across these scenarios. Averaged across scenarios and backbones, TraceSIR achieves a 9.7% rela- tive improvement under human evaluation, indicat- ing a clear expert preference for its reports. Aver- aged across the five 10-point evaluation dimensions, TraceSIR further achieves mean improvements of 0.9, 1.3, 1.2, 0.8, and 0.9 points, respectively, with particularly pronounced gains in error analysis and root cause analysis, demonstrating its strengthened diagnostic capability and its effectiveness in sup- porting systematic analysis and report generation. Results from LLM-as-a-judge evaluation exhibit a highly consistent trend. With the GLM-5 back- bone, TraceSIR improves the overall score by 5.3%, 8.6%, and 26.0%, respectively. When paired with Claude-4.6, the corresponding improvements are 1.3%, 3.3%, and 0.7%. On average, TraceSIR yields a 7.5% relative improvement over Claude- Code under LLM-based evaluation. Notably, the largest gains occur in more challenging settings, particularly Agentic Coding with weaker backbone models, highlighting TraceSIR’s robustness in pro- ducing diagnostically meaningful reports under constrained model capacity. Although the absolute scores differ between the two evaluation settings, the overall conclusions remain consistent. Human experts tend to provide more conservative and strin- gent assessments, whereas the LLM judge assigns relatively higher scores, reflecting a more permis- sive evaluation behavior. Despite this difference in score calibration, both evaluation methods pro- duce highly aligned rankings and converge on the same qualitative conclusion that TraceSIR gener- ates more structured, comprehensive, and action- able reports than ClaudeCode across all scenarios, with consistent advantages observed across all di- mensions defined in the ReportEval protocol. 5 Conclusion We presented TraceSIR, a framework for struc- tured analysis and reporting of agentic execution traces. By introducingTraceFormatto abstract long execution traces while preserving critical be- havioral information, TraceSIR enables scalable is- sue localization and insight-driven root cause anal- ysis. The coordinated design of StructureAgent, InsightAgent, and ReportAgent further supports both instance-level diagnosis and cross-case aggre- gation, producing coherent and actionable analysis reports. Experiments on TraceBench show that TraceSIR generates high-quality reports aligned with practical research and engineering needs. 6 Limitations Despite its effectiveness, TraceSIR has several lim- itations. First, TraceSIR relies on LLM-based agents for trace abstraction, diagnostic reasoning, and report generation, and the quality of the result- ing analyses may be influenced by the capabilities of the underlying language models, particularly in challenging domains such as complex coding tasks or highly specialized technical settings. Sec- ond, while TraceSIR is designed to support scal- able cross-case analysis, its current report genera- tion strategy assumes a moderate number of task instances. Applying TraceSIR to very large collec- tions of traces may require additional mechanisms for hierarchical aggregation or incremental report- ing to maintain report clarity and efficiency. Fi- nally, TraceSIR may incur relatively long response times and high token consumption, and its reports may vary due to the inherent randomness of LLMs. These factors limit its efficiency and reproducibil- ity in practical use, and mitigating latency, cost, and variability remains an important direction for future work. Ethics and Broader Impact This work is conducted in accordance with the ACM Code of Ethics. TraceSIR is a system for structured analysis and reporting of agentic ex- ecution traces, designed to help researchers and engineers understand agent behaviors, diagnose failures, and identify root causes in complex, long- horizon agentic systems. By transforming raw exe- cution traces into structured, analyzable representa- tions and aggregating evidence across multiple task instances, TraceSIR aims to improve transparency, reliability, and usability of agentic systems. Data and Privacy. TraceSIR operates on execu- tion traces generated by agentic systems and does not require access to personal data by design. The datasets used in our experiments are publicly avail- able benchmarks, and all traces are collected from synthetic or benchmark-defined tasks. Neverthe- less, when TraceSIR is applied in real-world de- ployments, execution traces may contain sensitive information such as proprietary prompts, tool out- puts, or user-provided content. In such settings, appropriate data handling practices, including ac- cess control, anonymization, and secure storage, should be followed. Potential Benefits. TraceSIR is intended to sup- port researchers and engineers who develop, de- ploy, and maintain agentic systems, by enabling systematic issue localization, root cause analysis, and cross-case reporting that are difficult to achieve through manual inspection. By providing coherent and evidence-grounded analysis reports with ac- tionable optimization suggestions, TraceSIR may improve debugging efficiency, support informed en- gineering decisions, and contribute to safer, more reliable, and more interpretable agentic systems. Potential Risks and Misuse.TraceSIR relies on LLM-based agents for trace abstraction, diagnos- tic reasoning, and report generation. As a result, its outputs may reflect limitations, biases, or inac- curacies of the underlying models. Over-reliance on automatically generated analyses without hu- man verification could lead to incorrect conclusions or suboptimal system modifications. TraceSIR is therefore designed as a decision-support tool rather than an autonomous authority, and its outputs are intended to be interpreted by practitioners. Broader Impact. More broadly, this work aims to promote principled and transparent analysis prac- tices for agentic systems. While TraceSIR can be used to improve system quality and robustness across a range of research and engineering settings, it may also be applied to analyze large-scale agent deployments. Responsible use of such systems requires careful consideration of transparency, ac- countability, and data governance, particularly in real-world applications. We encourage future work to further explore safeguards, evaluation practices, and governance mechanisms for responsible use of automated trace analysis and reporting systems. References OFM Riaz Rahman Aranya and Kevin Desai. 2026. Trace: Temporal radiology with anatomical change explanation for grounded x-ray report generation. Preprint, arXiv:2602.02963. Marc Aubreville, Taryn A. Donovan, and Christof A. Bertram. 2025.Exploring general-purpose au- tonomous multimodal agents for pathology report generation. Preprint, arXiv:2601.11540. Elad Ben Avraham, Changhao Li, Ron Dorfman, Roy Ganz, Oren Nuriel, Amir Dudai, Aviad Aberdam, Noah Flynn, Elman Mansimov, Adi Kalyanpur, and Ron Litman. 2026. Dream: Deep research evaluation with agentic metrics. Preprint, arXiv:2602.18940. 7 Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025.τ 2 -bench: Evaluat- ing conversational agents in a dual-control environ- ment. Preprint, arXiv:2506.07982. Yanyu Chen, Jiyue Jiang, Jiahong Liu, Yifei Zhang, Xiao Guo, and Irwin King. 2026. Trace: Trajectory- aware comprehensive evaluation for deep research agents. Preprint, arXiv:2602.21230. Yun-Shiuan Chuang, Chaitanya Kulkarni, Alec Chiu, Avinash Thangali, Zijie Pan, Shivani Shekhar, Yirou Ge, Yixi Li, Uma Kona, Linsey Pang, and Prakhar Mehrotra. 2026. Toward scalable verifiable reward: Proxy state-based evaluation for multi-turn tool- calling llm agents. Preprint, arXiv:2602.16246. Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. 2025. Trail: Trace reasoning and agentic issue local- ization. Preprint, arXiv:2505.08638. Shuyu Gan, Renxiang Wang, James Mooney, and Dongyeop Kang. 2025. A2p-vis: an analyzer-to- presenter agentic pipeline for visual insights genera- tion and reporting. Preprint, arXiv:2512.22101. GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Hao- ran Wang, and 168 others. 2026. Glm-5: from vibe coding to agentic engineering. Preprint, arXiv:2602.15763. Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng. 2026. Mas-fire: Fault injection and reliability evaluation for llm-based multi-agent systems. Preprint, arXiv:2602.19843. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language mod- els resolve real-world github issues? In The Twelfth International Conference on Learning Representa- tions. Juampablo E. Heras Rivera, Dickson T. Chen, Tianyi Ren, Daniel K. Low, Asma Ben Abacha, Alberto Santamaria-Pang, and Mehmet Kurt. 2026. Btreport: A framework for brain tumor radiology report gen- eration with clinically relevant features. Preprint, arXiv:2602.16006. Marco Salmè, Federico Siciliano, Fabrizio Silvestri, Paolo Soda, Rosa Sicilia, and Valerio Guarrasi. 2026. Concept-enhanced multimodal rag: Towards inter- pretable and accurate radiology report generation. Preprint, arXiv:2602.15650. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McK- inney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet chal- lenging benchmark for browsing agents. Preprint, arXiv:2504.12516. Ruoqing Zhao, Runze Xia, and Piji Li. 2026. Hal- lucination mitigating for medical report generation. Preprint, arXiv:2601.15745. 8 TraceSIR Agent Demo LLM Configuration Model API KeyBase URL Upload File Optional Additional Requirements for Report Generation Query Status Download Result Rerun Task Optional Additional Requirements for Report Generation Run Rerun Query Download Language, Template, Structure, Format, Scope ...... Language, Template, Structure, Format, Scope ...... Enter job_id Enter job_id Return job_id Enter job_id Choose File *.json / *.zip Task Logs Report Console Send For first-time use, please make sure to enter the job_id first. After the initial report is returned, you can proceed with multi-turn conversational interactions. # StructureAgent ...... # InsightAgent ...... # ReportAgent ...... > job_id conclude_report.md # ...... # ...... > Generate the report in English ...... > More data analysis ...... A B C D E F G H Figure 3: The demonstration of TraceSIR. A TraceBench Construction For the BrowseComp, Tau2Bench, and SWE-bench benchmarks, we run GLM-4.6 4 as the target agent model on the official test split and retain the com- plete execution traces in the standard OpenAI mes- sage format. Task performance is evaluated us- ing the official evaluation protocols provided by each benchmark, yielding a binarygold_score. To focus on diagnostic analysis of agent failures, we randomly sample 50 failure cases from each benchmark, where thegold_scoreequals 0. In to- tal, TraceBench contains 150 failed task instances across the three scenarios. All collected execution traces are used as inputs to TraceSIR for structured analysis and reporting of agentic execution traces. B Experimental Settings We evaluate TraceSIR under a controlled experi- mental setting. The length thresholdθfor trace abstraction is set to 100 words or 1,000 charac- ters. During processing, report generation is auto- matically triggered when the number of processed cases reaches a multiple of 10. For evaluation con- sistency, however, we compare only the final re- ports generated after all 50 cases in each dataset have been processed. For each agentic scenario in TraceBench, all methods operate on the same set of 50 sampled failure cases with identical ex- ecution traces, producing one report per scenario for comparison. Notably, in practical deployments and our demonstration system, in addition to the default report generation setting described above, we enforce the generation of a final analysis report regardless of the number of input cases, in order to improve user experience. As a result, a reference 4 https://docs.bigmodel.cn/cn/guide/models/text/glm-4.6 # 模型智能体 Bad Case 深度分析报告 --- ## 1. 全局概览 ...... ## 2. 常见错误分析 ### 2.1 错误类型分布总览 | 错误大类 | 涵盖的 key_error 标签 | 案例数 | 占比 | |:---------|:---------------------|:------:|:----:| | 理解偏差/概念混淆 | 理解偏差(4)、概念理解偏差(1)、概念混淆错误(1) | 6 | **12%** | | 确认偏误/过早锁定 | 锁定错误(2)、锚定偏误(1)、错误定位(1) | 4 | **8%** | | 搜索策略失效 | 策略混乱(2)、搜索失焦(1)、搜索不精准(1)、检索策略失败(1)、检索策略失误(1) | 6 | **12%** | | 验证不足/逻辑断裂 | 验证不足逻辑混乱(1)、证据不足(1)、验证过度(1)、浅尝辄止(1) | 4 | **8%** | | 时间推理错误 | 时间判断失误(1)、时间验证不足(1)、时间验证失误(1)、时间核实失误(1)、时间逻辑混乱(1) | 5 | **10%** | | 目标丢失/未闭环 | 目标丢失(1)、未提交答案(1)、未填参数(1)、半途而废(1)、偏离主线(1) | 5 | **10%** | | 信息整合失败 | 整合推理不足(1)、搜索整合失效(1)、关键断链(1)、检索逻辑断裂(1) | 4 | **8%** | | 识别/匹配错误 | 识别不准确(1)、身份识别错误(1)、目标识别错误(1)、识别笔名(1)、识别计算错误(1) | 5 | **10%** | | 其他（谜题解码、距离误判等） | 谜题解码失败(1)、距离误判(1)、坐标选取错误(1)、平均当时点(1)等 | 11 | **22%** | ### 2.2 十大错误趋势深度分析 #### **趋势 1：硬约束条件被软化处理（占比 ~30%，15/50 案例）** **模式描述**：当任务包含精确的数值、时间或逻辑约束时，智能体倾向于将这些"必须满足"的硬性条件降级为"尽量满足"的软性偏 好。 **典型案例**： - **TraceBench-25**：题目明确要求"2015 年前发行"，但智能体最终给出 2020 年播出的《Tell Me What You Saw》，并赋予 95% 置 信度，完全忽视时间硬约束 - **TraceBench-12**：题目要求德州分店距 Town East Mall **5.1 英里**，实际验证为 7.6 英里，智能体试图通过改变测量方式"强行 匹配" - **TraceBench-23**：题目要求"1989-1992 年间组建乐队"，但 Loose Change 乐队官网明确标注 1988 年成立，智能体引用第三方平 台"est. 1990"来合理化矛盾 **成因分析**：智能体缺乏约束分类机制，将所有条件视为等权重的"特征匹配项"，用累积评分替代了布尔逻辑的"全部满足"要求。 当多数软条件匹配时，个别硬约束的违反被概率性地"稀释"。 ...... ## 3. 分数分布分析 ...... ## 4. 模型缺点模式 基于 disadvantages 字段的系统性分析，归纳出以下六大典型缺点模式： ### 模式 A：验证闭环缺失（出现在 ~80% 的案例中） ...... ## 5. 根因分析与洞察生成 ### 洞察 1：概率匹配范式与逻辑验证需求的根本性冲突 ...... ## 6. 结论与建议 ### 6.1 核心发现总结 ### 6.2 优化建议 #### 短期可执行措施（Prompt 工程层） #### 中期架构改进 #### 长期训练策略 ### 6.3 趋势预测 ...... ## 附录：TraceBench 处理后的数据 ### TraceBench-1 ...... Figure 4: An excerpt from a Chinese analysis report generated by TraceSIR using Claude-4.6 on the Deep Research scenario of BrowseComp. report is always produced at the end of analysis, even when only a single case is provided. For report evaluation, we adopt a hybrid setting that combines expert human assessment with au- tomated LLM-based judging. Specifically, six do- main experts with extensive experience in agentic systems manually evaluate the generated reports following theReportEvalprotocol. The final hu- man evaluation score is obtained by averaging the ratings across experts, with each scenario evaluated by at least two experts. In addition, we adopt an LLM-as-a-judge approach under theReportEval protocol using GPT-5 5 , running the evaluation three times and averaging the scores to improve stability, thereby enabling scalable and consistent assessment complementary to human evaluation. C TraceSIR Demonstration Figure 3 presents an end-to-end demonstration of TraceSIR, illustrating how the system supports structured analysis and reporting of agentic execu- tion traces in practice. The demo provides a unified interface for configuring language models, submit- ting execution traces, monitoring analysis progress, retrieving results, and interactively refining gener- ated reports. It is designed to reflect realistic usage scenarios of TraceSIR and to support transparent and reproducible analysis workflows. 5 https://developers.openai.com/api/docs/models/gpt-5 9 CategoryCapabilityTraceSIRClaudeCode System Design 1. Multi-agent architecture explicitly designed for analysis and reporting, with separated roles for trace structuring, diagnosis, and reporting ✓△ 2. Fine-grained, per-case diagnostic outputs with structured fields✓△ 3. Native cross-case aggregation and statistical analysis over all traces✓△ 4. Incremental analysis with resume support for large trace collections✓× 5. Configurable analysis and report constraints via explicit settings✓△ Methodology 1. Full execution trace modeling with explicit Thought–Action–Observation✓× 2. Structured trace abstraction preserving causal and temporal dependencies✓× 3. Insight-driven root cause analysis beyond surface-level error description✓△ 4. Cross-step causal reasoning over long-horizon agent behaviors✓× 5. Failure-centric analysis designed for systematic diagnosis✓△ Reporting 1. Multi-dimensional analysis reports with explicit diagnostic categories✓△ 2. Trace-level evidence linking conclusions to specific execution steps✓× 3. Actionable optimization suggestions grounded in diagnosed root causes✓△ 4. Generation of targeted SFT samples for downstream model improvement✓× 5. Robust analysis report quality under weaker backbone language models✓× Table 3: Capability-level comparison between TraceSIR and ClaudeCode.✓indicates native and systematic support, △ indicates partial or non-systematic support, and× indicates lack of support. The demo interface consists of the following components, labeled from A to H in Figure 3. A. LLM Configuration.This component allows users to configure the underlying model, including model name, API key, and base URL, enabling flex- ible deployment across different LLM backends. B. Upload Analysis File. Users can submit exe- cution traces in JSON format for a single case or in ZIP format for multiple cases. This component also supports an optional additional user-specified requirement field that constrains or guides report generation. Upon submission, a unique job identi- fier is returned for subsequent operations. C. Query Task Status. Given a job identifier, this component retrieves the current execution sta- tus and metadata of the analysis task, including processing progress, timestamps, and detailed logs. D. Download Analysis Results.This component enables downloading the analysis outputs. The returned archive contains all processed trace files and generated analysis reports. E. Rerun Task. This component supports re- executing an existing task using the same input traces. It allows users to resume interrupted analy- ses, which is particularly useful when processing large collections of trace cases. Previously ana- lyzed cases are not reprocessed, and the system continues only with unfinished cases. After all cases have been analyzed, users may also rerun the task to regenerate analysis reports without re- processing completed traces, optionally updating report generation requirements. F. Task Logs.This panel displays execution logs produced during trace processing and analysis, pro- viding transparency into the behavior of Struc- tureAgent, InsightAgent, and ReportAgent. G. Report Console. After report generation, users can interactively refine the analysis report through multi-turn dialogue, enabling targeted re- visions without rerunning the full analysis process. H. Documentation. This entry provides access to system documentation, including supported func- tionalities, input formats, and operational guide- lines for using TraceSIR. D Example Analysis Report To illustrate the qualitative characteristics of re- ports generated by TraceSIR, we present a repre- sentative excerpt from an analysis report, as shown in Figure 4. The report is generated in Chinese, reflecting practical deployment settings and user requirements. The complete reports, along with all related data, are available in our repository. E TraceSIR VS ClaudeCode To better contextualize the design choices and em- pirical results of TraceSIR, we provide a capability- level comparison with ClaudeCode, a strong base- line for agentic analysis. Table 3 summarizes the key differences between the two systems across system design, methodology, and reporting capa- bilities, highlighting TraceSIR’s strengths in trace- centric diagnosis and cross-case reporting. 10