Paper deep dive
CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations
Xingyu Yan, Tingting Dai, Antonio De Domenico, Mohamed Sana, Nicola Piovesan, Changchang Li, Bowen Liu, Kun Jiang, Mengjie Zhang, Dingcheng Shan, Jing-Cheng Pang, Chenwei Wu, Sijie Wu, Lianying Chao, Haoran Cai, Jiantao Ye, Xubin Li, Simon Mark Lucas, Xin Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.
Tags
Links
- Source: https://arxiv.org/abs/2608.12002v1
- Canonical: https://arxiv.org/abs/2608.12002v1
Trouble viewing inline? Open PDF directly →
Full Text
92,055 characters extracted from source content.
Expand or collapse full text
CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations Xingyu Yan ∗ , Tingting Dai ∗ , Antonio De Domenico † , Mohamed Sana † , Nicola Piovesan † , Changchang Li ∗ , Bowen Liu ∗ , Kun Jiang ∗ , Mengjie Zhang ∗ , Dingcheng Shan ∗ , Jing-Cheng Pang ∗ , Chenwei Wu ∗ , Sijie Wu ∗ , Lianying Chao ∗ , Haoran Cai ∗ , Jiantao Ye ∗ , Xubin Li ∗ , Simon Mark Lucas + , Xin Chen ∗ ∗ Huawei Technologies, China † Paris Research Center, Huawei Technologies, Boulogne-Billancourt, France + Queen Mary University of London Abstract Agents are increasingly considered for automating network operations and mainte- nance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restora- tion. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evalu- ate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis. 1 Introduction Network operations and maintenance (NetO&M) is a demanding operational domain where engineers must interpret heterogeneous telemetry, issue device-specific commands, reconstruct service paths, and diagnose faults across multiple network layers. Unlike many static question-answering settings, NetO&M requires sequential decision making: an operator must form hypotheses, select diagnostic actions, inspect command outputs, rule out alternative explanations, and produce an evidence- supported conclusion. These requirements make telecom troubleshooting a natural and challenging testbed for evaluating agentic AI systems. Recent advances in large language model (LLM) agents have demonstrated strong capabilities in multi-step reasoning, tool use, and interactive problem solving [18,17,19]. In communication technology, such agents could assist with root cause analysis, path reconstruction, configuration checking, and closed-loop remediation. Deployment remains difficult because real networks are partially observable, carry cascading cross- layer faults, and expose multi-vendor command interfaces under strict operational safety requirements. Preprint. Under review. arXiv:2608.12002v1 [cs.AI] 12 Aug 2026 BenchmarkRCAPath Restora- tion Partial Observ- ability Network Hetero- geneity Expert- Annotated Evidence Multi- Dimensional Metrics Real Equip- ment NIKA✓×✓× NetArena✓×✓× NetAgentBench✓×✓× CTBench (ours)✓ Table 1: Conceptual comparison between CTBench and existing telecom benchmarks. Existing telecom benchmarks do not reproduce these conditions. They assume full observability, use single-domain topologies, and do not focus on real network equipment. In addition, these benchmarks cannot distinguish an agent that is unable to localize a fault in a malfunctioning equipment from one that returns a correct label without supporting evidence. For operational NetO&M, this process-level transparency is a prerequisite for trust, since a diagnosis must be justified before it can be acted on. Furthermore, a realistic benchmark should assess the capabilities of any tool-using agent in a professional setting: acting under partial observability, adapting across heterogeneous interfaces, and attributing network issues across multi-hop propagation rather than stopping at the first visible symptom. We introduce CTBench, an agentic benchmark for realistic telecom troubleshooting, covering two operational task families. In RCA tasks, the agent must identify the affected node and object and assign a normalized root-cause label. In path-restoration tasks, the agent must identify source and destination endpoints and reconstruct the forwarding path. The benchmark contains 234 expert- curated tasks, 126 for RCA and 108 for path restoration. Evaluation in CTBench is grounded in the agent reasoning traces and expert practice rather than final answers alone. Each task carries an expert-normalized answer together with expert-validated golden evidence steps that capture the observations and diagnostic actions required to solve it. Tasks are built by 15 senior telecom experts and validated by an independent expert who solves the tasks without access to ground truths. Each task is further annotated with metadata describing its structural difficulty, including evidence observability, vendor and device heterogeneity and protocol complexity. Our contributions are as follows: •We introduce CTBench, a public benchmark for evaluating agentic troubleshooting in telecom NetO&M, covering 234 expert-curated RCA and path-restoration tasks. • We define telecom-expert-grounded capability metrics that separately evaluate localization, root- cause identification, path restoration, evidence acquisition, observability robustness, heterogeneous device/vendor handling, and resource usage. •We provide a task metadata schema covering observability, device/vendor heterogeneity, protocol complexity, root-cause count, restored path count, fault-propagation chains, golden solution length, and root-cause categories, enabling fine-grained diagnostic analysis beyond aggregate accuracy. •We evaluate representative agent-model combinations and show that current agents exhibit substan- tial gaps in evidence-grounded diagnosis, partial-observability reasoning, and robust handling of heterogeneous telecom environments. 2 Related Work 2.1 Benchmarks for LLM Agent General-purpose LLM agent benchmarks [9,5,6,10,2] focus on evaluating multi-step reasoning, tool use, web navigation, software engineering, and interactive planning ability. These benchmarks have been useful for measuring whether agents can decompose tasks, call tools, and synthesize intermediate observations. However, most general agent benchmarks do not capture the specific operational constraints of telecom NetO&M: vendor-specific command syntax, network topology reasoning, control-plane and data-plane interaction, partial device access, and evidence-grounded fault isolation. As a result, high performance on general agent benchmarks does not necessarily imply operational competence in telecom troubleshooting. 2 2.2 Benchmarks for Network and Telecom Several benchmarks target networking and telecom domains. Knowledge-oriented benchmarks such as TeleQnA [8] and ORAN-Bench [4] evaluate whether models understand telecom specifications or standards. Troubleshooting-oriented benchmarks such as NIKA [16], TeleLogs [12], WirelessAgent++ [13], TelcoAgent-Bench [1], NetAgentBench [14], and NetArena [20] move closer to operational network reasoning by introducing diagnostic cases, configuration tasks, and network automation scenarios [7]. These efforts provide important foundations for evaluating LLMs and agents in communication technology. Nevertheless, existing benchmarks often leave at least one major operational gap: they may assume full observability, focus on single-domain topologies, use standardized interfaces, or evaluate only final answers. CTBench complements prior work by emphasizing expert-grounded trajectory evalua- tion, partial observability, heterogeneous network environments, and evidence acquisition. Table 1 summarizes the conceptual comparison between CTBench and existing telecom benchmarks. 3 CTBench Benchmark This section formalizes the tasks in CTBench, and presents the associated datasets. 3.1 Task Formulation CTBench evaluates telecom troubleshooting as an interactive decision-making problem. Each benchmark instance is a task instructionq τ , whereτ ∈RCA, Pathdenotes the task type. Letπ θ denote an evaluated agent, including both the agent harness and the underlying model with parameters θ. At each interaction turnt, the agent selects a diagnostic action conditioned on the task instruction and the interaction history: a t ∼ π θ (·| q τ ,h t−1 ),(1) wherea t is an action such as executing a device command or querying an allowed telemetry source, andh t−1 = (a 1 ,o 1 ,...,a t−1 ,o t−1 )is the history of previous actions and observations. The task environmentE τ executes the action and returns an observationo t =E τ (q τ ,h t−1 ,a t ), whereo t may contain command outputs, configuration parameters, routing information, interface states, policy rules, or an error message if the action is invalid or unavailable. The full interaction trajectory is T (q τ ) = q τ ,a 1 ,o 1 ,a 2 ,o 2 ,...,a T ,o T , from which, the agent predicted answer ˆy τ is decoded. Root Cause Analysis. An RCA task provides a problem description and an interactive diagnostic interface. The agent must issue permitted queries or commands, inspect the returned observations, and output a normalized diagnosis. For RCA tasks, the target answer is a set of normalized root-cause triples: y ∗ RCA =(n i ,o i ,c i ) m i=1 ,(2) wheren i is the affected node,o i is the affected object, andc i is the normalized root-cause label. The affected object may be an interface, route, tunnel, access control list rule, network address translation policy, virtual private network instance, service object, or another telecom component. Path Restoration. A path-restoration task asks the agent to reconstruct a service forwarding path. For path-restoration tasks, the target answer is a reconstructed forwarding path comprising an ordered sequence of path elements p j : y ∗ Path =p 1 ,p 2 ,...,p k ,(3) Each path element p j may include a node, interface, next hop, policy, or branch decision. 3.2 Dataset Construction CTBench is constructed through an expert-in-the-loop process. Candidate tasks are abstracted from realistic telecom maintenance cases and sanitized to remove sensitive production information. 15 senior telecom domain experts with an average of 20 years of professional experience define the 3 Task Construction Tasksa r ec o n s t r u c t e d fromrealisticNetO&M casesbydomainexperts. • Problem • Evidence • Answer Independent Review Anindependentexpertsolves thetaskwithoutaccesstothe ground-truthanswer. • No ground truth • No reasoning path Consensus Review Reviewer and author compare answers, evidence, and reasoning trajectory. • Solvability • Consistency • Completeness Tasks are revised, discarded, or retained based on the review outcome. Quality Decision • Retain • Revise • Remove Check Blind Expert Review DecisionSpecification Figure 1: Process of expert-involved data quality review for CTBench. Candidate tasks are sanitized, independently reviewed by telecom experts, checked for answer and evidence consistency, revised when necessary, and retained only after consensus on solvability and operational validity. problem statement, available device outputs, permitted command interface, standard answer, and task-side metadata. The current benchmark contains 126 RCA tasks and 108 path-restoration tasks. To ensure task validity, each candidate task is reviewed by an independent telecom expert who is not involved in the original construction. The reviewer receives the problem description and permitted interface but not the ground-truth answer or intended trajectory. The reviewer independently solves the task, after which the task author and reviewer compare final answer, supporting evidence, and reasoning trajectory. Tasks are retained only when experts reach consensus on solvability, answer completeness, evidence consistency, and alignment with the intended network behavior. Ambiguous, underspecified, or unverifiable cases are revised or discarded. Figure 1 summarizes this expert-involved data quality workflow, from task construction and sanitiza- tion to independent expert review, consensus checking, revision, and final release. 3.3 Golden Evidence Annotation A key feature of CTBench is that it evaluates not only the final answerˆy τ , but also the diagnostic process used to obtain it. Each task is associated with an expert-validated golden solution represented as a set of golden actions: A ∗ (q τ ) =a ∗ 1 ,a ∗ 2 ,...,a ∗ T ∗ ,(4) where eacha ∗ i denotes a key diagnostic action that a telecom expert considers necessary for solving the task and T ∗ is the total number of steps involved. In CTBench, golden actions are defined from expert trajectories and therefore provide a deterministic reference for evaluating whether an agent can follow the key operational steps used by telecom experts. For RCA, golden actions capture the diagnostic checks needed to localize and justify the root cause. For path restoration, they capture the checks needed to reconstruct the forwarding path, including endpoint reachability, hop adjacency, routing decisions, interface transitions, policy constraints, tunnel state, or branch behavior. Concrete annotation examples are provided in the supplementary material. 3.4 Task Metadata CTBench annotates tasks with metadata that enables fine-grained analysis of agent capability. Evidence Observability. For RCA tasks, CTBench defines two levels of evidence observability, summarized in Table 2. This label, denoted byO(q τ )∈O 1 ,O 2 , indicates whether a fault evidence can be obtained directly through a permitted diagnostic command, or whether the agent must infer the fault from partial observations obtained through indirect commands. Root-Cause Categories. Each RCA task is assigned to one or more root-cause categories. We define the categorization taxonomy at the level of individual root causes rather than at the task level: a single troubleshooting scenario may involve multiple underlying faults and can therefore be associated with multiple categories. Table 3 summarizes the resulting five-category taxonomy. 4 Label ObservabilityExplanation O1FullThe agent can obtain decisive evidence of the fault by directly implementing a permitted diagnostic command. O2PartialThe direct diagnostic command is unavailable or fails. Therefore, the agent must rely on indirect commands that expose only partial observations, requiring additional reasoning and cross-checking to identify the correct root cause. Table 2: Observability levels in CTBench. Label CategoryDistribution C1Interface State and Link-Layer Faults28.04% C2Security, Network Address Translation (NAT), and Edge Access Control24.30% C3Routing Protocol and Policy Control16.82% C4High Availability and Reliability Mechanisms15.89% C5Service, Management, and Other Operational Faults14.95% Table 3: RCA category taxonomy and gold-entry distribution. Root Cause count.For RCA tasks, CTBench records the number of independent gold root causes, denoted byN r (q τ ). This metadata captures whether a troubleshooting scenario requires identifying a single failure or multiple simultaneous faults. Tasks with multiple root causes are structurally harder because the agent must distinguish independent causes from downstream symptoms and avoid returning an incomplete or overly broad diagnosis. Fault-propagation chain.For RCA tasks, CTBench annotates the fault-propagation-chain length N c (q τ )when applicable. This metadata describes the causal path from the latent root cause to intermediate network states and finally to the observed symptom. Longer or more indirect propagation chains increase diagnostic difficulty because the agent must reason beyond the first visible symptom and identify the underlying cause. Restored path count.For path-restoration tasks, CTBench records the number of restored forward- ing paths in the gold answer, denoted byN path (q τ ). A task may require reconstructing a single path or multiple paths caused by branching, redundancy, load balancing, or service-specific forwarding behavior. Multi-path cases are more difficult because the agent must restore path multiplicity rather than only one route. Protocol complexity. For both RCA and path-restoration tasks, CTBench records protocol com- plexity, denoted byN p (q τ ). This metadata captures the number and interaction depth of forwarding, control-plane, policy, redundancy, and service mechanisms involved in solving the task. Higher protocol complexity indicates that the correct answer depends on reasoning across multiple interacting mechanisms rather than inspecting a single local state. Network Heterogeneity. Network heterogeneity captures the number of distinct vendor environ- mentsN v (q τ )and device typesN d (q τ )the agent must handle. As shown in Table 4, we assign to each task a network heterogeneity level comparing the maximum between vendor heterogeneity and device-type heterogeneityh L (q τ ) = max(N v (q τ ),N d (q τ ))with task-calibrated thresholds for RCA and path-restoration tasks. Table 4 summarizes the reporting levels. Golden Solution Length. For both task types, CTBench records the golden solution length T ∗ (q τ ) = |A ∗ (q τ )|, defined as the number of key evidence steps required to solve the task. This metadata captures the procedural burden of the task: a longer golden path indicates that the agent must collect, connect, and use more pieces of evidence before reaching a justified answer. Using Metadata to Characterize Task Difficulty. These metadata fields allow CTBench to characterize task difficulty as a structural property of the task. Evidence observability captures information availability; root-cause count and restored path count capture answer multiplicity; fault- propagation chain length captures causal depth; protocol complexity captures cross-layer reasoning 5 LabelNameRCAPath Restoration H-LowLow heterogeneityh L = 1 h L ≤ 6 H-High High heterogeneityh L ≥ 2 h L > 6 Table 4: Network heterogeneity levels. The levels are defined based on the number of vendors and devices type involved in the solution. Ground Truth Action Space Execution Constraints Environment Data Harness Model Skills Container (Docker) API Layer CTBenchScenarios CTBenchRuntime Environment Legend Benchmark Builder Developer Build & RegisterBuild & Register Command Echo Setup Scenario Load Agent System Agent-Access Layer (AAL) Acc. Runtime ...... Evidence Evaluator Action Info Agent Result Rank & Publish Benchmark Builder Developer CTBench Scenario AI Agent Container (Docker) Network Simulator Evaluator Leaderboard AI Agent Scenario Example Question: Access from EMPLOYEE_WIFI_CLIENT02 to SZ_Server_Cluster2(10.3.20.1) is unreachable Ground Truth: (node, object, root cause) = (core1, GE1/0/3, shutdown) Metadata: heterogeneity = Low · observability = O2 · GSP = short Env Data: topology · devices(vendor, type) · command echo Interaction Example Action: core1 "display interface brief" Info: S>* GE1/0/3 down down -- -- -- (shutdown) Answer: core1, GE1/0/3, shutdown correct Pre-defined Problem Pool Metadata Question Codex GPT SKILL.md (SOP) Leaderboard Hermes ClaudeCode Qwen DeepSeek Network Simulator Network Simulator ■ switch · HUAWEI ■ router · CISCO ■ firewall · H3C ■ host · Linux Multi-Vendor HUAWEI · CISCO · H3C Multi-Type router · switch · firewall · host Figure 2: CTBench Automatic Evaluation framework. burden; device/vendor heterogeneity captures semantic and operational diversity; and golden solution length captures the number of expert-required diagnostic steps. For RCA tasks, these dimensions describe the difficulty of localizing, identifying, and justifying root causes. For path-restoration tasks, difficulty is instead driven by endpoint identification, path multiplicity, forwarding structure, protocol interactions, heterogeneity, and the length of the evidence path. We use these annotations both for fine-grained reporting and for analyzing which operational conditions most challenge current agents. 4 Evaluation Protocol Figure 2 illustrates the overall evaluation framework of CTBench. Given a task instanceq τ and an agent trajectoryT (q τ ), the evaluator compares the predicted answerˆy τ and collected evidence ˆ A(q τ )against the ground truthy ∗ τ and the related expert-annotated metadataA ∗ (q τ ). In addition to the average task accuracyAcc =E[1(y ∗ τ = ˆy τ )], CTBench reports task-specific capability metrics including cost and efficiency metrics. 4.1 RCA Capability Metrics For RCA, CTBench defines three complementary capabilities. RCA Localization.RCA Localization measures the agent capability to identify where the problem occurs, including the affected node, interface or service object. LetL ∗ =(n i ,o i ) m i=1 denote the gold set of affected node-object pairs extracted fromy ∗ RCA and ˆ Lthe predicted set extracted from ˆy RCA . We compute the RCA localization score (RCA-Loc) as the intersection over union (IoU) between the predicted and gold localization sets: RCA-Loc = IoU( ˆ L,L ∗ ),(5) 6 where IoU(A,B) = |A∩B| |A∪B| . RCA Identification.RCA Identification measures the agent capability to identify the correct root- cause label, such as interface down, missing route or network misconfiguration. LetC ∗ =c i m i=1 denote the gold set of normalized root-cause labels and ˆ Cthe predicted set. Similar to RCA-Loc, we compute the RCA Identification score (RCA-ID) as the IoU between the predicted and gold root-cause label sets: RCA-ID = IoU( ˆ C,C ∗ ).(6) RCA Evidence. RCA Evidence quantitatively assesses whether the agent collects relevant key diagnostic evidence needed to identify the root cause. To do so, we compare the agent actions with the expert golden actions using the F1 score. Specifically, letA ∗ RCA and ˆ A RCA denote the golden evidence and the agent steps, respectively. We compute: RCA-Evidence = F1 ˆ A RCA ,A ∗ RCA .(7) A step is considered covered only if the agent queries the correct device with the correct command to retrieve the relevant observation required for diagnosing the problem. Remark (Hierarchical Scoring). The main RCA metrics above use strict set matching to accurately assess agent capabilities. To support near-miss analysis, we additionally define hierarchical RCA scoring and topology-aware node similarity as supplementary metrics. These metrics are not used as primary correctness measures; instead, they help distinguish close errors from completely unrelated predictions. The complete taxonomy, similarity rules, procedure, and associated results are provided in the supplementary material. 4.2 Path Restoration Capability Metrics Similar to RCA, CTBench defines three capability metrics. Path Localization. Path localization measures whether the agent correctly identifies the relevant endpoints involved in the path-restoration task. LetP ∗ end and ˆ P end be the gold and predicted endpoint sets. We compute path localization score as the IoU between the gold and the predicted sets: Path-Loc = IoU( ˆ P end ,P ∗ end ).(8) Path Restoration.Path Restoration measures whether the agent reconstructs the correct forwarding path. LetP ∗ denote the reference path edge sets and ˆ P the predicted path edge sets. We compute the path restoration score as IoU between predicted and reference path edge sets: Path-Res = IoU( ˆ P,P ∗ ).(9) Path Evidence. Path Evidence measures whether the agent collects and uses the key evidence required to reconstruct the path, such as routing entries, interface states or forwarding-table outputs. Similar to RCA evidence score, letA ∗ Path and ˆ A Path denote the golden evidence steps and the agent steps, respectively. We compute: Path-Evidence = F1 ˆ A Path ,A ∗ Path .(10) 4.3 Cost and Efficiency Metrics We additionally report latency, interaction rounds, and token consumption. Token consumption is defined as: Tokens total = Tokens input + Tokens output .(11) These quantities are treated as cost and efficiency descriptors rather than substitutes for diagnostic capability. 7 Agent RCAPath Restoration RCA Acc RCA-LocRCA-ID RCA-Evid. Path AccPath-LocPath-Rest Path-Evid. Codex+GPT-5.547.62±4.5 52.90±3.7 66.83±3.7 15.80±1.0 87.96±3.2 99.38±0.6 95.28±1.4 47.84±1.5 ClaudeCode+Qwen3.7-Plus19.84±3.6 34.68±3.4 38.96±3.8 12.62±0.8 17.59±3.7 87.65± 2.8 48.19±3.0 26.58±1.6 HermesAgent+DeepSeek-V4-Pro 17.46±3.4 29.43±3.6 40.08±3.9 10.36±0.8 26.85±4.3 85.49±2.9 59.26±2.8 21.85±1.3 HermesAgent+Qwen3.7-Max25.40±3.9 37.59±3.6 52.38±3.9 12.59±0.8 40.74±4.8 81.48± 3.6 59.00±3.5 24.06±1.7 HermesAgent+TelecomGPT-R14.76±1.9 12.41±2.9 11.37±2.5 5.23±0.71.85±1.3 75.31±3.6 10.71±2.0 6.13±0.8 Table 5: Overall capability results. Scores are percentages and reported as mean±standard deviation error. 5 Experiments and Analysis Weevaluatefiverepresentativeagent-modelcombinations:Codex+GPT-5.5, ClaudeCode+Qwen3.7-Plus, HermesAgent+DeepSeek-V4-Pro, HermesAgent+Qwen3.7-Max, and HermesAgent+TelecomGPT-R1 [15]. All agents are evaluated on the same task set and are not given access to ground-truth answers or golden evidence steps during inference. Each task is executed in a reactive sandbox environment that exposes the permitted telecom diagnostic interface. Agents interact with the environment by issuing commands or tool calls and receiving observations. We log the full trajectory, including device queries, returned observations, final answers, latency, interaction rounds, and token usage. The same answer normalization and evidence-matching rules are applied across all agents. Full environment details, prompts, and code are provided in the supplementary material. 5.1 Overall Capability Results Table 5 reports the main capability results. We report the task accuracy together with the capability metrics presented in Sec. 4: localization, identification, restoration, and evidence acquisition. In our experiments, agents perform better on path-restoration tasks than on RCA tasks. Codex+GPT-5.5 obtains the strongest overall results, and it performs especially well on path restoration. The results show also that final accuracy alone hides important differences between agent capabilities. For example, Codex+GPT-5.5 reaches only 47.84% Path-restoration evidence, showing that producing a correct or plausible answer does not imply that the agent produces the evidence expected from a telecom troubleshooting engineer. Similarly, an agent (e.g., HermesAgent+DeepSeek-V4-Pro) may achieve high path-localization performance (Path-Loc) while still failing to restore the full path (Path Macro IoU) or collect sufficient diagnostic evidence (Path-Evidence). In addition, agents are better in RCA identification than in RCA localization e.g., 66.83%vs 52.9%for Codex+GPT-5.5 and 52.38% vs 37.59%for HermesAgent+Qwen3.7-Max, respectively, indicating that agents sometimes infer the right fault type but fail to identify the precise faulty node or object. 5.2 Cost and Efficiency Table 6 reports interaction rounds, latency, and token consumption. These metrics are cost and effi- ciency descriptors, rather than capability measures. The results show that higher computational cost does not necessarily translate into stronger diagnostic capability. For instance, ClaudeCode+Qwen3.7- Plus consumes substantially more rounds and tokens than Codex+GPT-5.5, but obtains lower overall capability scores. In constrast, HermesAgent+Qwen3.7-Max outperforms HermesAgent+DeepSeek- V4-Pro in path restoration (see Path Acc) but at a cost of higher latency and token consumption. This motivates reporting cost alongside capability rather than using latency or token consumption as proxies for agent competence. Remark (Human expert results). As a reference, during the independent review phase (see Figure 1), human experts have achieved 92.6%and 56.4%accuracy on path-restoration tasks and RCA tasks, respectively. In addition, it took them between 40 and 60 minutes to complete each of these tasks. 5.3 RCA Performance by Fault Category Figure 3 visualizes RCA accuracy across the five root-cause categories presented in Table 3. The category-level results show that agent failures are not uniformly distributed across telecom fault 8 AgentTaskRdsLat.(s)Tokens Codex+GPT-5.5 RCA 10.81 333.40476.5k Codex+GPT-5.5 Path 14.14 419.80 1019.2k ClaudeCode+Qwen3.7-Plus RCA 81.36 1234.64 2751.7k ClaudeCode+Qwen3.7-Plus Path 93.85 1241.09 3143.4k HermesAgent+DeepSeek-V4-Pro RCA 36.62 483.62 1491.7k HermesAgent+DeepSeek-V4-Pro Path 38.66 497.84 1453.1k HermesAgent+Qwen3.7-Max RCA 31.09 1352.93 1218.6k HermesAgent+Qwen3.7-Max Path 36.77 1482.64 1574.6k HermesAgent+TelecomGPT-R1 RCA 31.76 641.77656.2k HermesAgent+TelecomGPT-R1 Path 35.53 934.40s 797.1k Table 6: Efficiency and cost results. 40% 60% C1 Interface/Link C2 Security/NAT Access C3 Routing/Policy C4 HA/Reliability C5 Service/Mgmt Other Ops Codex+GPT-5.5 ClaudeCode+Qwen3.7-Plus HermesAgent+DeepSeek-V4-Pro HermesAgent+Qwen3.7-Max HermesAgent+TelecomGPT-R1 Figure 3: RCA accuracy across root-cause categories for the five evaluated agent-model combinations. The radar plot shows that agent performance varies substantially across fault families rather than shifting uniformly across all categories. families. Two patterns holds across all agents. C2 faults are the most reliably diagnosed by every agent, and C1 is uniformly hard, with accuracy between 0.00% and 10.71%. The remaining ordering is agent-dependent. C3 separates the agents sharply: no agent other than Codex+GPT-5.5 solves a single C3 task, while Codex+GPT-5.5 reaches 40.00% and fails instead on C1. Codex+GPT-5.5 is also the only agent with a wide gap between C4 and C5 (50.00% versus 28.12%); the remaining agents differ by at most four points on these two categories. This suggests that CTBench does not only measure generic troubleshooting ability: it also reveals which telecom fault families remain challenging for current agents. 5.4 Impact of Observability and Heterogeneity We next analyze how task metadata explains agent behavior. To save space, the main paper reports this analysis for ClaudeCode+Qwen3.7-Plus as a representative agent. Full results for all agents are provided in the supplemental material. 9 Observability.For ClaudeCode+Qwen3.7-Plus, RCA performance drops under partial observabil- ity. On fully observable RCA tasks, the agent obtains 20.56% accuracy, 37.02% localization, 42.92% RCA identification, and 38.29% evidence coverage. On partially observable tasks, accuracy decreases to 15.79%, localization drops to 17.14%, and RCA identification drops to 16.22%. Interestingly, evidence coverage increases to 62.20%, suggesting that the agent can still collect some indirect observations but struggles to convert them into precise fault localization and root-cause identification. This supports the role of observability as a structural difficulty factor: partial evidence does not merely slow the agent down; it changes the nature of the reasoning required. Network heterogeneity. Network heterogeneity also affects performance, especially for path restoration. For ClaudeCode+Qwen3.7-Plus, path accuracy decreases from 29.31% in low hetero- geneity settings to 4.00% in high heterogeneity settings. Path localization also decreases from 94.96% to 72.73%, while path restoration IoU drops from 68.58% to 35.05%. This indicates that heteroge- neous multi-device settings make it harder for the agent to integrate evidence across device roles, vendor-specific command outputs, and forwarding contexts. For RCA, the effect is less uniform: accuracy decreases from 33.33% to 13.79%, while localization and RCA identification remain similar or slightly increase. This suggests that heterogeneity interacts with other factors such as fault category, observability, and protocol complexity, and should not be used alone to define task complexity. 6 Discussion Performance is multi-faceted. A key lesson from CTBench is that agent performance should be measured in detailed ways: separating localization, identification, restoration, and evidence acquisition exposes distinct variations in abilities. Evidence-grounded diagnosis remains difficult. Even when agents produce plausible final an- swers, they often do not produce evidences that telecom experts consider necessary. This is critical for operational trust, where a diagnosis must be justified before remediation. Partial observability and heterogeneity expose agent limitations. Under partial observability, agents must reason from indirect evidence and cross-checks. Under heterogeneity, they must normal- ize device roles, vendor-specific commands, and output formats. The observed drops in localization, identification, and restoration show that these are not merely implementation details; they are core dimensions of telecom-agent capability. CTBench reflects general agent challenges.Although focusing on telecom operations, CTBench presents broader agentic capabilities required in complex professional environments, including partial-observation reasoning, interaction with heterogeneous environment, causal attribution, and long-horizon evidence planning. The supplementary material provides full capability analysis derived from trajectory-level failures. 7 Limitations CTBench can be extended in future work. First, CTBench covers only two types of tasks: Path- Restoration and RCA. The capability of dynamically repairing faults in the network environment by the evaluation agent is not covered. This will be supplemented in the future. In addition, while the benchmark covers realistic telecom troubleshooting, it does not yet exhaustively cover all domains such as full radio-access-network operations, core-network slicing, or cloud-native telecom infrastructure. 8 Conclusion We presented CTBench, an agentic benchmark for realistic telecom network operations and mainte- nance. Using with fine-grained data annotations and multi-dimensional metrics, CTBench assesses agent proficiency in fault localization, root-cause isolation, path reconstruction, expert-aligned 10 evidence extraction, and reasoning robustness under partial observability and heterogeneous envi- ronments. Experiments with representative agent-model combinations reveal that current agents remain far from reliable telecom troubleshooting operators, especially in evidence-grounded RCA and partially observable settings. We hope CTBench will support future research on trustworthy, domain-grounded, and operationally safe AI agents for communication networks. References [1]Lina Bariah, Brahim Mefgouda, Farbod Tavakkoli, Enrique Molero, Louis Powell, and Mer- ouane Debbah. Telcoagent-bench: A multilingual benchmark for telecom ai agents, 2026. [2]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ 2 -Bench: Evaluating conversational agents in a dual-control environment. In Proceedings of the Forty- Third International Conference on Machine Learning (ICML), 2025. [3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [4]Pranshav Gajjar and Vijay K. Shah. ORAN-Bench-13K: An open source benchmark for assessing LLMs in open radio access networks. In Proceedings of the 2025 IEEE 22nd Consumer Communications & Networking Conference (CCNC), pages 1–4, 2025. [5]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. [6]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. [7] Ali Maatouk, Kenny Chirino Ampudia, Rex Ying, and Leandros Tassiulas. Tele-llms: A series of specialized large language models for telecommunications. IEEE Access, 14:86424–86441, 2026. [8] Ali Maatouk, Fadhel Ayed, Nicola Piovesan, Antonio De Domenico, Merouane Debbah, and Zhi- Quan Luo. TeleQnA: A benchmark dataset to assess large language models telecommunications knowledge. IEEE Network, 2025. [9]Gr ́ egoire Mialon, Cl ́ ementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. In Proceedings of the Twelfth Interna- tional Conference on Learning Representations (ICLR), 2024. [10] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Hao Peng, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. [11]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. [12]Mohamed Sana, Nicola Piovesan, Antonio De Domenico, Yibin Kang, Haozhe Zhang, Mer- ouane Debbah, and Fadhel Ayed. Reasoning language models for root cause analysis in 5g wireless networks, 2025. [13] Jingwen Tong, Zijian Li, Fangyu Liu, Wei Guo, and Jun Zhang. Wirelessagent++: Automated agentic workflow design and benchmarking for wireless networks, 2026. 11 [14]Ahmed Twabi, Yepeng Ding, and Tohru Kondo. Netagentbench: A state-centric benchmark for evaluating agentic network configuration, 2026. [15]Bohao Wang, Chenwei Wu, Haoyu Li, Hang Zou, Yu Tian, Lina Bariah, Chongwen Huang, Yongliang Shen, Zhaoyang Zhang, and M ́ erouane Debbah. TelecomGPT-R1: Post-training recipes for universal reasoning in telecom. https://huggingface.co/KU-DFI/TelecomGPT-R1, 2026. [16]Zhihao Wang, Alessandro Cornacchia, Alessio Sacco, Franco Galante, Marco Canini, and Dingde Jiang. A network arena for benchmarking ai agents on network troubleshooting. In Proceedings of the Fourteenth International Conference on Learning Representations (ICLR), 2026. [17]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824– 24837, 2022. [18]Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, Qi Zhang, and Tao Gui. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025. [19] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023. [20]Yajie Zhou, Jiajun Ruan, Eric S. Wang, Sadjad Fouladi, Francis Y. Yan, Kevin Hsieh, and Zaoxing Liu. Netarena: Dynamic benchmarks for ai agents in network automation. In Proceedings of the Fourteenth International Conference on Learning Representations (ICLR), 2026. 12 Supplementary Material for Paper: CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations Table of Contents A Dataset and Annotation Details14 A.1 RCA task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 A.2 Path Restoration task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 A.3 Annotation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 A.4 Metadata Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.5 RCA Metadata Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.6 Path-Restoration Metadata Distributions . . . . . . . . . . . . . . . . . . . . . . .17 A.7 Task Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 B Supplementary Evaluation Metrics17 B.1 Hierarchical RCA Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 B.2 Topology-Aware Node Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . .19 B.3 Root-Cause Type Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 C Supplementary Evaluation Results19 C.1 Full RCA Category Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 C.2 Hierarchical RCA Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 C.3 Observability Results for All Agents . . . . . . . . . . . . . . . . . . . . . . . . .19 C.4 Heterogeneity Results for All Agents . . . . . . . . . . . . . . . . . . . . . . . . .19 C.5 Structural Difficulty Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 C.6 Evidence-Grounded Performance Analysis . . . . . . . . . . . . . . . . . . . . . .22 C.7 Harness Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 D Reproducibility Details24 D.1 Evaluation Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24 D.2 Agent Harnesses and Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . .25 D.3 Command Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 D.4 Prompt Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 E Trajectory-Level RCA Failure Analysis26 E.1 Overview of Trajectory-Level Failure Analysis . . . . . . . . . . . . . . . . . . .26 E.2 Failure Mode Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27 E.3 Representative Failure Trajectories . . . . . . . . . . . . . . . . . . . . . . . . . .27 E.4 Implications for Agent Design . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 F Discussion of Agent Capabilities in CTBench29 13 Real-World TelecommunicationsNetwork Complexity Internet CE Switch USG Firewall Core Switch Aggregation Switch Huawei Huawei H3C Huawei ASR Router Cisco Servers Linux Router Huawei Xclient Linux PC Linux L2 Access & Link:VLAN, LACP, MSTP, LLDP L3 Routing / Underlay:Static Route, OSPF, ISIS, BGP Tunneling & Overlay:VXLAN, MPLS LDP, SRv6 VPN & Services:BGP VPN, BGP EVPN Security, HA & Net Services: VRRP, NAT, ACL, DHCP Protocols Figure 4: Representation of the telecom network under study with multi-vendor equipment and realistic networking protocols. A Dataset and Annotation Details CTBench is constructed through an expert-in-the-loop process. CTBench involves real network equipment with different vendors, equipment, protocols, and interfaces (see Figure 4). Candidate tasks are abstracted from realistic telecom maintenance cases and sanitized to remove sensitive production information. 15 senior telecom domain experts with an average of 20 years of professional experience define the problem statement, available device outputs, permitted command interface, standard answer, and task-side metadata. The current benchmark contains 126 RCA tasks and 108 path-restoration tasks. A.1 RCA task The RCA subset contains 126 tasks and 214 normalized root-cause triples. Since a single troubleshoot- ing task may contain multiple independent faults, the number of root-cause entries can exceed the number of tasks. The dataset also contains 176 task-level root-cause mentions, where each root-cause type is counted at most once per task. The difference between 214 entries and 176 mentions reflects repeated affected objects or devices for the same root-cause type within some tasks. The 17 normalized root-cause labels are grouped into seven operational fault domains for dataset characterization. The main paper merges sparse domains into C5 for stable reporting, while Table 7 reports the complete unmerged distribution. Fault domainTypesEntriesRatio Interface and link-state faults16028.04% Security, NAT, and boundary access control35224.30% Routing protocol and policy control23616.82% High availability and reliability mechanisms23415.89% Overlay and VPN service faults42612.15% Basic addressing and Layer-2 access configuration452.34% Monitoring and operational visibility110.47% Total17214100.00% Table 7: Root-cause category coverage in CTBench fault-localization tasks. Table 8 reports the fine-grained root-cause distribution. The distribution is intentionally not class- balanced: CTBench preserves frequent operational failure modes while retaining diagnostically distinct long-tail cases. We further merge the fine-grained root-causes into five-category taxonomy shown in Table 9. 14 Fine-grained root causeFault domainEntriesRatioTask ment. ShutdownInterface/link state6028.04%28 Security policy does not permit the userSecurity/NAT/access control4119.16%41 IP prefix list misses the corresponding source IP Routing/policy control2411.21%24 Global STP is not enabledHigh availability/reliability2310.75%23 L3VPN misconfigurationOverlay/VPN service177.94%17 OSPF misconfigurationRouting/policy control125.61%6 Global HRP hot-standby protocol is not enabled High availability/reliability115.14%11 NAT outside-interface attribute error or omission Security/NAT/access control73.27%7 NAT inside-interface attribute error or omission Security/NAT/access control41.87%4 SRv6-Policy tunnel planning errorOverlay/VPN service41.87%4 VXLAN misconfigurationOverlay/VPN service41.87%4 MAC address misconfigurationBasic L2/access config20.93%2 Interface IP errorBasic L2/access config10.47%1 Interface VLAN misconfigurationBasic L2/access config10.47%1 VPN configuration missingOverlay/VPN service10.47%1 Host-information collection missingMonitoring/visibility10.47%1 Loopback IP conflictBasic L2/access config10.47%1 Total–214100.00%176 Table 8: Fine-grained root-cause distribution in CTBench. Task mentions count each root-cause type at most once per task. CategoryEntriesRatio C1 Interface/link state6028.04% C2 Security/NAT/access5224.30% C3 Routing/policy3616.82% C4 HA/reliability3415.89% C5 Service/management/other3214.95% Table 9: Merged RCA category distribution over 214 gold root-cause entries. A.2 Path Restoration task The path-restoration subset contains 108 tasks. Each task asks the agent to reconstruct the forwarding path from a source endpoint to a destination endpoint or destination IP. Unlike RCA tasks, whose answers are sets of root-cause triples, path-restoration answers are ordered network paths. A valid answer must preserve hop order, path multiplicity, and physical egress-interface semantics. The required output uses one line per path, connects consecutive hops with->, writes each non-terminal hop as nodeegress-interface, and writes the terminal endpoint as the node name only. Each golden path is decomposed into normalized node records with node names, egress interfaces, and raw output segments. This representation supports exact-match scoring while also enabling more diagnostic overlap measures, such as endpoint correctness, ordered edge overlap, interface overlap, and path-count correctness. A.3 Annotation Protocol For RCA tasks, standard answers are normalized into one or more minimal root-cause triples of the form(faultnode,faultobject,rootcause). For path-restoration tasks, standard answers preserve node order, path multiplicity, and interface semantics. Golden solution paths are annotated as key evidence steps rather than complete command logs. Each retained step must correspond to necessary evidence acquisition, verification, or use. For multi-root-cause and multi-path tasks, reviewers check answer completeness and evidence sufficiency rather than enforcing a single unique reasoning trajectory. Cases with unresolved expert disagreement are revised or removed before release. 15 Metadata nameFault localizationPath restoration Golden answerNormalized root-cause triples in standard answer.faults Ordered endpoint-to-endpoint paths in standardanswer.paths Root-cause / path count datalabels.rootcausecount; datalabels.hasmultiplerootcauses datalabels.restoredpathcount; datalabels.uniquenodecount RCA category labels datalabels.rcacategorylabels; datalabels.rcacategories Not applicable Evidence observabil- ity evidenceobservability.tier; direct/degraded observation pairs Not applicable Fault-propagation chain fault-propagation chain. total propagationstepcount; causal propagation steps Not applicable Golden solution path goldensolutionpath. goldensolutionlength;diagnostic key steps goldensolutionpath. goldensolutionlength;path- reconstruction key steps Network heterogene- ity network heterogeneity.level; vendor/type/device counts and key devices networkheterogeneity.level; vendor/type/device counts and key devices Protocol complexity protocolcomplexity. mechanismcount; root-cause, diagnos- tic, propagation, and observability mecha- nisms protocolcomplexity. mechanismcount; forwarding, diagnos- tic, and path-restoration mechanisms Table 10: Metadata fields for the two evaluated CTBench task types. Evidence Observability TierNRatio O1 direct107 84.92% O2 indirect19 15.08% Root-Cause Count TierNRatio RC166 52.38% RC244 34.92% RC3+16 12.70% Fault-Propagation Chain TierNRatio Low (≤ 4)19 15.08% Medium (5–13)87 69.05% High (> 13)20 15.87% Golden Solution Path TierNRatio Short (≤ 4)73 57.94% Long (> 4)53 42.06% Device Heterogeneity TierNRatio H-Low (=1)39 30.95% H-High (¿1)87 69.05% Protocol Complexity TierNRatio Low/Mod. (≤ 7) 76 60.32% High (> 7)50 39.68% Table 11: RCA question-level component distributions over 126 tasks. A.4 Metadata Schema CTBench annotates each task with task-side metadata for dataset auditing, structural difficulty analysis, and post-hoc trajectory evaluation. These metadata fields are not provided to agents during inference. Table 10 summarizes the metadata fields used for the two evaluated task types. A.5 RCA Metadata Distributions Table 11 summarizes the question-level component distributions for the 126 RCA tasks. These metadata fields characterize observability, answer multiplicity, causal depth, evidence-path length, network heterogeneity, and protocol complexity. The O2 subset contains tasks in which decisive evidence must be inferred from indirect observations. The multi-root-cause subset tests answer completeness. Fault-propagation-chain length captures causal depth, while golden solution length captures the number of expert-required evidence steps. Device heterogeneity and protocol complexity capture the operational diversity and cross-layer reasoning burden of the task. 16 Restored Path Count TierNRatio P1 one path64 59.26% P2 two paths20 18.52% P3+ three or more24 22.22% Golden Solution Path TierNRatio Short/Mod. (≤ 33)74 68.52% Long (> 33)34 31.48% Device Heterogeneity TierNRatio H-Low (≤ 6)58 53.70% H-High (> 6)50 46.30% Protocol Complexity TierNRatio Low/Mod. (≤ 4)48 44.44% High (> 4)60 55.56% Table 12: Path-restoration question-level component distributions over 108 tasks. A.6 Path-Restoration Metadata Distributions The path-restoration subset contains 108 tasks. Table 12 summarizes the main path-restoration metadata distributions. Component counts are computed from normalized gold paths and metadata in the released task files. Most path-restoration tasks have one gold path, but 44 tasks require multiple paths. These multi-path cases stress branching, redundancy, load balancing, and service-specific forwarding behavior. Path- restoration tasks also have longer evidence paths than RCA tasks because the agent must reconstruct complete forwarding behavior rather than identify one localized root cause. A.7 Task Examples We provide one RCA example and one path-restoration example to illustrate how task descriptions, gold answers, metadata, and golden evidence steps are connected. Agents receive only the task description and permitted tools during evaluation; they do not receive gold answers, metadata labels, or golden evidence steps. RCA example. RCA-90 asks for the minimal set of root causes explaining why ping fromSite1 area, Access-PC-01to20.1.1.10is unreachable. The gold answer contains two port-fault entries:BoardLeaf-01;GE1/0/0;shutdownandBoardLeaf-01;GE1/0/1;shutdown. Both en- tries belong to C1 interface-state faults. The task is an RC2 task with two independent gold root causes. It is O1 because decisive interface-state evidence is directly observable onBoardLeaf-01. The golden solution path has three key steps, including interface description, current configuration, and LLDP neighbor checks. The task is H-Low because the normalized heterogeneity count is one device-role family. Path-restoration example.PR-100 asks the agent to restore the path fromSZServerCluster3 in the Shenzhen data center toSHSALPC01at10.2.10.1in the Shanghai branch. The gold answer contains one 12-element path fromSZServerCluster3through the Shenzhen core, PE devices, Beijing gateway, Shanghai aggregation, and finallySHSALPC01. The task is a P1 task because the normalized gold answer contains one path. Its golden solution path has 33 expert key steps, placing it at the upper boundary of the Short/Moderate tier. Representative evidence includes route-table checks and LLDP-neighbor checks. The task is H-Low under the path-restoration threshold because its normalized heterogeneity count is six, and its protocol-complexity mechanism count is four. B Supplementary Evaluation Metrics The main paper reports strict IoU-based RCA localization and identification metrics. This section defines supplementary metrics used only for diagnostic error analysis. These metrics do not replace the primary scores. 17 Match levelSimilarityInterpretation Exact label1.0The predicted normalized root-cause label is identical to the gold label. Same subtype0.8The prediction falls under the same fine-grained failure sub- type but misses the exact normalized label. Same technical do- main 0.6The prediction belongs to the same operational domain, such as security/NAT access control, routing/policy control, or high-availability mechanisms. Samesuper- category 0.4 The prediction remains within the same broad fault fam- ily, such as connectivity, policy, service, redundancy, or operational-visibility faults. Unrelated0.0The prediction belongs to a different fault family or cannot be mapped to the root-cause taxonomy. Table 13: Root-cause hierarchy similarity used by hierarchical RCA scoring. Node relationSimilarity compo- nent Interpretation Exact node1.0The normalized predicted node is identical to the gold node. Same logical cluster0.7 taxonomy scoreThe nodes are in the same redundancy pair, HA pair, or inferred logical device cluster. Same site and role0.5 taxonomy scoreThe nodes share both site and device-role metadata. Same site or same role0.3 taxonomy scoreThe nodes share only one of site or role. Topology distance 10.6 topology scoreThe nodes are one hop apart in the LLDP-derived topol- ogy graph. Topology distance 20.3 topology scoreThe nodes are two hops apart in the LLDP-derived topol- ogy graph. Disconnected or farther0.0 topology scoreThe nodes are disconnected, unknown, or at distance at least three. Table 14: Topology-aware node similarity used by hierarchical node location. B.1 Hierarchical RCA Scoring Strict RCA identification treats all incorrect root-cause labels equally. This binary treatment hides wheteher an incorrect label is outside the relevant mechanism family or only differs at a finer taxonomy. We therefore report a supplementary hierarchical RCA score based on the expert-defined RCA taxonomy in Table 13. The score gives partial credit when the predicted root-cause label matches the gold label at coarser levels of the taxonomy, such as the same subtype, technical domain, or broad fault family. This follows the common use of semantic hierarchies in classification analysis, for example in ImageNet-style label taxonomies [3, 11]. We use this score only for diagnostic analysis of failed trajectories. It separate predictions that are unrelated to the gold root cause from predictions that identify a related mechanism but choose the wrong normalized label. The primary benchmark score remains the strict exact-match metric; hierarchical RCA is reported as an auxiliary measure for interpreting near misses. Letsim(c, ˆc)denote the similarity between a gold root-cause labelcand a predicted labelˆc. We assign similarity1.0for an exact label match,0.8for the same fine-grained subtype,0.6for the same technical domain,0.4for the same super-category, and0otherwise. For multi-root-cause tasks, we defineG q as the set of gold RCA entries andP q as the set of predicted RCA entries, compute all pairwise similarities between predicted and gold RCA entries, perform maximum-weight bipartite matching, and normalize the matched similarity by the number of gold RCA entries: Hier-RCA(q) = max M∈M(P q ,G q ) P (p,g)∈M sim(c g ,c p ) ∥G q | .(12) 18 B.2 Topology-Aware Node Similarity We also define a topology-aware node similarity score as a supplement to exact node-object localiza- tion. Exact node matches receive full credit. Non-identical nodes receive partial credit when they belong to the same logical cluster, same site and role, or nearby topology neighborhood (see Ta- ble 14). This score evaluates node proximity only; it does not include object-level or root-cause-label similarity. For non-identical nodes, the final hierarchical node similarity combines the taxonomy and topology components: nodesim = 0.7· taxonomysim + 0.3· topologysim.(13) If either node is absent from the LLDP-derived topology, the score falls back to the taxonomy component inferred from normalized node names, site, role, and redundancy metadata. B.3 Root-Cause Type Hierarchy Table 15 lists the normalized root-cause labels defined by the RCA output schema and their hi- erarchical grouping. The L0 column contains the complete set of schema-level root-cause labels that a model may output under the task instructions, including labels that do not appear as gold answers in the current 126-task release. L1, L2, and L3 provide progressively coarser groupings for near-miss analysis. Legacy gold-answer aliases are normalized to these schema-level L0 labels before hierarchical matching. C Supplementary Evaluation Results This section provides detailed result tables omitted from the main paper. C.1 Full RCA Category Results Table 16 reports localization, identification, and evidence scores for each model across the five merged RCA categories. These results complement the RCA category radar plot in the main paper. C.2 Hierarchical RCA Results Table 17 compares strict exact scores with supplementary hierarchical scores. Across the five evaluated agents, hierarchical RCA identification exceeds exact RCA identification by 2.65–13.14 percentage points, and hierarchical node localization exceeds exact localization by 16.28–31.24 points. These gaps measure the additional credit captured by the relaxed hierarchy when a prediction preserves a coarser root-cause category or topology neighborhood even though the strict normalized label or faulty object is incorrect. The gap between exact localization and hierarchical node localization suggests that agents often reach the correct topology neighborhood but fail to identify the exact faulty node or object required for strict operational correctness. C.3 Observability Results for All Agents Table 18 reports RCA performance by observability level for all evaluated agents. The O2 subset contains tasks in which decisive evidence is unavailable or must be inferred from indirect observations. The O2→O1 rows correspond to an ablation setting where the originally unobservable elements are made directly observable. Since this ablation changes the golden evidence specification, we use it primarily to analyze final accuracy, localization, and identification rather than to draw conclusions from the evidence score. C.4 Heterogeneity Results for All Agents Table 19 reports performance by network heterogeneity level. Heterogeneity captures the need to reason across different device roles, vendors, command syntaxes, and output formats. The stratified results expose differences across heterogeneous settings that are hidden in aggregate performance. 19 L3 super-categoryL2 technical domainL1 root-cause subtypeL0 normalized root-cause label Physical/Link State Interface stateAdministrative shutdown shutdown Physical/Link State Interface performanceBandwidth congestiontraffic congestion occupying port bandwidth Physical/Link State Interface MTUMTU mismatchMTU value misconfiguration Security / Boundary Control Security policyMissing permit rulesecurity policy rule does not per- mit the corresponding user Security / Boundary Control NAT role bindingNAT external-interface at- tribute error NAT external interface attribute misconfiguration or missing con- figuration Security / Boundary Control NAT role bindingNAT internal-interface at- tribute error NAT internal interface attribute misconfiguration or missing con- figuration Routing / Policy Control Static routeBlackhole routeblackhole route Routing / Policy Control Static routeMissing static routemissing static route Routing / Policy Control Static routeIncorrect static routeincorrect static route Routing / Policy Control ARP / adjacencyARP configurationARP configuration error Routing / Policy Control Forwarding loopLayer-3 loopLayer 3 loop Routing / Policy Control EGP control planeBGP configurationBGP configuration error Routing / Policy Control Route policy Prefix-list coverage miss- ing IP prefix list missing the corre- sponding user source IP address Routing / Policy Control IGP control planeOSPF configurationOSPF configuration error Routing / Policy Control IGP control planeISIS configurationISIS configuration error HA / Reliability Layer-2 loop preven- tion STP not enabledglobal STP not enabled HA / ReliabilityLayer-2 loop preven- tion Port STP not enabledport STP not enabled HA / ReliabilityHot-standbyredun- dancy VRRP/HRP redundancy not enabled global VRRP hot standby redun- dancy protocol not enabled Service / OverlayVPN service isolation L3VPN configurationL3VPN configuration error Service / OverlayVPN service isolation L2VPN configurationL2VPN configuration error Service / OverlayVXLAN overlayVXLAN configurationVXLAN configuration error Service / OverlaySRv6 policy tunnelSRv6-Policy planning SRV6-Policy tunnel planning er- ror Service / OverlayVPN service provi- sioning Missing VPN configura- tion VPN configuration missing Access Configura- tion Layer-2forwarding identity MAC address configura- tion MAC address configuration er- ror Access Configura- tion Interface addressingInterface IP configurationinterface IP error Access Configura- tion VLAN access bindingInterface VLAN configu- ration interface VLAN configuration error Access Configura- tion Address uniquenessLoopback IP conflictloopback IP configuration con- flict Operations Support Host information col- lection Missing host-information collection host information collection func- tion missing Table 15: Root-cause type hierarchy used for hierarchical RCA scoring over the complete RCA output-schema label space. C.5 Structural Difficulty Trends We use task metadata to examine how performance changes across structurally easier and harder subsets. Because evidence observability and network heterogeneity are analyzed separately, Table 20 20 AgentCategoryRCA-AccRCA-LocRCA-IDRCA-Evid. Codex+GPT-5.5C17.1455.7478.5710.76 Codex+GPT-5.5C265.3180.3380.6513.45 Codex+GPT-5.5C340.0023.3361.549.94 Codex+GPT-5.5C450.0051.4351.4318.50 Codex+GPT-5.5C528.1246.5157.509.43 ClaudeCode+Qwen3.7-PlusC13.5740.3060.619.77 ClaudeCode+Qwen3.7-PlusC228.5760.5667.6914.05 ClaudeCode+Qwen3.7-PlusC30.000.0010.004.49 ClaudeCode+Qwen3.7-PlusC414.7121.6225.0012.20 ClaudeCode+Qwen3.7-PlusC518.7536.9637.789.32 HermesAgent+DeepSeek-V4-ProC17.1439.3957.589.06 HermesAgent+DeepSeek-V4-ProC226.5355.5661.027.78 HermesAgent+DeepSeek-V4-ProC30.000.0031.946.48 HermesAgent+DeepSeek-V4-ProC411.7613.8913.8911.23 HermesAgent+DeepSeek-V4-ProC515.6232.5035.907.04 HermesAgent+Qwen3.7-MaxC110.7157.8165.528.17 HermesAgent+Qwen3.7-MaxC234.6972.7380.3312.64 HermesAgent+Qwen3.7-MaxC30.000.0044.448.57 HermesAgent+Qwen3.7-MaxC423.5321.0535.2912.70 HermesAgent+Qwen3.7-MaxC521.8826.8325.005.69 HermesAgent+TelecomGPT-R1C10.0012.9026.673.85 HermesAgent+TelecomGPT-R1C212.2435.0034.436.43 HermesAgent+TelecomGPT-R1C30.000.000.002.03 HermesAgent+TelecomGPT-R1C40.000.000.004.55 HermesAgent+TelecomGPT-R1C50.000.000.003.50 Table 16: RCA results by fault category. Scores are percentages. C5 merges low-frequency service, overlay, configuration, management, and other operational categories. AgentExact RCA-IDHier. RCA-IDExact Loc.Hier. Node Loc. Codex+GPT-5.566.5072.5252.6973.80 ClaudeCode+Qwen3.7-Plus39.5950.1935.0358.70 HermesAgent+DeepSeek-V4-Pro39.7550.1929.1458.67 HermesAgent+Qwen3.7-Max52.3662.9938.0364.87 HermesAgent+TelecomGPT-R111.2813.9312.3328.61 Table 17: Exact and hierarchical RCA scores. Scores are percentages. Hierarchical scores are recall-oriented over gold RCA entries and are supplementary diagnostics rather than replacements for exact IoU metrics. focuses on root-cause count, restored path count, fault-propagation-chain length, golden-solution length, and protocol complexity. Table 20 reports the average easiest-to-hardest Accuracy drop for each component. For RCA tasks, fault-propagation-chain length shows the largest drop: average Accuracy falls from 64.21% on FPC- Low cases to 7.00% on FPC-High cases. Root-cause count is also a strong factor, with RC1-to-RC3+ Accuracy decreasing by 31.63 points. Golden-solution length and protocol complexity show smaller but consistent drops across all five agents. For path-restoration tasks, the drops are more balanced. Multiple restored paths, longer expert paths, and higher protocol complexity each reduce average Accuracy by roughly 20–23 points from the easy to hard tier. These metadata fields therefore separate easier and harder subsets for both task families, but should not be read as isolated causal variables, since each stratum still mixes different fault families, topologies, and evidence conditions. 21 AgentObs.NAcc.RCA-LocRCA-IDRCA-Evid. Codex+GPT-5.5O110749.5354.1172.4116.45 Codex+GPT-5.5O21936.8442.8635.4812.11 Codex+GPT-5.5O2→O11973.6881.8275.0014.08 ClaudeCode+Qwen3.7-PlusO110720.5637.0242.9212.85 ClaudeCode+Qwen3.7-PlusO21915.7917.1416.2211.55 ClaudeCode+Qwen3.7-PlusO2→O11931.5837.5025.7111.66 HermesAgent+DeepSeek-V4-ProO110717.7629.5545.059.99 HermesAgent+DeepSeek-V4-ProO21915.7928.5715.0012.37 HermesAgent+DeepSeek-V4-ProO2→O11947.3744.8335.4810.15 HermesAgent+Qwen3.7-MaxO110723.3637.8553.0412.11 HermesAgent+Qwen3.7-MaxO21936.8435.4848.2815.84 HermesAgent+Qwen3.7-MaxO2→O11952.6359.0958.339.63 HermesAgent+TelecomGPT-R1O11075.6114.0013.435.80 HermesAgent+TelecomGPT-R1O2190.002.500.001.36 HermesAgent+TelecomGPT-R1O2→O1190.000.000.002.50 Table 18: RCA results by observability level and observability ablation. Scores are percentages. O2→O1 is evaluated on the same 19 originally O2 tasks after removing the unobservable elements. AgentTaskHNAcc.Loc.ID/Rest.Evid. Codex+GPT-5.5RCAH-Low (= 1)3966.6752.7357.6912.47 Codex+GPT-5.5RCAH-High (> 1)8739.0852.9469.9317.20 Codex+GPT-5.5PathH-Low (≤ 6)5889.66100.0095.5238.89 Codex+GPT-5.5PathH-High (> 6)5086.0098.0295.1153.29 ClaudeCode+Qwen3.7-PlusRCAH-Low (= 1)3933.3330.3834.2511.76 ClaudeCode+Qwen3.7-PlusRCAH-High (> 1)8713.7936.2440.9112.96 ClaudeCode+Qwen3.7-PlusPathH-Low (≤ 6)5829.3194.9668.5825.72 ClaudeCode+Qwen3.7-PlusPathH-High (> 6)504.0072.7335.0527.14 HermesAgent+DeepSeek-V4-ProRCAH-Low (= 1)3928.2128.5726.098.12 HermesAgent+DeepSeek-V4-ProRCAH-High (> 1)8712.6429.6945.6611.05 HermesAgent+DeepSeek-V4-ProPathH-Low (≤ 6)5844.8391.3873.4620.32 HermesAgent+DeepSeek-V4-ProPathH-High (> 6)506.0072.4150.7722.91 HermesAgent+Qwen3.7-MaxRCAH-Low (= 1)3946.1538.7142.869.57 HermesAgent+Qwen3.7-MaxRCAH-High (> 1)8716.0937.2755.8413.97 HermesAgent+Qwen3.7-MaxPathH-Low (≤ 6)5848.2891.3872.3819.72 HermesAgent+Qwen3.7-MaxPathH-High (> 6)5032.0067.9250.2826.87 HermesAgent+TelecomGPT-R1RCAH-Low (= 1)3910.268.575.972.98 HermesAgent+TelecomGPT-R1RCAH-High (> 1)872.3013.6413.306.04 HermesAgent+TelecomGPT-R1PathH-Low (≤ 6)583.4583.1917.978.96 HermesAgent+TelecomGPT-R1PathH-High (> 6)500.0058.685.244.18 Table 19: Selected results by network heterogeneity. Scores are percentages. C.6 Evidence-Grounded Performance Analysis In this subsection, evidence coverage is defined as the fraction of expert golden evidence steps observed in the trajectory,| ˆ A∩A ∗ |/|A ∗ |. This recall-style quantity measures how much of the expert diagnostic chain the agent reaches, without penalizing additional commands. It is therefore different from the main evidence score used in the capability tables, which is an Evidence F1 score that accounts for both covered golden steps and extra executed device-command observations. Table 21 reports evidence coverage on exactly solved tasks. Even when agents produce the correct final answer, their evidence coverage remains limited, suggesting that current agents often solve tasks without reconstructing the complete expert diagnostic chain. The table is conditioned on solved tasks, whereas Figure 5 analyzes all model-question samples. 22 ComponentTask Easy Acc. Hard Acc. Drop Agents Fault Propagation Length RCA64.217.0057.215/5 Root-cause CountRCA37.886.2531.635/5 Golden Solution LengthRCA27.6716.6011.075/5 Protocol ComplexityRCA26.8417.209.645/5 Restored Path CountPath41.2520.0021.255/5 Golden Solution LengthPath42.1619.4122.755/5 Protocol ComplexityPath46.2526.0020.255/5 Table 20: Accuracy impact of CTBench components excluding evidence observability and network heterogeneity. Easy/Hard Accuracy is averaged over agents using the easiest and hardest tier of each component; Drop is the mean per-agent easy-to-hard Accuracy decrease in percentage points. AgentRCAPath Restoration Codex+GPT-5.558.05%53.45% ClaudeCode+Qwen3.7-Plus51.45%38.56% HermesAgent+DeepSeek-V4-Pro38.02%19.61% HermesAgent+Qwen3.7-Max40.22%27.75% HermesAgent+TelecomGPT-R116.67%39.13% Table 21: Evidence coverage among exactly solved tasks. 020406080100 Mean evidence score (%) 0 20 40 60 80 Exact accuracy (%) RCA Path Restoration RCA: R 2 =0.433, n=126 pairs/bin Path Restoration: R 2 =0.978, n=108 pairs/bin Figure 5: Evidence–accuracy relationship using equal-frequency quintiles sorted by evidence score. Each point aggregates the same number of model-question samples within a task family; the reported R 2 values are computed over the quintile points. The lines connect quintile aggregates and are not fitted regression lines. Figure 5 further relates evidence coverage to exact-answer Accuracy after collecting all trajectories. Path Restoration shows a strong monotonic trend: higher evidence coverage corresponds to substan- tially higher exact Accuracy. RCA exhibits a weaker relationship; the lower RCAR 2 indicates that evidence coverage explains only part of the variance in exact correctness. This weaker relationship is consistent with the RCA error analysis: even when an agent reaches high evidence coverage, it may still fail to identify the correct fault node, affected object, root-cause label, or complete multi-fault answer. Thus, evidence acquisition is necessary for reliable troubleshooting, but RCA also requires stronger causal reasoning and answer normalization. C.7 Harness Sensitivity Analysis To separate model choice from execution-harness effects, we compare raw trajectory sets that use the same DeepSeek-V4-Pro model but different agent harnesses:ClaudeCode,HermesAgent, and Codex. This analysis treats each harness-model pair as an executable agent because the harness controls tool scheduling, state serialization, stopping behavior, and final-answer extraction. All runs 23 use the same task set, gold answers, and scoring scripts. The comparison is therefore intended as a raw harness sensitivity analysis under a fixed model family, rather than as an ablation that isolates each individual harness mechanism. Missing or empty final answers are kept in the denominator and counted as incorrect. TaskHarness + ModelAcc.Loc. ID/Rest. Evid. RuntimeTokens RCAClaudeCode+DeepSeek-V4-Pro21.43 26.1934.85 37.71410.52s 3809.7k RCAHermesAgent+DeepSeek-V4-Pro 17.46 29.4340.08 33.38481.13s 1750.8k RCACodex+DeepSeek-V4-Pro15.08 21.4534.26 30.88482.79s 1083.7k Path Restoration ClaudeCode+DeepSeek-V4-Pro14.81 77.5958.51 26.35416.66s 5440.1k Path Restoration HermesAgent+DeepSeek-V4-Pro 26.85 81.9059.26 18.84492.97s 2155.5k Path Restoration Codex+DeepSeek-V4-Pro26.85 77.5959.25 24.88545.00s 1660.8k Table 22: Harness sensitivity for the same DeepSeek-V4-Pro model. Scores are percentages except runtime and tokens. ID/Rest. denotes RCA identification IoU for RCA and restoration-edge IoU for Path Restoration. Table 22 shows that CTBench measures a harness-model combination rather than a model in isolation. On RCA, ClaudeCode+DeepSeek-V4-Pro obtains the highest exact Accuracy and evidence coverage among the three harnesses, while HermesAgent+DeepSeek-V4-Pro obtains the highest localization and identification IoU. Codex+DeepSeek-V4-Pro uses the fewest tokens and rounds, but its RCA exact Accuracy and evidence coverage are lower. On Path Restoration, HermesAgent and Codex reach the same exact Accuracy, whereas HermesAgent has the best localization IoU and Codex achieves nearly the same restoration-edge IoU with substantially fewer tokens and rounds than HermesAgent. ClaudeCode collects the most evidence for Path Restoration but uses far more tokens and obtains lower exact Accuracy. Runtime, rounds, and tokens should be interpreted as complementary efficiency measures: fewer interaction rounds or fewer tokens do not necessarily imply lower wall-clock latency because model-side response time and harness orchestration overhead can differ. The disagreement counts in Table 23 show that different harnesses solve partly different subsets of tasks even when the underlying model is fixed. For example, on RCA, ClaudeCode solves 11 tasks that Codex misses, while Codex solves 3 tasks that ClaudeCode misses. On Path Restoration, Codex solves 19 tasks missed by ClaudeCode, while HermesAgent and Codex each solve 18 tasks missed by the other. Overall, these results show that the harness matters even when the underlying model is fixed. Different harnesses solve different subsets of tasks, so we report performance at the harness–model level rather than treating the model as the only source of variation. D Reproducibility Details D.1 Evaluation Environment All CTBench agent trajectories are executed on a single orchestration host running EulerOS 2.0. The host is equipped with an Intel Xeon Gold 6230N CPU at 2.30GHz, 40 physical cores, 80 logical processors, and 502GB of physical memory. The host runs the agent harnesses, local sandbox workspaces, simulator/tool servers, logging, and result collection. Model inference is not served locally; all evaluated models are accessed through their original vendor APIs. We pre-collected command-line interface (CLI) logs from the target network devices and served them through a mock backend. For supported commands, the backend returns the recorded device outputs; for invalid queries, it returns the corresponding error messages. Commands that are syntactically valid but absent from the collected logs return a simulated “permission denied” response, so agents cannot query outside the predefined observation set. We evaluate five harness–LLM configurations with fixed harness versions: Codex (v0.141.0) with GPT-5.5, ClaudeCode (v2.1.63) with Qwen3.7-Plus, and Hermes Agent (v0.16.0) with DeepSeek- V4-Pro, Qwen3.7-Max, or TelecomGPT-R1. Each benchmark task runs in a fresh Docker container. The workspace, tool configuration, logs, and temporary files are recreated for each task to avoid state carryover between runs. 24 TaskPairBothFirst onlySecond onlyNeither RCACC vs. HA1512792 RCACC vs. CX1611396 RCAHA vs. CX1210797 Path Rest.C vs. HA6102369 Path Rest.C vs. CX1061973 Path Rest.HA vs. CX11181861 Table 23: Pairwise exact-answer disagreement under the same DeepSeek-V4-Pro model. C, HA, and CX denote ClaudeCode, HermesAgent, and Codex, respectively. “First only” and “Second only” refer to the order in the Pair column. D.2 Agent Harnesses and Concurrency We evaluate three harness implementations:Codex,ClaudeCode, andHermesAgent. The reported agent-model settings are instantiated from these harnesses: • Codex+GPT-5.5, • ClaudeCode+Qwen3.7-Plus, • HermesAgent+DeepSeek-V4-Pro, • HermesAgent+Qwen3.7-Max, and • HermesAgent+TelecomGPT-R1. During evaluation, the task queue is capped at three concurrent agent runs to control API request pressure and avoid local simulator contention and the maximum running time for each initiated task is 3600 seconds. D.3 Command Form Each harness run follows the same orchestration pattern: select the harness and model, provide the CTBench task JSON, write trajectories and parsed results to a run-specific output directory, and set the maximum concurrency. python <runner>.py --harness codex --model gpt-5.5 --input <ctbench_task_json> --output <result_json> --workspace-base <sandbox_root> --concurrency 3 --timeout 3600 python <runner>.py --harness claudecode --model qwen3.7-plus --input <ctbench_task_json> --output <result_json> --workspace-base <sandbox_root> --concurrency 3 --timeout 3600 python <runner>.py --harness hermesagent --model <model_name> --input <ctbench_task_json> --output <result_json> --workspace-base <sandbox_root> --concurrency 3 --timeout 3600 For the Codex harness, the per-task child process created by the runner uses the following command shape, with the task prompt supplied on standard input: codex.cmd -a never exec --cd <workspace_dir> --skip-git-repo-check --sandbox workspace-write --color never --output-last-message last_msg.txt --model gpt-5.5 --ephemeral D.4 Prompt Template All agent-model combinations use the same task-level prompt template shown in Figure 6. The prompt contains the task description, question identifier, allowed tool interface, and required answer format. It does not include ground-truth answers, task metadata, root-cause categories, difficulty labels, or golden evidence steps. 25 Agent Prompt Template You are an AI agent evaluating comprehensive capabilities for CTBench. Your task is to solve the given CTBench evaluation problem using only the information and tools allowed by the task. Follow the task-specific instructions exactly. The task description defines the valid fault categories, answer fields, separators, line order constraints if any, device names, destination formats, and whitespace requirements. These task-specific requirements override any general wording in this prompt. Work method: 1. Identify the affected endpoints, objects, and evaluation scope from the task. 2. Query only relevant devices and commands needed to confirm or reject plausible root causes. 3. Prefer direct evidence and cross-checking evidence from the task-allowed command outputs. 4.Interpret potential faults in the context of the intended network design; do not report a root cause that contradicts the topology, configuration model, or design logic implied by the task. 5.Do not perform broad exhaustive sweeps when a focused path can answer the task. Avoid repeated equivalent commands after the evidence is sufficient. 6.If multiple independent root causes exist, return the minimal complete set of root causes. Do not include downstream symptoms or duplicate faults. 7. If the task restricts the fault type, output only faults of the allowed type. 8. Use exact device names, interface names, destination IPs, object names, and root-cause labels from the task and observed outputs. Evidence and safety rules: • Do not use hidden ground truth, answer files, prior trajectory files, scoring scripts, or local simulator output files. • Do not infer answers from filenames, dataset metadata, process state, or filesystem paths. • Query simulator device outputs only through the API or tool interface explicitly allowed by the task. • If an allowed command is unavailable, use the closest allowed command form from the task description; do not treat a missing command output as direct proof of a fault. Before finalizing, silently check: • Does every output line match the exact format requested by the task? • Are all required fields present and in the requested order? • Are separators exactly as requested, usually English semicolons? • Are there no extra spaces, comments, labels, markdown, or explanations? • Are all independent root causes included, and are non-root-cause symptoms excluded? Put only the final answer inside <result></result> tags. Task: question description Question ID:questionid Return format: <result>your final answer here</result> Figure 6: Agent Prompt Template. E Trajectory-Level RCA Failure Analysis E.1 Overview of Trajectory-Level Failure Analysis This appendix summarizes a trajectory-level post-hoc analysis of exact-match RCA failures for Codex+GPT-5.5andHermesAgent+Qwen3.7-Max. The goal is to characterize why an agent’s final answer fails after interacting with the diagnostic environment, rather than only reporting whether the final answer matches the gold RCA tuple. Among the 126 RCA tasks,Codex+GPT-5.5has 66 exact-match failures andHermesAgent+Qwen3.7-Maxhas 94 exact-match failures; 60 tasks are failed by both systems. The main failure pattern is not failed tool use. In many cases, agents issue relevant commands and observe useful intermediate evidence, but still fail when mapping that evidence to the final RCA tuple set. The observed errors include missing independent causes, adding non-minimal causes, 26 LevelLeaf failure labelUnitDefinition Output-levelF5.Answer Realization Failure Whole answerThe trajectory does not produce a valid final RCA answer, or the answer cannot be parsed into the required tuple schema. Set-level F1. Causal Coverage DeficitAnswer setThe answer omits one or more indepen- dent gold root causes, including dual-root, multi-device, or multi-interface faults. Set-levelF2. Causal Minimality Vio- lation Answer setThe answer includes redundant, symptom- level, or non-independent causes beyond the minimal gold root-cause set. Entry-levelF4. Mechanism Discrimina- tion Error Root-cause entryThe returned entry uses the wrong causal mechanism, such as confusing redundancy, routing, policy, address, or forwarding- layer faults. Entry-levelF3. Fault-Object Grounding Error Root-cause entryThe mechanism is largely correct, but the causal object is grounded to the wrong de- vice, interface, IP address, policy object, underlay/overlay layer, or active/standby role. Table 24: Hierarchical RCA failure taxonomy used for trajectory-level error analysis. Rows follow the annotation priority order. choosing the wrong fault mechanism, grounding the fault to the wrong node or object, and violating the required output schema. We identify this pattern through manual inspection of the diagnostic traces, checking each failed task against the task statement, final answer, issued commands, and observed command outputs. E.2 Failure Mode Taxonomy We organize trajectory failures using the hierarchical RCA failure taxonomy in Table 24. The taxonomy contains one output-level failure, two answer-set-level failures, and two entry-level failures. This taxonomy is used only as a post-hoc explanation of exact-match errors; it is not a formal component of the benchmark, is not part of the primary benchmark score, and does not relax the exact-answer criterion. This structure is important for reproducible annotation: when a prediction both misses an independent root cause and mislocalizes a returned tuple, the set-level coverage error is assigned first; entry-level mechanism or grounding errors are used only when the answer set is otherwise comparable to the gold set. For each task, three expert annotators with telecom RCA expertise independently assigned failure labels, and the taxonomy was applied with a fixed priority order. Disagreements were resolved through adjudication and discussion under the same priority rules. The annotators first checked whether the final answer was parseable under the required RCA schema. If not, the failure was labeled as answer realization. For parseable answers, they next compared the predicted and gold root-cause sets to identify missing independent causes or non-minimal extra causes. Only after the answer set was comparable did they label entry-level failures: mechanism discrimination errors when the causal mechanism was wrong, and fault-object grounding errors when the mechanism was largely correct but the device, interface, address, policy object, or topology role was incorrect. Within entry-level annotation, mechanism mismatch has priority over object mismatch: a prediction with both a wrong mechanism and a wrong object is labeled F4 rather than F3. Table 25 reports the resulting distribution. Coverage deficits and mechanism discrimination dominate Codex+GPT-5.5’s RCA errors, whileHermesAgent+Qwen3.7-Maxshows a more distributed error profile with substantially more minimality and answer-realization failures. E.3 Representative Failure Trajectories The full trajectories for these examples are provided in the Code and Data Supplement under modeltrajectory, organized by model name, task type, and task identifier. All representative 27 ModelF1 F2 F3 F4 F5 Total GPT-5.5281 13 23166 Qwen3.7-Max 32 16 18 16 1294 Table 25: Failure-mode counts for exact-match RCA failures. F1–F5 refer to the labels in Table 24; each failed trajectory is assigned one primary failure label. examples below were checked against the recorded final answers and diagnostic traces of the corresponding agents. F5Answerrealizationfailure:RCAq12. Thegoldansweris FW02;114.114.114.114;global VRRP hot standby redundancy protocol not enabled .HermesAgent+Qwen3.7-Maxreturns an empty final answer despite successfully interacting with the environment. The trajectory lacks valid result tags, separators, or parseable lines, leading to a complete output failure independent of the agent’s actual domain knowledge. F1 Causal coverage deficit: RCA q7. The gold answer contains two independent root causes: FW01;10.3.10.1;security policy rule does not permit the corresponding user andBJHQCSR1000VGW01;10.1.120.251;IP prefix list missing the corresponding user source IP address.Codex+GPT-5.5outputs only the firewall security-policy root cause and omits the prefix-list fault, prematurely halting its diagnosis once the first locally plausible cause is identified. F2 Causal minimality violation: RCA q18. The gold answer contains a VRRP redundancy configuration fault onFW02, whileHermesAgent+Qwen3.7-Maxreturns both a prefix-list fault and a VRRP fault on a different firewall. Although this answer also contains localization errors, the set-level F2 label takes precedence because the final submitted set retains unpruned, non-gold causal assertions alongside downstream symptoms. F4Mechanismdiscriminationerror:RCAq10. Thegoldrootcauseis FW02;114.114.114.114;global VRRP hot standby redundancy protocol not enabled.Codex+GPT-5.5localizes the answer to the correct firewall and address but in- correctly reports an OSPF configuration error, successfully grounding the location while confusing redundancy-state semantics with routing-protocol configurations. F3 Fault-object grounding error: RCA q11. The gold root cause isFW02;8.8.8.8;global VRRP hot standby redundancy protocol not enabled.HermesAgent+Qwen3.7-Max outputsCoreSW01;8.8.8.8;global VRRP hot standby redundancy protocol not enabled. The mechanism and address are aligned with the gold answer, but the causal device is misattributed to a neighboring core switch rather than the faulty firewall itself. The agent fails to decouple its observation point from the actual causal node. E.4 Implications for Agent Design The failure taxonomy suggests several checks that are useful for troubleshooting agents. During a run, the agent should track which symptoms have been explained, which candidate causes remain possible, which observations support or reject each candidate, and whether the final answer satisfies the required schema. The observed failure modes suggest several useful checks for agent design: •Causal Coverage (F1) & Minimality (F2): A global diagnostic state tracker may help ensure all symptoms are explained without including redundant downstream effects. •Semantic Grounding (F3 & F4): Domain-specific verification may reduce mechanism confusion and mislocalization by checking protocol constraints (e.g., routing vs. redundancy states) and maintaining role-aware topology mapping to separate observation nodes from causal origins. 28 Failure taxonomyGeneral agent capabilityCTBench property F1 Causal coverage deficitCausal attribution and long-horizon evidence planning rootcausecount; goldensolutionpath F2 Causal minimality violationCausal attribution and symptom prun- ing fault propagationchain; goldensolutionpath F3 Fault-object grounding errorInteraction with heterogeneous envi- ronments networkheterogeneity F4 Mechanism discrimination error Partial-observation reasoning and causal attribution protocolcomplexity; evidenceobservability F5 Answer realization failure Reliable structured answer realizationOutput schema constraints Table 26: Qualitative mapping from CTBench RCA failure modes to broader agent capabilities and related task properties. •Robust Finalization (F5): Deterministic output validation can enforce schema constraints before final submission. Overall, this fine-grained analysis suggests that, in these RCA failures, current agents more often struggle with multi-step diagnostic reasoning and causal minimization than with basic tool execution. F Discussion of Agent Capabilities in CTBench Although CTBench is grounded in telecom network operations, the trajectory-level RCA failures reflect broader agent capabilities required in complex professional environments. The goal of this discussion is to clarify how the failure modes summarized in Table 24 relate to general agent capabilities and how these pressures arise from CTBench task characteristics. In many failed trajectories, agents can collect relevant observations but still fail to maintain causal coverage, prune downstream symptoms, ground evidence to the correct object, distinguish similar mechanisms, or realize the final answer in the required structured format. These failure modes are consistent with the task properties of CTBench RCA: heterogeneous device roles, partial observability, layered protocol mechanisms, causal propagation, multiple independent faults, and multi-step evidence requirements. Table 26 summarizes how the observed failure modes relate to broader agent capabilities and the CTBench properties that stress them. 29