Paper deep dive
Autonomous Incident Resolution at Hyperscale: An Agentic AI Architecture for Network Operations
Arun Malik
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/9/2026, 4:11:42 AM
Summary
This paper introduces an agentic AI architecture for autonomous incident resolution in hyperscale cloud network operations. The system utilizes a multi-agent orchestration framework comprising Intake, Planning, Execution, and Verification agents. It integrates a skills-based tool architecture inspired by the Model Context Protocol (MCP), structured knowledge encoding from operational runbooks, and a progressive autonomy framework. Deployed in production, the architecture achieves over 90% autonomous resolution rates for common incidents, significantly reducing mean time to resolution while maintaining safety through layered authorization, blast radius containment, and automated rollback mechanisms.
Entities (8)
Relation Signals (8)
Agentic AI Architecture → achieves → 90% Autonomous Resolution Rate
confidence 95% · demonstrating that agentic AI systems can achieve autonomous resolution rates exceeding 90% for common incident categories
Agentic AI Architecture → utilizes → Multi-Agent Orchestration Framework
confidence 95% · Our system employs a multi-agent orchestration framework where specialized AI agents collaborate to detect, diagnose, and remediate network incidents without human intervention.
Progressive Autonomy Framework → enables → Incremental Authority Increase
confidence 93% · A progressive autonomy framework with automatic promotion and demotion mechanisms that allows organizations to incrementally increase AI agent authority while maintaining safety invariants.
Multi-Agent Orchestration Framework → comprises → Planning Agent
confidence 92% · 2) Planning Agent: The Planning Agent receives enriched incident context and produces a structured remediation plan.
Multi-Agent Orchestration Framework → comprises → Execution Agent
confidence 92% · 3) Execution Agent: The Execution Agent translates the structured plan into concrete actions against the infrastructure.
Multi-Agent Orchestration Framework → comprises → Verification Agent
confidence 92% · 4) Verification Agent: The Verification Agent provides closed-loop assurance that the remediation was successful.
Multi-Agent Orchestration Framework → comprises →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cloud network infrastructure at hyperscale presents unique operational challenges where traditional human-driven incident response cannot keep pace with the volume, velocity, and complexity of failures. This paper presents an agentic AI architecture for autonomous incident resolution in large-scale network operations. Our system employs a multi-agent orchestration framework where specialized AI agents collaborate to detect, diagnose, and remediate network incidents without human intervention. We describe the architectural principles, including hierarchical agent decomposition, skills-based tool invocation via standardized protocols, structured knowledge encoding from operational runbooks, progressive autonomy with safety boundaries, and closed-loop verification. The architecture has been deployed in production at a major cloud provider, demonstrating that agentic AI systems can achieve autonomous resolution rates exceeding 90% for common incident categories while maintaining safety guarantees through layered authorization and rollback mechanisms. We discuss design tradeoffs, failure modes, and lessons learned from operating autonomous AI agents at scale.
Tags
Links
- Source: https://arxiv.org/abs/2606.09122v1
- Canonical: https://arxiv.org/abs/2606.09122v1
Trouble viewing inline? Open PDF directly →
Full Text
26,441 characters extracted from source content.
Expand or collapse full text
Autonomous Incident Resolution at Hyperscale: An Agentic AI Architecture for Network Operations Arun Malik Microsoft Azure Networking Email: arunma@microsoft.com ORCID: 0009-0005-6650-6711 Abstract—Cloud network infrastructure at hyperscale presents unique operational challenges where traditional human-driven incident response cannot keep pace with the volume, velocity, and complexity of failures. This paper presents an agentic AI architecture for autonomous incident resolution in large-scale network operations. Our system employs a multi-agent orches- tration framework where specialized AI agents collaborate to detect, diagnose, and remediate network incidents without human intervention. We describe the architectural principles, including hierarchical agent decomposition, skills-based tool invocation via standardized protocols, structured knowledge encoding from operational runbooks, progressive autonomy with safety bound- aries, and closed-loop verification. The architecture has been deployed in production at a major cloud provider, demonstrating that agentic AI systems can achieve autonomous resolution rates exceeding 90% for common incident categories while maintaining safety guarantees through layered authorization and rollback mechanisms. We discuss design tradeoffs, failure modes, and lessons learned from operating autonomous AI agents at scale. Index Terms—autonomous operations, agentic AI, network in- cident resolution, multi-agent systems, hyperscale infrastructure, AIOps, Model Context Protocol I. INTRODUCTION A. The Operational Challenge at Hyperscale Modern cloud providers operate network infrastructure spanning millions of devices across hundreds of data centers worldwide. This infrastructure generates a continuous stream of operational incidents, including hardware failures, software bugs, configuration drift, capacity exhaustion, and cascading failures triggered by complex interdependencies. Traditional operations models rely on human engineers (often called on-call engineers or Site Reliability Engineers) to investigate and resolve these incidents. However, this approach faces fundamental scalability limitations: • Volume: The number of incidents grows linearly (or super-linearly) with infrastructure scale, while the engi- neering workforce cannot scale proportionally. • Velocity: Network incidents can cascade within seconds, but human investigation and remediation cycles operate on timescales of minutes to hours. • Complexity: Modern network architectures involve in- tricate dependencies between layers (physical, data link, network, transport, application), making root cause anal- ysis a combinatorial challenge. • Knowledge distribution: Operational expertise is often concentrated in a small number of senior engineers, cre- ating knowledge bottlenecks and single points of failure. B. From Automation to Autonomy The industry has progressed through several generations of operational tooling: 1) Manual operations: Engineers SSH into devices, run diagnostic commands, and apply remediations by hand. 2) Scripted automation: Runbooks are codified into scripts that automate specific remediation steps, but still require human triggering and supervision. 3) Rule-based automation: Event-driven systems apply predetermined actions when specific conditions match, limited to known failure modes. 4) AI-assisted operations (AIOps): Machine learning models provide recommendations and anomaly detec- tion, but humans remain in the loop for decision-making. 5) Autonomous operations: AI agents independently per- ceive the environment, reason about failures, plan reme- diation strategies, execute actions, and verify outcomes. This paper presents an architecture for the fifth generation: progressively autonomous incident resolution using agentic AI systems with built-in safety guarantees and human oversight mechanisms. Rather than proposing a binary shift from manual to fully autonomous, the architecture enables organizations to incrementally increase agent authority while maintaining safety invariants at every level. C. Contributions This paper makes the following contributions: 1) An end-to-end architecture for progressively au- tonomous incident resolution using multi-agent orches- tration, including agent decomposition, coordination protocols, and safety mechanisms. 2) A skills-based tool architecture inspired by extensible agent runtimes (e.g., Model Context Protocol), enabling composable, governed, and independently deployable operational capabilities. 3) A structured knowledge encoding methodology for converting tribal operational knowledge into machine- executable playbooks verified against production behav- ior. arXiv:2606.09122v1 [cs.SE] 8 Jun 2026 4) A progressive autonomy framework with automatic pro- motion and demotion mechanisms that allows organiza- tions to incrementally increase AI agent authority while maintaining safety invariants. 5) Production deployment experience and lessons learned from operating autonomous AI agents at hyperscale network infrastructure. I. BACKGROUND AND RELATED WORK A. AIOps and Automated Operations The term AIOps (Artificial Intelligence for IT Operations) was coined to describe the application of machine learning techniques to operational data. Early AIOps systems focused on anomaly detection [1], root cause analysis [2], [3], and alert correlation [4]. These systems provide decision support but leave execution to human operators. B. Self-Healing Systems Self-healing architectures [5] implement closed-loop control where systems detect failures, diagnose root causes, and apply remediations automatically. However, prior work focuses pri- marily on application-level healing (restarting services, scaling resources) rather than infrastructure-level operations requiring physical device interaction. Furthermore, existing self-healing systems typically operate in a binary mode (fully manual or fully automatic) without graduated trust mechanisms. Our work addresses both limitations by targeting network infras- tructure and introducing progressive autonomy levels with safety-bounded authority. C. Multi-Agent Systems Multi-agent systems [6] provide a theoretical founda- tion for decomposing complex tasks into cooperating spe- cialized agents. Recent advances in large language models (LLMs) have enabled practical implementations of tool-using agents [7], [8] that can reason about complex problems and interact with external systems. D. Positioning of This Work Our architecture combines insights from AIOps (telemetry- driven diagnosis), self-healing systems (closed-loop remedi- ation), multi-agent coordination (task decomposition), and modern extensible agent runtimes (skills-based tool use via protocols such as MCP) into an integrated system specifically designed for hyperscale network operations. Unlike prior work that addresses individual aspects or proposes fully autonomous solutions without safety guarantees, we present a production- deployed system that addresses the full lifecycle from detec- tion through verified resolution with progressive autonomy and bounded authority at every stage. I. ARCHITECTURE A. System Overview The architecture is organized into four functional layers, each responsible for a distinct aspect of autonomous operations (see Figure 1): B. Agent Decomposition The orchestration layer decomposes incident resolution into four specialized agent roles: 1) Intake Agent: The Intake Agent is the system’s en- try point, responsible for receiving raw incident signals and preparing them for automated processing. It classifies the in- cident type, assesses priority and urgency, enriches the incident context with relevant topology and history, and determines whether the incident falls within the system’s autonomous resolution capability. 2) Planning Agent: The Planning Agent receives enriched incident context and produces a structured remediation plan. It identifies the most likely root cause based on symptoms and historical patterns, selects appropriate remediation strategies from the knowledge base, considers dependencies and ordering constraints, and generates a structured execution plan with explicit success criteria and abort conditions. 3) Execution Agent: The Execution Agent translates the structured plan into concrete actions against the infrastructure. It acquires necessary device locks and authorization tokens, ex- ecutes diagnostic commands to gather pre-action state, applies remediation actions sequentially while checking intermediate results, handles partial failures and adapts execution based on observed outcomes, and records all actions for auditability and rollback capability. 4) Verification Agent: The Verification Agent provides closed-loop assurance that the remediation was successful. It executes post-action health checks specific to the incident type, compares device state against expected outcomes defined in the plan, monitors for regression within a configurable bake- in period, triggers rollback if verification fails, and updates the incident record with resolution evidence. C. Coordination Protocol Agents communicate through a structured message-passing protocol with the following guarantees: • Ordered delivery: Messages between agents within a single incident resolution flow are delivered in order. • At-least-once semantics: No message is silently lost; agents must explicitly acknowledge receipt. • Timeout-based escalation: If any agent fails to respond within a configurable timeout, the orchestration layer escalates to human operators. • State checkpointing: The state of each agent’s reasoning is persisted at key decision points, enabling recovery after transient failures. D. Structured Knowledge Encoding A critical challenge in autonomous operations is bridging the gap between human operational expertise (often undoc- umented “tribal knowledge”) and machine-executable pro- cedures. We address this through a systematic knowledge encoding process: 1) Observation: The system observes human engineers re- solving incidents, capturing the sequence of commands, decisions, and verifications. Fig. 1. Four-Layer Architecture for Autonomous Incident Resolution. The architecture separates concerns into orchestration (agent coordination), knowledge (operational data and skill registry), safety (guardrails and constraints), and infrastructure (device and telemetry access). 2) Extraction: Patterns are identified across multiple res- olution instances for similar incident types. 3) Formalization: Extracted patterns are encoded into structured playbooks with explicit preconditions, steps, decision points, and verification criteria. 4) Verification: Formalized playbooks are validated against historical data to confirm they produce correct outcomes. 5) Refinement: Playbooks are continuously updated based on agent execution outcomes and human feedback. This process converts implicit knowledge into explicit, auditable, and machine-executable procedures. The resulting playbooks serve as the primary knowledge representation for the planning and execution agents. E. Skills-Based Tool Architecture A key design principle borrowed from modern agent run- time architectures is the separation of agent reasoning from tool execution through a skills-based abstraction. Inspired by extensible tool-use frameworks such as Model Context Protocol (MCP) and plugin-based agent runtimes, each oper- ational capability is encapsulated as a discrete, independently versioned skill with a well-defined interface. 1) Skill Composition: Each skill defines: • Interface contract: A typed schema describing inputs, outputs, and error conditions, enabling compile-time ver- ification of agent-skill compatibility. • Capability declaration: A machine-readable description of what the skill can do, used by the planning agent during tool selection. • Permission requirements: The minimum authorization scope required to invoke the skill, enforced by the safety layer. • Idempotency guarantees: Whether repeated invocation produces the same result, critical for retry and recovery logic. 2) Skill Registry and Discovery: The knowledge layer maintains a skill registry that enables dynamic capability dis- covery. When the planning agent encounters a novel situation, it queries available skills based on semantic matching between the incident context and skill capability declarations. This decouples the agent’s reasoning from the specific set of tools available, allowing new operational capabilities to be deployed without modifying agent logic. 3) Sandboxed Execution: Skills execute within isolated sandboxes that enforce: • Resource boundaries: Each skill invocation operates within allocated compute, memory, and time budgets. • Network isolation: Skills can only access infrastructure endpoints explicitly granted by the authorization engine. • Output validation: Skill outputs are validated against the declared schema before being consumed by downstream agents. • Audit logging: Every skill invocation, including inputs, outputs, and side effects, is recorded in an immutable audit trail. This skills-based architecture enables composability (agents can chain multiple skills to solve complex problems), exten- sibility (new capabilities are added by registering new skills without modifying the orchestration logic), and governance (each skill’s blast radius and permission scope can be inde- pendently controlled). IV. SAFETY FRAMEWORK A. Design Principles Operating AI agents autonomously on production infrastruc- ture requires robust safety guarantees. Our framework is built on four principles: 1) Least privilege: Agents are granted only the minimum permissions required for their current task. 2) Blast radius containment: No single agent action can affect more than a bounded number of devices or services. 3) Reversibility: All actions must be reversible, with au- tomated rollback mechanisms. 4) Progressive trust: Agent authority increases incremen- tally based on demonstrated reliability. B. Layered Authorization The authorization engine implements defense-in-depth through multiple layers: • Agent identity: Each agent instance has a unique identity with specific capability grants. • Action classification: Actions are classified by risk level (read-only, low-risk modification, high-risk modification, destructive). • Scope restriction: Agents can only operate on devices within their assigned scope (e.g., specific data centers or device types). • Rate limiting: Maximum action frequency prevents run- away automation. • Concurrent operation limits: Controls how many de- vices can be simultaneously affected. C. Blast Radius Calculation Before executing any remediation action, the system evalu- ates its potential blast radius: • Topology analysis: Determines which services and cus- tomers would be affected if the action fails or produces unexpected results. • Redundancy validation: Verifies that sufficient redun- dancy exists to absorb the temporary loss of the target device. • Concurrent impact assessment: Checks whether other ongoing operations could compound the impact. Actions exceeding configurable blast radius thresholds are automatically blocked and escalated to human operators. D. Rollback Mechanisms The system maintains rollback capability at multiple levels: • Configuration rollback: Device configurations are snap- shot before modification and can be restored atomically. • State rollback: For stateful operations (e.g., traffic engi- neering changes), the previous state is preserved and can be reapplied. • Automatic rollback triggers: If post-action verification fails or health metrics degrade beyond thresholds, roll- back is triggered automatically without human interven- tion. V. PROGRESSIVE AUTONOMY A. Trust Levels Organizations adopting autonomous operations rarely tran- sition directly from manual to fully autonomous. Our archi- tecture supports a spectrum of autonomy levels (Table I): TABLE I PROGRESSIVE AUTONOMY LEVELS LevelNameDescription 0AdvisoryAgent suggests actions; human executes 1SupervisedAgentexecuteswithhumanpre- approval 2MonitoredAgent executes autonomously; human reviews post-hoc 3AutonomousAgent executes without human involve- ment for approved categories 4Self-improvingAgent can refine its own operational procedures based on outcomes B. Promotion Criteria Agent authority is promoted based on quantifiable perfor- mance metrics: • Success rate: Percentage of incidents resolved correctly without human intervention. • Mean time to resolution: Average time from incident creation to verified resolution. • False positive rate: Percentage of actions taken that were unnecessary or incorrect. • Rollback frequency: How often automated rollbacks are triggered. • Human override rate: How often humans override agent decisions. Promotion from one level to the next requires sustained per- formance above thresholds across all metrics for a minimum evaluation period. Fig. 2.Operational Capability Comparison across three generations of operational technology: manual operations, rule-based automation, and the proposed agentic AI architecture. The radar plot shows relative performance across six dimensions. C. Demotion and Circuit Breakers The system includes automatic demotion mechanisms: • Per-category circuit breakers: If the failure rate for a specific incident category exceeds a threshold within a time window, the agent is automatically demoted to a lower trust level for that category. • Global circuit breakers: If the overall failure rate across all categories spikes, the entire system can fall back to advisory mode. • Human override tracking: Patterns of human overrides trigger automatic review and potential demotion. VI. EVALUATION A. Deployment Context The architecture has been deployed in a production cloud network serving millions of customers. The infrastructure comprises networking devices spanning multiple device types (routers, switches, load balancers, firewalls) across geograph- ically distributed data centers. B. Resolution Effectiveness After progressive deployment over multiple months, the system achieved the following outcomes: • Autonomous resolution rate: The system resolves the majority of incidents in supported categories without human intervention, with resolution rates exceeding 90% for well-understood failure modes. • Resolution time improvement: For incidents handled autonomously, resolution time decreased by two orders of magnitude compared to the human-driven baseline, from hours to minutes. • Accuracy: False positive remediation (taking action when no action was needed) occurs in less than 5% of cases, with no cases resulting in customer-visible impact due to the safety framework. C. Operational Efficiency • On-call burden reduction: The volume of incidents re- quiring human attention decreased significantly, allowing engineers to focus on novel failure modes and system improvements. Fig. 3. Incident Volume Distribution by Autonomy Level. As playbooks mature and gain trust through successful executions, incidents progressively shift from human-supervised to fully autonomous resolution. Fig. 4. Mean Time to Resolution (MTTR) Comparison. Autonomous resolu- tion achieves two orders of magnitude improvement over traditional human- driven processes for well-understood incident categories. • Knowledge preservation: Structured playbooks capture institutional knowledge that previously existed only in the minds of senior engineers, reducing organizational knowledge loss due to team transitions. • Consistent quality: Autonomous resolution eliminates variability in incident handling quality that occurs with human operators due to fatigue, experience level, and time pressure. D. Safety Performance • Zero critical incidents: The safety framework prevented all potentially harmful actions from reaching production without appropriate verification. • Rollback effectiveness: Automatic rollbacks were trig- gered in a small percentage of execution attempts, all recovering within the defined time bounds. • Blast radius compliance: No autonomous action ex- ceeded its predicted blast radius boundaries. VII. LESSONS LEARNED A. LLM Reliability in Safety-Critical Contexts Large language models exhibit stochastic behavior that is fundamentally at odds with the determinism required for safety-critical operations. We address this through: • Structured output enforcement: Agent outputs are con- strained to predefined schemas, reducing the surface area for LLM hallucination. • Multi-model consensus: Critical decisions require agree- ment between multiple independent model invocations. Fig. 5. Incident Resolution Outcomes. The distribution shows the proportion of incidents resolved fully autonomously, resolved with human oversight, escalated to human operators, and safely rolled back by the system. Fig. 6. Safety Framework Effectiveness across key safety dimensions. The radar chart compares the proposed layered safety approach against a baseline single-layer authorization model. • Deterministic verification: All LLM-generated plans are verified against deterministic safety checks before execution. B. The Importance of Observability • Decision tracing: Every decision made by every agent is logged with full context, enabling post-hoc analysis. • Causal attribution: The system maintains causal links between observations, decisions, and outcomes. • Human-readable explanations: Each autonomous res- olution includes a natural language explanation suitable for engineering review. C. Handling Novel Failures • Confidence estimation: Agents estimate their confidence in diagnosis and remediation plans, escalating when con- fidence is below thresholds. • Novelty detection: Statistical methods identify incidents that deviate significantly from known patterns, triggering human involvement. • Graceful degradation: When autonomous resolution is not possible, the system still provides diagnostic enrich- ment and suggested actions to assist human operators. D. Organizational Change Management • Trust building: Engineers must develop confidence in the system through transparency and demonstrated relia- bility. • Role evolution: The operational engineer role shifts from reactive incident response to system improvement and novel problem solving. • Accountability frameworks: Clear ownership and ac- countability structures must be defined for autonomous agent decisions. VIII. DISCUSSION A. Generalizability While our deployment focuses on network operations, the architectural patterns are applicable to other operational do- mains including compute infrastructure, storage systems, and application services. The key abstractions (agent decomposi- tion, progressive autonomy, safety framework) are domain- independent. B. Limitations • Coverage: The system currently handles a subset of all possible incident types. Long-tail and novel failures still require human expertise. • Multi-domain reasoning: Incidents spanning multiple operational domains (e.g., network + compute) require coordination across independent autonomous systems, which remains an area of active development. • Regulatoryconsiderations: Autonomous decision- making in critical infrastructure raises questions about accountability and auditability that are not fully addressed by technical mechanisms alone. C. Future Directions • Cross-domain orchestration: Extending the multi-agent framework to coordinate across infrastructure domains. • Formal verification: Applying formal methods to prove safety properties of agent behavior within bounded con- ditions. • Federated learning: Enabling knowledge sharing across organizational boundaries without exposing proprietary operational details. IX. CONCLUSION This paper presented an agentic AI architecture for au- tonomous incident resolution in hyperscale network opera- tions. The system demonstrates that multi-agent orchestration, combined with robust safety frameworks and progressive au- tonomy mechanisms, can achieve high autonomous resolution rates while maintaining the safety guarantees required for critical infrastructure. Our deployment experience shows that the transition from human-driven to autonomous operations is not merely a technology problem but requires careful attention to knowledge management, organizational change, and trust building. We believe this architecture represents a significant step toward truly autonomous infrastructure operations and hope it provides a useful reference for others pursuing similar goals. REFERENCES [1] H. Xu, W. Chen, N. Zhao, Z. Li, J. Bu, Z. Li, Y. Liu, Y. Zhao, D. Pei, Y. Feng, J. Chen, Z. Wang, and H. Qiao, “Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications,” in Proceedings of the 2018 World Wide Web Conference (W), 2018. [2] M. Chen, A. X. Zheng, J. Lloyd, M. I. Jordan, and E. Brewer, “Cause- Infer: Automatic and distributed performance diagnosis with hierarchical causality graph in large distributed systems,” in Proceedings of IEEE INFOCOM, 2014. [3] P. Wang, J. Xu, M. Ma, W. Lin, D. Pan, Y. Wang, and P. Chen, “CloudRanger: Root cause identification for cloud native systems,” in Proceedings of the 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2018. [4] L. Li, X. Zhang, X. Zhao, H. Zhang, Y. Kang, P. Zhao, B. Qiao, S. He, P. Lee, J. Sun, F. Gao, L. Yang, Q. Lin, S. Rajmohan, Z. Xu, and D. Zhang, “Fighting the fog of war: Automated incident detection for cloud systems,” in 2021 USENIX Annual Technical Conference (USENIX ATC’21), 2021. [5] G. Kang, J. Liu, B. Cao, and Y. Luo, “Self-healing microservice archi- tecture using multi-agent systems,” in Proceedings of IEEE International Conference on Services Computing (SCC), 2020. [6] M. Wooldridge, An Introduction to MultiAgent Systems, 2nd ed.John Wiley & Sons, 2009. [7] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in Pro- ceedings of the International Conference on Learning Representations (ICLR), 2023, arXiv:2210.03629. [8] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2302.04761.