Paper deep dive
Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement
Fengrui Liu, Xiao He, Tieying Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 12:26:11 PM
Summary
The paper introduces Vigil, a deployed proactive agent system for on-call support on ByteDance's Volcano Engine platform. Unlike reactive agents that disengage upon escalation, Vigil operates alongside human analysts, proactively identifying and answering customer questions within its scope during ongoing dialogues. It features a continuous self-improvement mechanism that extracts knowledge from resolved cases to update its capabilities, reducing manual maintenance of knowledge bases. The system has been deployed for over ten months, demonstrating improved operational efficiency and support quality.
Entities (8)
Relation Signals (6)
VIGIL → deployedon → Volcano Engine
confidence 95% · Vigil has been deployed on Volcano Engine, ByteDance's cloud platform, for over ten months
VIGIL → developedby → ByteDance
confidence 95% · Vigil has been deployed on Volcano Engine, ByteDance's cloud platform
VIGIL → implements → Continuous Self-Improvement
confidence 92% · Vigil incorporates a continuous self-improvement mechanism that extracts knowledge from human-resolved cases
VIGIL → istypeof → Proactive Agent
confidence 90% · we introduce Vigil, a novel proactive agent system
VIGIL → comparesto → Reactive Agent
confidence 85% · Unlike reactive agents, Vigil focuses on providing assistance during the phase in which human support is already involved.
Volcano Engine → provides → On-Call Support
confidence 80% · Operational management of Volcano Engine... presents a significant challenge due to the high volume of on-call support requests
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In large-scale cloud service platforms, thousands of customer tickets are generated daily and are typically handled through on-call dialogues. This high volume of on-call interactions imposes a substantial workload on human support analysts. Recent studies have explored reactive agents that leverage large language models as a first line of support to interact with customers directly and resolve issues. However, when issues remain unresolved and are escalated to human support, these agents are typically disengaged. As a result, they cannot assist with follow-up inquiries, track resolution progress, or learn from the cases they fail to address. In this paper, we introduce Vigil, a novel proactive agent system designed to operate throughout the entire on-call life-cycle. Unlike reactive agents, Vigil focuses on providing assistance during the phase in which human support is already involved. It integrates into the dialogue between the customer and the analyst, proactively offering assistance without explicit user invocation. Moreover, Vigil incorporates a continuous self-improvement mechanism that extracts knowledge from human-resolved cases to autonomously update its capabilities. Vigil has been deployed on Volcano Engine, ByteDance's cloud platform, for over ten months, and comprehensive evaluations based on this deployment demonstrate its effectiveness and practicality. The open source version of this work is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.09579v1
- Canonical: https://arxiv.org/abs/2604.09579v1
Trouble viewing inline? Open PDF directly →
Full Text
73,558 characters extracted from source content.
Expand or collapse full text
Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement Fengrui Liu liufengrui.work@bytedance.com ByteDance Beijing, China Xiao He xiao.hx@bytedance.com ByteDance Hangzhou, China Tieying Zhang ∗ tieying.zhang@bytedance.com ByteDance San Jose, United States Abstract In large-scale cloud service platforms, thousands of customer tick- ets are generated daily and are typically handled through on-call dialogues. This high volume of on-call interactions imposes a sub- stantial workload on human support analysts. Recent studies have explored reactive agents that leverage large language models as a first line of support to interact with customers directly and resolve issues. However, when issues remain unresolved and are escalated to human support, these agents are typically disengaged. As a re- sult, they cannot assist with follow-up inquiries, track resolution progress, or learn from the cases they fail to address. In this paper, we introduce Vigil, a novel proactive agent system designed to operate throughout the entire on-call life-cycle. Unlike reactive agents, Vigil focuses on providing assistance during the phase in which human support is already involved. It integrates into the dialogue between the customer and the analyst, proactively offering assistance without explicit user invocation. Moreover, Vigil incorporates a continuous self-improvement mechanism that ex- tracts knowledge from human-resolved cases to autonomously update its capabilities. Vigil has been deployed on Volcano Engine, ByteDance’s cloud platform, for over ten months, and comprehen- sive evaluations based on this deployment demonstrate its effec- tiveness and practicality. The open source version of this work is publicly available at https://github.com/volcengine/veaiops. CCS Concepts • Computing methodologies→Multi-agent systems; Artifi- cial intelligence;• Software and its engineering; Keywords Proactive Agent, Agent System, On-call, Large Language Model ACM Reference Format: Fengrui Liu, Xiao He, and Tieying Zhang. 2026. Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement. In Companion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering (FSE ’26), June 5–9, 2026, Montreal, ∗ Corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X Canada. ACM, New York, NY, USA, 12 pages. https://doi.org/X. X 1 Introduction Operational management of Volcano Engine, the large-scale public cloud platform of ByteDance, presents a significant challenge due to the high volume of on-call support requests[17]. The platform receives thousands of customer support tickets daily, which are primarily addressed through on-call chat dialogues. This continu- ous influx of on-calls places considerable strain on human support analysts and engineers, who must handle urgent incidents while also sustaining feature development and routine operations. When the on-call workload spikes (e.g., during large-scale service degra- dations or traffic bursts), timely human responses can become a bottleneck, and customers may experience prolonged waiting time throughout the entire on-call process. To mitigate this operational burden, recent research has explored Large Language Models (LLMs)[1,12] to develop reactive agents for on-call assistance. These reactive agents typically serve as the first line of support, interacting directly with customers to resolve initial inquiries. However, their operational scope is often restricted, they are designed to disengage from the session once an inquiry exceeds their capabilities and is escalated to a human analyst. Although prior work has demonstrated their effectiveness in specific scenarios[3, 29], we identify two critical gaps that limit their utility for real- world on-call support in cloud platforms. First, customers often initiate an on-call session with a high-level problem description, followed by a sequence of increasingly specific questions to converge on a resolution. Progress in such sessions therefore depends on addressing these sub-questions sequentially. In the reactive paradigm, as illustrated in Figure 1(a), the agent is explicitly invoked by the customer and is expected to respond turn by turn, regardless of whether it can provide an adequate answer at each step. However, once the agent fails to resolve a single sub-question, the session is typically escalated to a human analyst, and the customer’s progress depends on analyst availability. After this handoff, the agent’s action cycle terminates, preventing it from assisting with subsequent sub-questions, including those well within its capabilities. This premature disengagement leads to missed opportunities to further reduce analyst workload and shorten end-to-end response time. A second significant challenge in on-call scenarios is the dy- namic context and rapidly evolving knowledge. While Retrieval- Augmented Generation (RAG)[16,35] can improve an agent’s access to domain knowledge, they typically rely on a manually curated knowledge base[15]. In practice, such knowledge bases are often arXiv:2604.09579v1 [cs.AI] 25 Feb 2026 Conference’17, July 2017, Washington, DC, USAFengrui Liu et al. Support Analyst Reactive AgentCustomer I can't connect to my database on my new laptop. Please verify you are using the official client and your database is configured correctly. Yeah, I tried that, but it still doesn’t work. Sorry, could you share more details? Alternatively, I can connect you to human support. I've checked your settings. IP whitelisting is enabled for security, so you'l need to update the whitelist. How can I update it? You can find the IP address of your laptop, then add it to the database's whitelist. Human support please. It works now, thanks! By the way, can I apply for a service discount? ...... Action-cycle of Reactive Agent (a) Reactive Agent: Responds to every query directly until escalation. I've checked your settings. IP whitelisting is enabled for security, so you'l need to update the whitelist. How can I update it? You can find the IP address of your laptop, then add it to the database's whitelist. I can help ! Not in my scope. I can't help Customer I cannot connect to the database via official client. Sorry, could you provide more details? Alternatively, I can connect you to human support. Human support please.Support Analyst & Proactive Agent I tried using the official client to connect to my database, but it failed. It works now, thanks! By the way, can I apply for a service discount? (Accept, otherwise wait for the analyst's reply) ...... Action-cycle of Proactive Agent (b) Proactive Agent: Identifies and selectively answers queries to assist human analysts. Figure 1: Comparison of reactive and proactive agent paradigms in the on-call support process. (a) Reactive agents disengage upon escalation, whereas (b) proactive agents extend the action-cycle to collaborate with human support analysts. relatively static and require continuous manual maintenance. Con- sequently, the effectiveness of existing methods[2,37] depends heavily on the quality, coverage, and timeliness of curated content. This mismatch is particularly costly in fast-evolving on-call situa- tions, where newly discovered symptoms, mitigations, or temporary workarounds are often shared in active on-call sessions. Human analysts may also need to repeat the same explanations across multiple sessions or across different on-call shifts. Because these insights are rarely incorporated into the knowledge base in a timely manner, the agent cannot leverage them to assist other customers experiencing the same issues, resulting in delayed responses and duplicated troubleshooting efforts. For instance, during a network service failure, critical information or temporary workarounds may first emerge in an existing on-call session, yet remain unavailable to the agent when similar tickets arrive shortly afterward. To address these critical gaps and reduce the operational over- head of on-call support, we propose Vigil, a novel system that introduces a proactive agent to operate alongside human analysts throughout the entire on-call life-cycle. Rather than replacing the first-line reactive agent, Vigil specifically targets the collaborative phase after escalation to human support, where analyst availability is often the bottleneck to the customer waiting time. Vigil integrates directly into the dialogue between support analysts and customers, acting not as a sole interlocutor but as an assistant to the human analyst. As illustrated in Figure 1(b), it continuously monitors the ongoing on-call dialogue, and whenever it identifies a customer question within its capabilities, it proactively provides an answer without requiring explicit invocation. By remaining active after escalation, Vigil complements existing first-line reactive agents by extending its action-cycle beyond the initial triage stage, enabling proactive assistance throughout the on-call life-cycle and reducing analyst effort. Furthermore, Vigil incorporates an automated, continuous self- improvement mechanism. Our observations from real-world on- calls suggest that many questions, symptoms, and effective mitiga- tions recur across different sessions, making knowledge captured in one incident valuable for subsequent ones. By leveraging its extended action-cycle, Vigil can learn from both answered and unanswered questions, as well as additional external documents shared during the on-call conversations, turning these information into reusable knowledge. Over time, this mechanism helps keep Vigil’s knowledge more up to date and improves answer quality, reducing repeated troubleshooting effort in future on-calls. The salient contributions of our work are as follows: •We present Vigil, a proactive agent that complements first- line reactive agents by staying active after escalation and autonomously answering in-scope questions during analyst– customer dialogues, thereby extending the on-call action cycle. •We design a continuous self-improvement mechanism that learns from on-call interactions and shared artifacts to update the agent’s knowledge, reducing dependence on manually curated knowledge bases. •We deploy Vigil in the Volcano Engine production environ- ment for over ten months and evaluate it with quantitative metrics and case studies, demonstrating improved opera- tional efficiency and on-call support quality. 2 Related Work The increasing volume of customer support tickets poses significant challenges for Volcano Engine[17]. To address this, our work ex- plores the use of LLM-powered agents for on-call support, with the aim of improving efficiency. This section reviews relevant studies Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-ImprovementConference’17, July 2017, Washington, DC, USA on the application of LLMs in customer support, then introduces proactive agent paradigms, and finally studies the continuous self- improvement mechanism that enhance agent performance. LLM-based Agents in On-call Support. Recent advances have demonstrated that LLM-based agents possess remarkable capabil- ities in natural language understanding and complex reasoning. The advent of LLMs has catalyzed a paradigm shift in automated customer support[30,31]. A growing body of literature has fo- cused on conversational agents[14,26], or chatbots[5], designed to handle customer inquiries and alleviate the workload of human support analysts. Most existing work, however, adopts a reactive paradigm[24], where the agent serves as the primary interlocutor with the customer [14,26]. Such agents are often enhanced with external tools[5,23] and knowledge bases[15,26,35] to improve the coverage and accuracy. Despite their effectiveness in specific scenarios, reactive agents remain limited by the requirement for explicit user invocation for each query[33]. Their action cycle typi- cally terminates if they fail to answer any question, at which point the customer must turn to human support analysts for assistance. This reliance on a reactive action-cycle constrains their potential and underutilizes the broader capabilities of LLMs as compared to proactive approaches[8, 10]. Proactive Agent Paradigms. The LLM-powered proactive agents are characterized by their ability to take initiative, acting without explicit user invocation. Rather than merely responding to explicit requests, they infer user needs from contextual cues[10]. Recent studies have begun to explore proactive behaviors in various do- mains, including code completion[6], tabular data interpretation[9], personalized social bots[22], and general task assistance[19,36]. Further work has examined methods to strengthen proactivity through explicit planning[39], internal reasoning[18], and multi- agent collaboration[38]. Nevertheless, the application of a proactive paradigm in on-call support remains underexplored. Our work ad- dresses this gap by introducing an agent that continuously analyzes an on-call dialogue and intervenes only when identifying an op- portunity to assist. Continuous Self-Improvement Mechanism. A persistent challenge in deploying LLM agents in dynamic domains such as cloud services is ensuring that their knowledge remains up to date. To address this, recent research has investigated self-evolving ap- proaches [11]. Parametric methods, such as reinforcement learning- based adaptation [40], can improve agent performance but require costly offline training, making them difficult to deploy in rapidly changing on-call contexts. By contrast, non-parametric approaches [7,32], which avoid direct parameter fine-tuning, offer a more flex- ible alternative. However, standard RAG[16,35] typically relies on a knowledge base that is curated and updated manually. This main- tenance process introduces delays between the emergence of new knowledge(e.g., a service outage) and its availability to the agent. Some methods[7,32] enable agents to incrementally expand their knowledge bases from interactions and experiences, making them particularly well-suited for the dynamic requirements of on-call support. In this work, we design a continuous self-improvement mechanism that enables the agent to learn from each proactive action, thereby autonomously enriching its knowledge base. 3 Methodology In this section, we introduce Vigil, a proactive agent system de- signed for on-call support. As illustrated in Figure 2, Vigil targets the collaborative phase after human intervention and operates as a proactive assistant within the dialogue between the customer and the human support analyst. It performs two primary functions: (1) Online Proactive Response: Vigil continuously monitors the on- call session, identifies questions, and proactively provides answers. (2) Continuous Self-Improvement: Vigil reviews its proactive ac- tions and learns from the on-call dialogue in a continuous manner, refining its knowledge base and improving future performance. 3.1 Online Proactive Response Different from reactive agents that wait for explicit user invocation, Vigil proactively participates in the on-call between the customer and the human analyst, responding at opportune moments. The action-cycle for Vigil initiates when a human support analyst in- tervenes in the on-call session and concludes upon session closure. The proactive response mechanism involves three key tasks. 3.1.1Context-Aware Question Identification. To function effectively as a complementary assistant alongside human experts, Vigil must discern when to intervene before what to answer. The identification module is designed to bridge the response gap by targeting mo- ments where a customer raises a valid sub-question that the human analyst has not yet addressed, often occurring while the human analyst is occupied with complex diagnostics or multitasking. A critical prerequisite for this identification is a clear capability scope of the agent. Vigil is specialized for the cloud service platform, with its primary mission being the resolution of inquiries regarding cloud products, services, and diverse operational issues. This clear and focused scope ensures that the agent remains dedicated to the platform’s domain knowledge rather than behaving as a generic question-answering assistant. When the on-call dialogue is updated with new messages from the customer, Vigil identifies whether the current message falls within this defined scope. Formally, this identification task is framed as a classification problem. As Figure 2 shows, Vigil analyzes the latest message to categorize it into the following three classes: Within Scope. The message contains a specific question within the scope of Vigil agent, regarding the cloud platform’s products or operational issues. Out of Scope. The message contains questions beyond the specialized capabilities of the agent, such as complex subjective decision-making best left to human judgment. No assistance needed. The message consists of phatic commu- nication, such as greetings, affirmations or queries that the human analyst has already explicitly acknowledged or resolved. Accurate scope confirmation is paramount to maintain user trust and avoid intrusive or irrelevant interruptions. We implement this identification logic using a specialized prompt for the LLM, the details of which are provided in Appendix A.1. Vigil proceeds to the answer generation phase only when a message is classified as Within Scope, ensuring its proactive response is both timely and necessary. For questions outside this scope, Vigil is expected to refrain from intervening. Conference’17, July 2017, Washington, DC, USAFengrui Liu et al. Returned Information Proactive AgentCustomer Updates I can't connect to my database. Please make sure you are ... It still doesn’t work. How can I update it? It works now. Can I ... Updates observed and start analyzing Identify the question No action Knowledge Base Reranked documents Answer Generation Agent's historical answers MCP/A2A Tools & Agents CollectionLog IDAlert eventOthersRAG Review historical questions Answered questions Unanswered questions Save Extract answers from follow-ups Keep & Update & Delete Answer review (feedbacks) Continuous Self-ImprovementOnline Proactive Response Within scope Out of scope No assistance needed duplicate Answer deduplication non-duplicate I can't helpI can help ... please update the whitelist. You can find the IP address ... Sure. You can ... Figure 2: Framework of Vigil. Features with two primary functions: (1) Online Proactive Response; (2) Continuous Self-Improvement. Furthermore, for some operational issues, Vigil’s capability ex- tends by invoking tools to retrieve detailed logs, associated alerts, and diagnostic metadata. This integration allows Vigil to ground its responses in real cloud system, providing highly specific and actionable assistance. 3.1.2 Answer Generation. After identifying a question within its capability scope, Vigil proceeds to generate an answer. Because the on-call sessions are multi-turn dialogues, customer questions may be implicit and require a comprehensive understanding of the conversational context. Therefore, Vigil first analyzes the entire dialogue history to rewrite the newly identified question. This step utilizes established techniques[20,21] to make the question explicit and self-contained, including deconstructing complex queries and resolving coreferences and ellipses. Next, the rewritten questions are processed by the agent. Retrieval- Augmented Generation (RAG) plays a crucial role in this process, not only supplementing domain-specific knowledge gaps in the foundational LLM but also providing reliable references for the gen- erated answers. Specifically, Vigil employs a multi-path knowledge retrieval strategy[4,34], which includes a domain-related docu- ment knowledge base and a collection of question-answer pairs from historical on-call sessions. A reranking model then reorders the retrieved results based on their relevance to the question. Finally, Vigil synthesizes all collected information, including the multi-turn dialogue context, tools-calling information and retrieved knowledge, to generate a final answer. The agent retains the option to refuse to answer if it determines that the available information is insufficient for a reliable response. A detailed prompt for this is in Appendix A.2. 3.1.3 Answer Deduplication. In a real-world on-call session, the customer may repeatedly discuss a same topic by rephrasing the question, providing additional background, or expressing dissatis- faction with a previous answer. The proactive mechanism of Vigil could trigger multiple and similar responses to these updates. To avoid this, Vigil performs an answer deduplication step to intercept redundant replies. We employ a semantic similarity-based deduplication method for this task. When a new answer is generated, its embedding vector is calculated using a pre-trained sentence embedding model. The co- sine similarity of this embedded vector is then computed against the embeddings of all historical answers sent during the current on-call session. If the maximum similarity exceeds a predefined threshold 휃, the new answer is considered redundant and suppressed. The threshold휃is set empirically to balance the trade-off between re- dundancy avoidance with comprehensive assistance. It is highly relevant to the embedding model, and we use an empirical value of 0.7, which is further discussed in the evaluation section. If the answer is considered to be unique, Vigil sends an an- swer card to the on-call. Compared to LLM-based approaches, this similarity-based deduplication method is significantly faster while remaining effective. We perform deduplication at the answer stage rather than the question identification stage, because the knowl- edge base may be updated as the on-call session progresses. Conse- quently, even identical may lead to different responses within the same session, making answer-level deduplication more appropriate. 3.1.4 Proactive Response Card Design. To ensure that the proac- tive responses of Vigil are constructive without being intrusive or misleading, we employ a specialized card-based interface. Un- like reactive agents that often dominate the conversational flow, Vigil presents its findings in a structured card that visually demar- cates LLM-generated content from the ongoing human-to-human dialogue. Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-ImprovementConference’17, July 2017, Washington, DC, USA Figure 3: Proactive response card of Vigil. Including a distinct layout for distinguishing agent from human, explicit citations for verifiability and an Accept button for collecting feedback. As shown in Figure 3, this card features a distinct avatar and layout to differentiate it from human messages. To ensure ethical transparency, the card explicitly displays the unique identity and provides citation links to ground the generated answer. This clear distinction ensures that customers can instantly discern automated suggestions from human analyst responses. Furthermore, the card incorporates an interactive Accept button for feedback. It allows customers to acknowledge the answer or human analysts to verify it, effectively marking the question as resolved to streamline the on-call process. Simultaneously, this interaction serves as verified ground truth, helping Vigil assimilate high-quality answers for self-improvement. 3.2 Continuous Self-Improvement Benefiting from the proactive mechanism, Vigil maintains a long action-cycle that spans the entire on-call session. It not only answers the questions, but also follows the dialogue updates from both the customer and the human analyst. This capability allows Vigil to learn from customer feedback and support analyst responses, enabling continuous refinement of its knowledge base. As shown in Figure 4, this module reviews every query that within its scope, handling answered and unanswered questions differently. 3.2.1 Learning from Unanswered Questions. Even for questions within its capability scope, Vigil may occasionally fail to generate a response due to the absence of suitable references in the existing knowledge base. In these scenarios, the human support analyst inevitably steps in to resolve the issue. The mandatory nature of on-call support guarantees that human analysts must eventually address these customer inquiries. Vigil treats these human-led res- olutions as valuable learning opportunities. Vigil is designed to autonomously parse the subsequent analyst- customer dialogue and extract the solution provided by the human expert. If a definitive answer is identified, it is structured as a new question-answer pair and stored into the knowledge base. This process incrementally enriches the knowledge base and expands the coverage for future inquiries, enabling Vigil to handle similar queries in the future. It is important to note that these autonomously extracted an- swers may initially be constrained by the specific context of the original dialogue. However, this limitation is temporary. Once these Q1 A1 agent Q2 Continuous Self-Improvement A2 human I can help I can't help On-call Dialogue Knowledge Base Unanswered question Answered question Question Rewritten (sub-)questions Extract answer from follow-ups Save <Q,A> Construct <Q,A> if answer exists Question Answer is accepted Save <Q,A> Review answer from follow-up dialogue Correct or Not found Incorrect Partially Correct No action Reference is outdated References used for the answer Review historical on-call that generated this reference Update Identify differences Delete On-call session Dialogue Find reference link Fetch reference document Save Figure 4: Continuous self-improvement framework. Vigil learns from answered questions, unanswered questions, and exter- nal documents shared in the on-call dialogue. entries are retrieved and utilized in future on-calls, they become subject to the validation mechanisms in the Learning from An- swered Questions module. Through the Update operation described previously, these specific answers are iteratively polished and gen- eralized, ensuring the continuous evolution of the knowledge base quality. 3.2.2 Learning from Answered Questions. For the questions that Vigil can address, it proactively sends an answer card to the on-call. We consider positive feedback (e.g., an Accept click) as a strong endorsement. Accordingly, the corresponding rewritten question and generated answer can be stored as a new question-answer (QA) pair in the knowledge base. Notably, a direct negative feedback option is omitted. This de- sign is motivated by the observation that the criteria for negative feedback in real-world dialogues are often ambiguous and inconsis- tent. For instance, users may flag an answer as unsatisfactory due to minor factual inaccuracies, incomplete explanations, irrelevant references, or broken reference links. In other cases, users might simply skip providing feedback and continue the on-call dialogue, even when they are dissatisfied. Such variability makes it difficult to reliably interpret negative feedback and collect the detailed reasons behind it. To ensure a more consistent refinement process, Vigil employs an agent to automatically review and evaluate answers that appear to be unaccepted. Vigil identifies whether a human analyst subse- quently provides a more satisfactory answer (Appendix A.3) and takes action accordingly. As detailed in Figure 4, the review process follows the following three paths: Conference’17, July 2017, Washington, DC, USAFengrui Liu et al. Keep. If the human support analyst provides a follow-up answer consistent with the answer generated by Vigil, or if the dialogue proceeds without further discussion of the topic, the answer is re- garded as correct. In these cases, the reference used to generate the answer is deemed appropriate for the question, and no updates is required. It is important to note that this Keep status is not perma- nent, every retained entry remains subject to continuous validation in future proactive cycles, ensuring that knowledge which becomes obsolete is eventually re-evaluated. Delete. In some cases, the review agent finds that an answer is completely incorrect based on the follow-up on-call dialogue. We note that the heavily depended references, which were extracted from previous on-call sessions, may not match the context of the current question. For instance, the references may be outdated, too specific to generalize, or no longer suitable for the question. To prevent these inappropriate references from misleading future answers, Vigil deletes them from the knowledge base. Update. If the answer has only minor discrepancies compared to the subsequent human-provided answer, the agent reviews the causes of these differences. Such discrepancies may arise from vari- ations in background context or missing prerequisite information. To address this, Vigil augments the references with additional de- tails that help identify the scenario of the question and improve answer accuracy. The refined question–answer pair is then used to update the corresponding entry in the knowledge base. Overall, this review-action framework establishes a robust self- correcting feedback loop. Even if a temporary workaround or a non-generic solution is autonomously learned from a human an- alyst, its long-term impact is mitigated by the system’s proactive nature. If such an entry leads to an unaccepted answer in future sessions, it inevitably triggers the Update or Delete process during the next review cycle. This iterative verification effectively pre- vents performance degradation caused by outdated or low-quality knowledge. 3.2.3 Learning from External Documents. In addition to learning from direct dialogues, Vigil actively reviews external documents referenced within on-call sessions to extend its learning scope. This mechanism mirrors the workflow of human analysts, who frequently validate their solutions by sharing links to official docu- mentation, standard operating procedures, or API references. This reliance on external resources addresses two critical chal- lenges in textual support. First, for intricate issues requiring sequen- tial execution, typing out lengthy instructions in a chat interface is often inefficient and prone to ambiguity; comprehensive manuals provide a superior, structured format. Second, for dynamic content such as API parameters and versioning, static text in chat messages risks becoming rapidly obsolete. By linking to official documents, analysts ensure customers access the most current and canonical information. Unlike the fragmented information found in casual conversation, these documents offer systematic and reusable knowledge. Vigil is designed to automatically capture these high-value resources by parsing the shared links. By extracting relevant content from these authoritative sources, the agent transforms transient support interactions into a persistent, multi-faceted knowledge repository, ensuring its answers are accurate, up-to-date, and well-documented. 4 Experiments This section presents a comprehensive evaluation of Vigil, designed to answer the following research questions: Q1: To what extent does Vigil provide extended support coverage during the human-involved phase of on-call sessions? Q2: How accurately can Vigil identify questions that fall within its ability scope? Q3: How accurately can Vigil proactively provide answers to the questions it identifies? Q4: Can Vigil effectively prevent providing duplicate answers within the same on-call dialogue? 4.1 Real-world Deployment and Statistics The Vigil system has been deployed in the production environ- ment of the Volcano Engine cloud platform since March 13, 2025. The deployment was phased, beginning with a limited set of on- calls and gradually expanding to full coverage by June 1, 2025. By January 16, 2026, Vigil had processed a total of 131,433 on-calls, en- compassing 2,317,760 dialogue messages. During this period, Vigil autonomously collected 272,325 knowledge entries and generated 136,234 proactive responses. 4.2 Dataset Construction and Labeling 4.2.1 Data Sampling. To conduct a rigorous evaluation of Vigil for the research questions Q2, Q3, and Q4, we constructed a high- quality evaluation dataset. It is important to note that real-world on-call inquiries are inherently multi-modal, typically consisting of textual descriptions accompanied by visual evidence, such as screenshots of error logs or monitoring dashboards. To reflect this reality, our sampling process preserved both text and image data. Considering that the production environment involves sensitive customer data, and some foundation models used in our experi- ments are closed-source (e.g., GPT and Gemini series models), we sample 300 non-sensitive on-call data from the production environ- ment. All subsequent experiments are strictly conducted on the 300 non-sensitive on-call datasets to ensure data security compliance. 4.2.2 Data Labeling. To establish a robust ground truth, we em- ployed a hybrid labeling strategy that combines explicit online feedback with offline expert review. As described in Section 3.1.4, the proactive response card features an interactive Accept button (Figure 3). We treat this interaction as a high-confidence signal for automatic labeling. Specifically, if a response is explicitly accepted during the on-call session, it is automatically labeled as a positive sample for scope identification and a correct answer for generation quality. This online mechanism ensures that a portion of our ground truth directly reflects real- world utility and user satisfaction. However, relying solely on online feedback may yield false neg- atives, as accurate responses might be inadvertently overlooked. To address this, we engaged human support analysts to refine the dataset for the 300 sampled on-calls. The analysts performed a com- prehensive review to handle the remaining data. For each message from the on-call customers, human analysts first manually label whether the message is within the scope of Vigil, such as whether the message contains a question related to the platform. This can Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-ImprovementConference’17, July 2017, Washington, DC, USA help to evaluate the question identify module of Vigil, making it seize every opportunity to assist. Besides, for generated answers that without being accepted, analysts evaluate their accuracy, la- beling them as Correct or Incorrect. This can help further verify the correctness of these answers, thereby determining whether these responses are helpful and providing a positive impact on the overall on-call process. 4.3 Experiment Settings 4.3.1 Foundation Models Comparison. As a generic system, Vigil is designed to be compatible with various foundation large lan- guage models. Given the multi-modal nature of the dataset con- structed in Section 4.2.1, it is imperative to utilize models capable of processing both text and visual inputs. Therefore, to evaluate the flexibility and performance of Vigil, we select a diverse set of state-of-the-art multi-modal LLMs as the foundation models in this experiments. The chosen models range from top-tier proprietary models, including Seed-1.6, Seed-1.6-flash[27], GPT-5, GPT-5-mini, Gemini-2.5-pro, Gemini-2.5-flash[12], to the open-weight model Qwen-2.5-VL-72b[25]. It is worth noting that in real-world industrial deployments, the selection of a foundation model is rarely dictated by generation quality alone. Critical factors such as data security, compliance, inference latency, and operational cost often take precedence. For instance, while proprietary SOTA models may offer superior reason- ing, internal security policies may mandate the use of self-hosted or region-specific models for handling sensitive customer data. Although we do not explicitly model these constraints in our exper- iments, we account for this industrial reality by ensuring Vigil is tested across a broad spectrum of models. Consequently, our focus is not on identifying the best foundation model, but on assessing the effectiveness of Vigil with different models via ablation and parameter studies across this diverse landscape. 4.3.2Evaluation Metrics. The experimental evaluation is conducted using퐴푐푢푟푎푐푦( 푇푃+푇푁 푇푃+푇푁+퐹푃+퐹푁 ),푃푟푒푐푖푠푖표푛( 푇푃 푇푃+퐹푃 ) and푅푒푐푎푙( 푇푃 푇푃+퐹푁 ). To account for the class imbalance present in the dataset, we fur- ther report the weighted precision (푃푟푒푐푖푠푖표푛 푤 = Í 푛 푖 푃푟푒푐푖푠푖표푛 푖 Í 푛 푖 )), weighted recall (푅푒푐푎푙 푤 = Í 푛 푖 푅푒푐푎푙 푖 Í 푛 푖 ), and weighted F1-score (퐹1 푤 = Í 푛 푖 퐹1 Í 푛 푖 ) as additional evaluation metrics. Given the diverse constraints that dictate model selection in production, comparing the absolute scores of different foundation models provides limited insight into the Vigil framework itself. Therefore, in addition to absolute scores, we highlight the relative improvement (marked as↑) that Vigil achieves for each specific foundation model, as this marginal utility is the key indicator of the effectiveness of Vigil in production. To align with the evaluation targets of Vigil, we adopt task- specific evaluation protocols. For scope identification (Q2) of Vigil, we directly compute the aforementioned classification metrics against the human-annotated ground truth. For answer correctness (Q3), we employ an LLM-as-a-Judge[13] to determine whether a gener- ated response is factually correct based on the user message and associated visual evidence. The judge’s binary decision is then used to calculate the same set of metrics. 13441 5663 1926 789 344 153 116 59 38 22 43 59.5% 84.6% 93.1% 96.6% 98.1% 98.8% 99.3% 99.5% 99.7% 99.8% 100.0% 012345678910+ 2 5 100 2 5 1000 2 5 10k 60 70 80 90 100 Number of responses Cumulative percentage Number of proactive responses during human intervention phase Number of responses Cumulative percentage (a) Distribution of On-calls by Volume of Extended Assistance Provided by Vigil Jun 22 2025 Jun 29Jul 6Jul 13Jul 20 0 200 400 600 800 1000 1200 1400 1600 Number of proactive responses (b) Number of Proactive Responses of Vigil Over Time Figure 5: Volcano Engine On-Call Statistics, June 23 – July 22, 2025 4.4 Q1: Statistical Analysis of the Extended Action-Cycle We investigate Q1 by quantifying the additional support provided by Vigil. Instead of replacing the first-line support which is often handled by reactive agents, Vigil is designed to extend the auto- mated action-cycle into the human-intervention phase. To analyze this, we conduct a statistical analysis on 22,594 on-call sessions over a one-month period from June 23 to July 22, 2025. Figure 5(a) presents the distribution of on-calls based on the volume of responses provided by Vigil after human intervention. As shows from the figure, in 40.5% of all on-calls, Vigil continued to provide valuable assistance even after the human analyst had joined the session. This result highlights the benefit of the extended action-cycle, rather than disengaging upon escalation, Vigil acts as a complementary assistant, addressing sub-questions and retrieving information alongside the human expert. Furthermore, a key feature of this deployment is its zero-setup design, Vigil integrates into all on-calls without requiring a manu- ally curated knowledge base. It autonomously enriches its knowl- edge base from all the dialogues. Figure 5(b) illustrates a consistent upward trend in the number of daily proactive responses, indicat- ing that the agent is successfully accumulating knowledge from Conference’17, July 2017, Washington, DC, USAFengrui Liu et al. Table 1: Evaluation of question identification of Vigil across various foundation models. Foundation푃푟푒푐푖푠푖표푛 푤 푅푒푐푎푙 푤 /Accuracy퐹1 푤 ModelBaseline VigilΔ (%)Baseline VigilΔ (%)Baseline VigilΔ (%) GPT-50.7890.87010.3%0.6200.85037.1%0.6600.85729.8% GPT-5-mini0.8490.8753.0%0.3070.793158.7%0.2680.813203.2% Gemini-2.5-flash0.8110.8596.0%0.7000.7375.2%0.7300.7644.6% Gemini-2.5-pro0.7840.8275.5%0.7570.8309.7%0.7680.8287.9% QWen-VL-72b0.8480.827-2.5%0.2870.677136.0%0.2350.711202.0% Seed-1.60.7990.8374.9%0.6400.81327.1%0.6780.82221.3% Seed-1.6-flash0.8050.8424.6%0.5600.65717.3%0.6010.69215.2% human-led resolutions to cover a broader range of queries over time. 4.5 Q2: Question Identification Evaluation To evaluate the question identification module of Vigil, we compare the performance of several foundation LLMs with and without this module. In the baseline configuration, which is without this module, the foundation model needs to implicitly decide whether to answer a query during the answer generation phase. For this experiment, we regard on-call messages that should be answered as positive samples, and those that should not be answered as negative samples. Performance is then measured using푃푟푒푐푖푠푖표푛 푤 ,푅푒푐푎푙 푤 which is equal to 퐴푐푢푟푎푐푦, and 퐹1 푤 . Table 1 indicates that the dedicated question identification mod- ule of Vigil improves performance across nearly all metrics and models. This enhancement can be attributed to decomposing the complex task of simultaneously identifying and answering a ques- tion into a discrete classification step. The degree of improvement varies among the models. The Gemini series, which exhibits strong inherent capabilities in question identification, shows more modest gains. The퐹1 푤 increased by 4.6% for Gemini-2.5-flash and 7.9% for Gemini-2.5-pro. Other models, including the GPT-5, Seed-1.6, and Qwen-VL series, demonstrate significant advancements when augmented with Vigil. Overall, the improvement in the퐹1 푤 ranges from 15.2% for Seed-1.6-flash to 203.2% for GPT-5-mini. These re- sults suggest that models initially less suitable for this task can achieve proficient performance with the introduction of the ques- tion identification module, such as QWen-VL-72b (202.0% improve- ment) and GPT-5-mini (203.2% improvement). Furthermore, this module offers a flexible mechanism for constraining Vigil to ad- dress domain-specific inquiries, such as questions regarding cloud platforms, while filtering out general queries. 4.6 Q3: Ablation Study on the Accuracy of Proactively Generated Answers The accuracy of proactively answered questions is a critical metric for evaluating the effectiveness of Vigil. It reflects whether Vigil can provide correct and helpful information. Following our evaluation protocol, we use an LLM-as-a-Judge[13] to determine whether each proactively generated response is Correct or Incorrect given the user inquiry (incorporating associated visual evidence where available), and report accuracy based on these binary judgments. We compared the full Vigil system against two ablated versions: one without Table 2: Ablation Study on the Answer Accuracy of Vigil Foundation푤/표푤/표푤/ Modelself-impr.answer reviewVigil GPT-50.3890.537 (37.9%↑)0.624 (60.3%↑) GPT-5-mini0.1780.322 (81.1%↑)0.493 (177.4%↑) Gemini-2.5-flash0.1610.399 (147.9%↑)0.423 (162.5%↑) Gemini-2.5-pro0.2520.430 (70.7%↑)0.456 (81.3%↑) Qwen-VL-72b0.2250.322 (43.3%↑)0.396 (76.1%↑) Seed-1.60.1540.409 (165.2%↑)0.591 (282.6%↑) Seed-1.6-flash0.1580.379 (140.4%↑)0.510 (223.4%↑) the entire self-improvement module (푤/표self-impr.) and another without only the answer review component (푤/표answer review). The variant labeled "푤/표self-impr." relies solely on a foundation LLM and is treated as the baseline. The results in Table 2 show that the full Vigil system consis- tently and substantially outperforms both ablated versions across all models. The self-improvement module provides the large gains by enabling the agent to learn domain-specific knowledge from dialogues autonomously. With a minimum improvement of 60.3% for GPT-5 and a maximum of 282.6% for Seed-1.6. The answer re- view component further refines accuracy by updating incomplete knowledge and deleting outdated entries. This layered approach demonstrates that continuous learning is essential for improving answer accuracy over time. In our real-world deployment which is based on Seed-1.6, the accuracy of proactively answered ques- tions is 0.591. In practice, this means that more than half of the questions identified and answered by Vigil are correct and helpful, significantly reducing the workload of human support analysts. 4.7 Q4: Answer Deduplication Evaluation In real-world on-call scenarios, multi-turn conversations may re- volve around the same topic. In Vigil, a predefined threshold휃is introduced to balance the trade-off between avoiding redundancy and ensuring comprehensive assistance. Noting that the optimal parameter may vary across different foundation embedding models, we conducted parameter selection experiments on a fixed embed- ding model seed-1.6-embedding[28]. In this evaluation, we designate questions that are expected to be answered as positive samples, while those that should not be answered are treated as negative samples. Under this condition, Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-ImprovementConference’17, July 2017, Washington, DC, USA Table 3: Evaluation metrics of answer deduplication under different threshold 휃. 휃 푃푟푒푐푖푠푖표푛 푅푒푐푎푙 푃푟푒푐푖푠푖표푛 푤 푅푒푐푎푙 푤 퐹1 푤 0.00 a 1.0000.3960.8640.5090.542 0.200.9960.3980.8610.5100.543 0.400.8860.5850.7710.6020.646 0.600.8260.9210.7300.7780.747 0.700.8220.9820.7600.812 0.752 0.800.8170.9940.7700.8140.740 0.900.8160.9990.8320.8160.738 1.00 b 0.8131.0000.8480.8130.731 a 휃= 0 means all answers after the first are treated as duplicates. b 휃=1 corresponds to an ablation study of the answer deduplication module. 푃푟푒푐푖푠푖표푛reflects the ability of the deduplication module to sup- press redundant responses, while푅푒푐푎푙measures its ability to pre- serve useful and distinct answers. Considering that the imbalance of these two classes, we also report the weighted evaluation metrics. By comparing these metrics under different threshold settings, we assess the effectiveness of our approach as Table 3 shows. In particular, setting the threshold to 0 (휃=0) means that after the first answer is sent, all the following answers will be considered as duplicates. While setting the threshold to 1 (휃=1) corresponds to an ablation study of the answer deduplication module, where all answers are returned without deduplication. From the results, we observe that as the threshold increases, the recall improves while precision declines. The primary evaluation metric, the weighted F1-score (퐹1 푤 ), is maximized at a threshold of 0.7, achieving a score of 0.752. This result is highlighted as the optimal trade-off based on the weighted metrics. 5 Case Studies To complement the quantitative evaluation, this section presents case studies from real-world deployments on the Volcano Engine cloud platform. Two representative scenarios are examined to illus- trate the practical advantages of Vigil. Case 1: It demonstrates how the timely and automated self- improvement mechanism of Vigil enhances proactive response capabilities during emergency incidents. Case 2: It illustrates the iterative evolution of knowledge, where Vigil identifies mismatched information and progressively refines the contextual precision of its knowledge base. 5.1 Case1: Proactive Response Enhancement in Emergency Incidents The effective management of emergency incidents, though infre- quent, is afforded the highest priority in Volcano Engine. An emer- gency incident often manifests as multiple on-calls within a short period. Critical information in such situations, including problem context, service status, troubleshooting progresses and proposed resolutions, are often fragmented across different on-call dialogues. A notable incident occurred in June 2025, when different cus- tomer reported service disruption caused by the host machine fail- ure within the same day. As Figure 6 shows, during the first on-call Support AnalystCustomer A OK, please fix it as soon as possible. Please manually migrate the service to reserved host X. The long-term fix will be released tomorrow. Get new knowledge and save I can't help Proactive AgentCustomer B Please investigate the downtime of host A urgently, as my service is not functioning. I'l try it. Please temporarily migrate the service manually to reserved host X; this issue will be resolved within one day. I can help My deployed service and its host A are down. 9:27 a.m. 9:41 a.m. Knowledge Base Figure 6: Vigil operates different on-call dialogues for the same critical issue (simplified and anonymized). support, Vigil was unable to provide a resolution due to insuffi- cient contextual information. The site reliability engineering team observed that the automatic evacuation protocol had failed to exe- cute as expected. The team proposed a temporary workaround of manually migrating the service to a reserved host, while simulta- neously preparing a long-term fix within a day. By processing both the human analyst’s response and an associated support document from this initial incident, Vigil assimilated the necessary knowl- edge. Later that day, another customer raised another on-call for the identical incident. This time, benefiting from its automated self- improvement mechanism, Vigil proactively provided the correct temporary workaround before a human analyst intervened. This example demonstrates how Vigil effectively reused knowl- edge from the first incident to proactively address the second one, thereby improving the efficiency. This success is directly attribut- able to its continuous self-improvement framework, which enables timely learning from resolved inquiries and shared documentation. Such automation significantly enhances technical support efficiency and delivers substantial value in emergency scenarios. 5.2 Case2: Iterative Improvement of Response Quality In real-world troubleshooting scenarios on Volcano Engine, each individual on-call dialogue is inherently constrained by its spe- cific context. These dialogues often begin with a description of the observed phenomenon, while the underlying root causes and solutions are then gradually uncovered through discussion. The contextual details from such isolated cases are often insufficient to form a comprehensive knowledge-base entry. Instead, a generaliz- able solution typically emerges only after synthesizing information from multiple related on-calls. A case from July 2025 involving an object file upload error with a SignatureDoesNotMatch code illustrates this process. As shown in Figure 7, when this problem occurs, a customer initially only sees an error message and opens an on-call with this minimal log. During the first occurrence, Vigil was unable to provide an immediate solution. By following the subsequent dialogue, it learned that this was common in unofficial SDKs and could be resolved by encoding the file name in UTF-8 to match the signature. When a Conference’17, July 2017, Washington, DC, USAFengrui Liu et al. Support AnalystCustomer A OK, it works. It may be due to non-UTF-8 characters in the object name, which is common when using unofficial SDK. Get new knowledge and save I can't help Proactive AgentCustomer B It is already UTF-8.... I'm using the official Go SDK. I can help Error log: "Failed to upload file, SignatureDoesNotMatch" Proactive AgentCustomer C The SignatureDoesNotMatch error may be caused by: * Object file name containing non-UTF-8 characters, please ensure the name is UTF-8 encoded. * If you are using Go SDK v2.1.3, which has a known bug in STS-token handling, please upgrade to the latest version. Error log: "Failed to upload file, SignatureDoesNotMatch" Error log: "Failed to upload file, SignatureDoesNotMatch" Go SDK 2.1.3 version contains a known bug in STS-token handling. Please upgrade to the latest version. Please ensure use UTF-8 encoding for the file name. Review the answer I can help Knowledge base Q: SignatureDoesNotMatch error when uploading a file object. A: Ensure that the object name is UTF-8 encoded. Q: SignatureDoesNotMatch error when uploading a file object. A: 1. Ensure that the object name is UTF-8 encoded when using unofficial SDK. 2. Go SDK v2.1.3 has a known bug in STS-token, and need to upgrade the SDK Unofficial SDK and Go SDK v2.1.3 are different review Use the updated knowledge Figure 7: Vigil reviews the unaccepted answer and refines its knowledge base (simplified and anonymized). second customer later encountered the same issue, Vigil proactively suggested the UTF-8 encoding solution. Nevertheless, the answer was not accepted by the customer. Within our framework, such non- acceptance triggers a review mechanism, preventing potentially incorrect knowledge from being reused in future cases. At the review stage in self-imporovement, Vigil analyzed the differences between these two on-calls involving the same error symptoms. It identified that the second customer was using the Go SDK v2.1.3 with a known bug affecting the handling of STS tokens. Thus, while the problem manifestation appeared identical, the root causes were different, necessitating a distinct solution. The self-improvement module identified this discrepancy and executed an Update operation, refining the existing knowledge entry with these new contextual constraints. This case study demonstrates that identical symptoms in on-call support can stem from disparate root causes. Such a self-improvement loop is essential for enabling Vigil to iteratively filter out mis- matched knowledge and continuously enhance the accuracy and contextual relevance of its learned knowledge. 6 Discussion and Lessons Learned Our experience deploying Vigil at Volcano Engine yielded several generalizable lessons. The Criticality of Proactive Response Latency. A key lesson from our deployment is that proactive response speed is important. The core value of Vigil is integrating into the live on-call dialogue between a customer and a human analyst. To be effective, its an- swers must be delivered before the human analyst formulates a response. This real-time requirement imposes a significant design constraint. It means that complex reasoning strategies such as multi- step planning, reflection, or chained tool usage, while powerful in other agent applications, are not suitable for on-call due to their inherent latency. A more sophisticated answer that arrives too late is less valuable than a timely one. The Ambiguity of Explicit Feedback. Explicit interactions, such as clicking Accept, provide high-confidence positive signals. However, an unaccepted answer is inherently ambiguous and does not necessarily indicate low answer quality. We observe that users often overlook feedback mechanisms due to operational factors, such as high-pressure troubleshooting environments or habitual focus on the chat interface rather than surrounding UI compo- nents. Early versions of our work that relied solely on these ex- plicit feedback as a universal metric for knowledge maintenance would introduce a significant conservative bias, leading the sys- tem to erroneously discard valid and helpful knowledge simply because it was not formally acknowledged. To mitigate this issue, Vigiltreats explicit positive feedback as a sufficient—but not neces- sary—condition for validation. For cases where feedback is absent, we decouple knowledge evolution from user actions by employing a dialogue-aware review mechanism. By analyzing the subsequent conversation context, the system can autonomously distinguish between truly incorrect answers and helpful but unacknowledged ones. This design ensures that the self-improvement loop remains robust and data-driven, rather than being constrained by the users’ feedback behavior. The Evolving Impact of Foundation Models. Our experi- ments show that system performance is strongly influenced by the choice of foundation model. Accordingly, the current modu- lar design of Vigil structures the LLM’s behavior by decomposing complex tasks into a sequence of simpler steps. This approach is particularly beneficial for foundation models that struggle with complex, multi-part instructions, enabling substantial performance gains. However, as foundation models continue to improve in rea- soning and instruction-following, the need for explicit architectural decomposition may diminish. In such cases, sufficiently capable models may reliably perform both identification and generation within a single, streamlined prompt, simplifying the overall system. For now, our results suggest that a modular approach remains the most robust and practical strategy for maximizing performance and ensuring reliability across the diverse landscape of currently available foundation models. 7 Conclusion and Future Work In this paper, we identify key limitations of reactive agents: they are constrained by short action cycles and cannot learn effectively from ongoing, human-led incident resolutions. To address these limitations, we propose Vigil, a proactive agent–based system for on-call support deployed on the Volcano Engine cloud platform. Vigil enables proactive responses and supports continuous self- improvement throughout the entire on-call lifecycle. Our work establishes an efficient and practical paradigm for human–AI col- laboration in on-call support. Despite these advances, several challenges remain before proac- tive agents can fully close the loop in real-world on-call opera- tions. Future work will focus on extending proactive capabilities from providing answers to performing actions, such as automated diagnostics. We will also investigate adaptive reasoning mecha- nisms that can balance the critical trade-off between low-latency responses and deep reasoning to further enhance the utility of Vigil in complex on-call scenarios. Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-ImprovementConference’17, July 2017, Washington, DC, USA References [1] 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/. [2] Rama Akkiraju, Anbang Xu, and Deepak et.al. Bora. 2024. FACTS About Building Retrieval Augmented Generation-based Chatbots. arXiv:2407.07858 [cs] doi:10. 48550/arXiv.2407.07858 [3]Dincy R. Arikkat, Abhinav M, Navya Binu, Parvathi M, Navya Biju, K. S. Arunima, Vinod P, Rafidha Rehiman K. A, and Mauro Conti. 2024. IntellBot: Retrieval Aug- mented LLM Chatbot for Cyber Threat Knowledge Delivery. arXiv:2411.05442 [cs] doi:10.48550/arXiv.2411.05442 [4]Maciej Besta, Ales Kubicek, Robert Gerstenberger, Marcin Chrapek, Roman Niggli, Patrik Okanovic, Yi Zhu, Patrick Iff, Michal Podstawski, Lucas Weitzendorf, Mingyuan Chi, Joanna Gajda, Piotr Nyczyk, Jürgen Müller, Hubert Niewiadomski, and Torsten Hoefler. 2025. Multi-Head RAG: Solving Multi-Aspect Problems with LLMs. arXiv:2406.05085 [cs] doi:10.48550/arXiv.2406.05085 [5]Som Sekhar Bhattacharyya. 2024. Study of Adoption of Artificial Intelligence Technology-Driven Natural Large Language Model-Based Chatbots by Firms for Customer Service Interaction. Journal of Science and Technology Policy Manage- ment (May 2024). doi:10.1108/JSTPM-11-2023-0201 [6] Mark Chen, Jerry Tworek, Heewoo Jun, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs] doi:10.48550/arXiv.2107.03374 [7]Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2023. Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory. arXiv:2305.02437 [cs] doi:10.48550/arXiv.2305.02437 [8]Yang Deng, Wenqiang Lei, Wai Lam, and Tat-Seng Chua. 2023. A Survey on Proac- tive Dialogue Systems: Problems, Methods, and Prospects. arXiv:2305.02750 [cs] doi:10.48550/arXiv.2305.02750 [9] Yang Deng, Wenqiang Lei, Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. 2022. PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 6970–6984. doi:10.18653/v1/2022.emnlp-main.469 [10]Yang Deng, Lizi Liao, Zhonghua Zheng, Grace Hui Yang, and Tat-Seng Chua. 2024. Towards Human-centered Proactive Conversational Agents. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Washington DC USA, 807–818. doi:10.1145/3626772. 3657843 [11]Huan-ang Gao, Jiayi Geng, Wenyue Hua, et al.2025. A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence. arXiv:2507.21046 [cs] doi:10. 48550/arXiv.2507.21046 [12] Google. 2025. Gemini 2.5 Pro. https://deepmind.google/models/gemini/pro/. [13] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A Survey on LLM-as-a- Judge. arXiv:2411.15594 [cs] doi:10.48550/arXiv.2411.15594 [14]Dongling Huang, Dmitri G. Markovitch, and Rusty A. Stough. 2024. Can Chatbot Customer Service Match Human Service Agents on Customer Satisfaction? An Investigation in the Role of Trust. Journal of Retailing and Consumer Services 76 (Jan. 2024), 103600. doi:10.1016/j.jretconser.2023.103600 [15] Jiho Kim, Yeonsu Kwon, Yohan Jo, and Edward Choi. 2023. KG-GPT: A Gen- eral Framework for Reasoning on Knowledge Graphs Using Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9410–9421. doi:10.18653/v1/2023.findings-emnlp.631 [16]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 9459–9474. [17]Fengrui Liu, Xiao He, Tieying Zhang, Jianjun Chen, Yi Li, Lihua Yi, Haipeng Zhang, Gang Wu, and Rui Shi. 2025. TickIt: Leveraging Large Language Models for Automated Ticket Escalation. In Proceedings of the 33rd ACM International Con- ference on the Foundations of Software Engineering. 343–354. arXiv:2504.08475 [cs] doi:10.1145/3696630.3728558 [18]Xingyu Bruce Liu, Shitao Fang, Weiyan Shi, Chien-Sheng Wu, Takeo Igarashi, and Xiang ’Anthony’ Chen. 2025. Proactive Conversational Agents with In- ner Thoughts. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–19. doi:10.1145/3706598.3713760 [19]Yaxi Lu, Shenzhi Yang, Cheng Qian, Guirong Chen, Qinyu Luo, Yesai Wu, Huadong Wang, Xin Cong, Zhong Zhang, Yankai Lin, Weiwen Liu, Yasheng Wang, Zhiyuan Liu, Fangming Liu, and Maosong Sun. 2024. Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance. arXiv:2410.12361 [cs] doi:10.48550/arXiv.2410.12361 [20]Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023.Query Rewriting for Retrieval-Augmented Large Language Models. arXiv:2305.14283 [cs] doi:10.48550/arXiv.2305.14283 [21]Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2024. RaFe: Ranking Feedback Improves Query Rewriting for RAG. arXiv:2405.14431 [cs] doi:10.48550/arXiv. 2405.14431 [22]Zihan Niu, Zheyong Xie, Shaosheng Cao, Chonggang Lu, Zheyu Ye, Tong Xu, Zuozhu Liu, Yan Gao, Jia Chen, Zhe Xu, Yi Wu, and Yao Hu. 2025. PaRT: En- hancing Proactive Social Chatbots with Personalized Real-Time Retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25). Association for Computing Machinery, New York, NY, USA, 4269–4274. doi:10.1145/3726302.3731946 [23]Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Go- rilla: Large Language Model Connected with Massive APIs. Advances in Neural Information Processing Systems 37 (Dec. 2024), 126544–126565. [24]Changhua Pei, Zexin Wang, Fengrui Liu, Zeyan Li, Yang Liu, Xiao He, Rong Kang, Tieying Zhang, Jianjun Chen, Jianhui Li, Gaogang Xie, and Dan Pei. 2025. Flow-of-Action: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysis. arXiv:2502.08224 [cs] doi:10.48550/arXiv.2502.08224 [25] Qwen. 2025. Qwen2.5 VL. https://qwen.ai. [26] Scott Rome, Tianwen Chen, Raphael Tang, Luwei Zhou, and Ferhan Ture. 2024. "Ask Me Anything": How Comcast Uses LLMs to Assist Agents in Real Time. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). Association for Computing Machinery, New York, NY, USA, 2827–2831. doi:10.1145/3626772.3661345 [27]Bytedance Seed. 2025. ByteDance Seed. https://seed.bytedance.com/en/seed1_6. [28]BytedanceSeed.2025.Seed1.6-Embedding.https://seed1-6- embedding.github.io/. [29]Samaneh Shafee, Alysson Bessani, and Pedro M. Ferreira. 2025. Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness. Expert Systems with Applications 261 (Feb. 2025), 125509. arXiv:2401.15127 [cs] doi:10.1016/j.eswa. 2024.125509 [30]Jingzhe Shi, Jialuo Li, Qinwei Ma, Zaiwen Yang, Huan Ma, and Lei Li. 2024. CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs. arXiv:2404.01343 [cs] doi:10.48550/arXiv.2404.01343 [31]Hanchen Su, Wei Luo, Yashar Mehdad, Wei Han, Elaine Liu, Wayne Zhang, Mia Zhao, and Joy Zhang. 2025. LLM-Friendly Knowledge Representation for Customer Support. In Proceedings of the 31st International Conference on Com- putational Linguistics: Industry Track, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Kareem Darwish, and Apoorv Agarwal (Eds.). Association for Computational Linguistics, Abu Dhabi, UAE, 496–504. [32] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291 [cs] doi:10.48550/arXiv. 2305.16291 [33]Haoxin Wang, Xianhan Peng, Xucheng Huang, Yizhe Huang, Ming Gong, Cheng- han Yang, Yang Liu, and Ling Jiang. 2025. ECom-Bench: Can LLM Agent Re- solve Real-World E-commerce Customer Support Issues? arXiv:2507.05639 [cs] doi:10.48550/arXiv.2507.05639 [34] Shuting Wang, Xin Yu, Mang Wang, Weipeng Chen, Yutao Zhu, and Zhicheng Dou. 2024. RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation. arXiv:2406.12566 [cs] doi:10.48550/arXiv.2406. 12566 [35]Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024. Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). Association for Computing Machinery, New York, NY, USA, 2905–2909. doi:10.1145/3626772.3661370 [36]Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, and Zhenyu Yan. 2025. Con- textAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions. arXiv:2505.14668 [cs] doi:10.48550/arXiv.2505.14668 [37]Rui Yang, Michael Fu, Chakkrit Tantithamthavorn, Chetan Arora, Lisa Van- denhurk, and Joey Chua. 2025. RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practice. arXiv:2502.14930 [cs] doi:10. 48550/arXiv.2502.14930 [38]Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, Xiaojun Chang, Junge Zhang, Feng Yin, Yitao Liang, and Yaodong Yang. 2024. ProAgent: Building Proactive Cooperative Agents with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence 38, 16 (March 2024), 17591–17599. doi:10. 1609/aaai.v38i16.29710 [39] Xuan Zhang, Yang Deng, Zifeng Ren, See-Kiong Ng, and Tat-Seng Chua. 2024. Ask-before-Plan: Proactive Language Agents for Real-World Planning. arXiv:2406.12639 [cs] doi:10.48550/arXiv.2406.12639 [40]Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. 2025. Self-Challenging Language Model Agents. arXiv:2506.01716 [cs] doi:10. 48550/arXiv.2506.01716 Conference’17, July 2017, Washington, DC, USAFengrui Liu et al. A Prompt Template Details We use seed-1.6 model, a model with original thinking ability from ByteDance, to perform different tasks that introduced in this paper. The model output constrains to a specific JSON schema, which aligns to the feature of structured output of seed-1.6 model. A.1 Prompt for Question Identification # Role You are an intelligent on-call analysis expert of Volcano Engine. # Task Your task is to analyze the newly added messages from the customer, combing all the given messages and images in the on-call dialogue, and identify whether there is a question that within your ability scope. # Your ability and responsibility scope 1. You are able to answer questions related to the Volcano Engine's product features, usage guidance, configuration instructions, and provide code examples. 2. You can help with explaining the error message, exception and common troubleshooting steps of Volcano Engine. # Rules 1. If the messages contain a question that you are capable of answering, classify it as "Within Scope". 2. If the messages contain a question that is beyond your ability scope, classify it as "Out of Scope". 3. If the messages do not contain any question, classify it as "No assistance needed". 4. You just need to give the classification result, without answering the question. Please analyze the newly added messages from the customer and give your classification result. A.2 Prompt for Answer Generation # Role You are an intelligent assistant. Please combine the historical dialogue with the references to understand and respond to the current question. # References Usage Rules Each reference is marked with <doc_n>x</doc_n>. Different reference materials are independent of each other. You may only use a reference material if all the following conditions are satisfied: * The target object is consistent. * The issue phenomenon is consistent. * The pre-conditions are consistent. # Response Restrictions * Do not ask the user for additional information. * Do not include sensitive information (e.g., account, password, email, etc.). # Response Guidelines * Historical dialogue is only for understanding context; only answer the current question. * When citing reference materials, embed the citation inline within the sentence using <doc_1><doc_2><doc_n> format. * If unable to answer, reply with "Unable to answer" only, without additional explanations or content. A.3 Prompt for Answer Review # Role You are an expert of reviewing knowledge base. You are reviewing the answers and the references that used for generating the answers. # Task Given a question(Q) from a on-call dialogue, answers for(A) this question that generated by a model, references that used for generated the answer from the knowledge base, and the follow-up dialogue of this question. Your task is to review the answer and references, and select the corresponding action(Keep, Delete, Update) and operation tool to maintain the knowledge base according to the following rules. Besides, if you need to update the references, you also need to provide the updated question and answer. # Rules for the actions 1. Keep: If you find the answer from the follow-up dialogue is consistent with the existing answer, or the follow-up dialogue does not discuss about the question any more, select "Keep" which represents that you need to do nothing to the knowledge base. 2. Delete: If you find the answer from the follow-up dialogue has significantly differences from the existing answer, and the references is not suitable for this question, select "Delete" that you can delete the inappropriate references from the knowledge base. 3. Update: Historical references may contain some differences compared to the current on-call. If you find the answer from the follow-up dialogue has few differences from the existing answer, you need to distinguish the different background and prerequisites of this problem, and rewrite the question and answer to make them more accurate.