Paper deep dive
DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution
Xin Shen, Zhishu Jiang, Jiaye Yang, Haibo Liu, Yichen Wan, Jiarui Zhang, Tingzhi Dai, Luodong Xu, Shuchen Wu, Guanqiang QI, Chenxi Miao, Jiahui Liang, Yang Li, Weikang Li, Deguo Xia, Jizhou Huang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 12:35:00 PM
Summary
The paper introduces DuCCAE, a hybrid conversational engine deployed in Baidu Search that decouples real-time response generation from asynchronous agentic execution to balance responsiveness and long-horizon task capability. It utilizes five subsystems (Info, Conversation, Collaboration, Augmentation, Evolution) and a shared state mechanism to synchronize fast-track lightweight interactions with slow-track complex multi-agent planning, resulting in improved user retention and task completion rates.
Entities (12)
Relation Signals (11)
DuCCAE â deployedon â Baidu Search
confidence 95% ¡ DuCCAE ... deployed within Baidu Search
DuCCAE â consistsof â Info System
confidence 92% ¡ The system orchestrates five subsystems-Info, Conversation, Collaboration, Augmentation, and Evolution
DuCCAE â consistsof â Conversation System
confidence 92% ¡ The system orchestrates five subsystems-Info, Conversation, Collaboration, Augmentation, and Evolution
DuCCAE â consistsof â Collaboration System
confidence 92% ¡ The system orchestrates five subsystems-Info, Conversation, Collaboration, Augmentation, and Evolution
DuCCAE â consistsof â Augmentation System
confidence 92% ¡ The system orchestrates five subsystems-Info, Conversation, Collaboration, Augmentation, and Evolution
DuCCAE â consistsof â Evolution System
confidence 92% ¡ The system orchestrates five subsystems-Info, Conversation, Collaboration, Augmentation, and Evolution
DuCCAE â achievesmetric â Day-7 User Retention
confidence 90% ¡ tripling of Day-7 user retention to 34.2%
DuCCAE â achievesmetric â Complex Task Completion Rate
confidence 90% ¡ surge in the complex task completion rate to 65.2%
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Immersive conversational systems in production face a persistent trade-off between responsiveness and long-horizon task capability. Real-time interaction is achievable for lightweight turns, but requests involving planning and tool invocation (e.g., search and media generation) produce heavy-tail execution latency that degrades turn-taking, persona consistency, and user trust. To address this challenge, we propose DuCCAE (Conversation while Collaboration with Augmentation and Evolution), a hybrid engine for immersive conversation deployed within Baidu Search, serving millions of users. DuCCAE decouples real-time response generation from asynchronous agentic execution and synchronizes them via a shared state that maintains session context and execution traces, enabling asynchronous results to be integrated back into the ongoing dialogue. The system orchestrates five subsystems-Info, Conversation, Collaboration, Augmentation, and Evolution-to support multi-agent collaboration and continuous improvement. We evaluate DuCCAE through a comprehensive framework that combines offline benchmarking on the Du-Interact dataset and large-scale production evaluation within Baidu Search. Experimental results demonstrate that DuCCAE outperforms strong baselines in agentic execution reliability and dialogue quality while reducing latency to fit strict real-time budgets. Crucially, deployment metrics since June 2025 confirm substantial real-world effectiveness, evidenced by a tripling of Day-7 user retention to 34.2% and a surge in the complex task completion rate to 65.2%. Our hybrid architecture successfully preserves conversational continuity while enabling reliable agentic execution, offering practical guidelines for deploying scalable agentic systems in industrial settings.
Tags
Links
- Source: https://arxiv.org/abs/2603.19248v1
- Canonical: https://arxiv.org/abs/2603.19248v1
Trouble viewing inline? Open PDF directly â
Full Text
79,600 characters extracted from source content.
Expand or collapse full text
DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution Xin Shen shenxin02@baidu.com Baidu Inc Beijing, China Zhishu Jiang jiangzhishu@baidu.com Baidu Inc Beijing, China Jiaye Yang yangjiaye01@baidu.com Baidu Inc Beijing, China Haibo Liu liuhaibo05@baidu.com Baidu Inc Beijing, China Yichen Wan wanyichen@baidu.com Baidu Inc Beijing, China Jiarui Zhang zhangjiarui05@baidu.com Baidu Inc Beijing, China Tingzhi Dai daitingzhi@baidu.com Baidu Inc Beijing, China Luodong Xu xuluodong@baidu.com Baidu Inc Beijing, China Shuchen Wu wushuchen@baidu.com Baidu Inc Beijing, China Guanqiang Qi qiguanqiang@baidu.com Baidu Inc Beijing, China Chenxi Miao miaochenxi@baidu.com Baidu Inc Beijing, China Jiahui Liang liangjiahui03@baidu.com Baidu Inc Beijing, China Weikang Li â wavejkd@pku.edu.cn Baidu Inc Beijing, China Yang Li liyang164@baidu.com Baidu Inc Beijing, China Deguo Xia xiadeguo@baidu.com Baidu Inc Beijing, China Jizhou Huang ââ huangjizhou01@baidu.com Baidu Inc Beijing, China Abstract Immersive conversational systems in production face a persistent trade-off between responsiveness and long-horizon task capabil- ity. Real-time interaction is achievable for lightweight turns, but requests involving planning and tool invocation (e.g., search and media generation) produce heavy-tail execution latency that de- grades turn-taking, persona consistency, and user trust. To ad- dress this challenge, we propose DuCCAE (Conversation while Collaboration withAugmentation andEvolution), a hybrid engine for immersive conversation deployed within Baidu Search, serving millions of users. DuCCAE decouples real-time response generation from asynchronous agentic execution and synchronizes them via a shared state that maintains session context and execution traces, enabling asynchronous results to be integrated back into the ongo- ing dialogue. The system orchestrates five subsystemsâInfo, Con- versation, Collaboration, Augmentation, and Evolutionâto support multi-agent collaboration and continuous improvement. We evalu- ate DuCCAE through a comprehensive framework that combines offline benchmarking on the Du-Interact dataset and large-scale production evaluation within Baidu Search. Experimental results â Corresponding authors: Weikang Li and Jizhou Huang. â Project lead: Jizhou Huang. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym âX, Woodstock, NY Š 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X demonstrate that DuCCAE outperforms strong baselines in agentic execution reliability and dialogue quality while reducing latency to fit strict real-time budgets. Crucially, deployment metrics since June 2025 confirm substantial real-world effectiveness, evidenced by a tripling of Day-7 user retention to 34.2% and a surge in the complex task completion rate to 65.2%. Our hybrid architecture successfully preserves conversational continuity while enabling re- liable agentic execution, offering practical guidelines for deploying scalable agentic systems in industrial settings. CCS Concepts ⢠Computer systems organizationâReal-time system archi- tecture;⢠Computing methodologiesâDiscourse, dialogue and pragmatics. Keywords Immersive Conversational Systems, Multi-Agent Collaboration, Multimodal Interaction, Self-Evolution, Baidu Search ACM Reference Format: Xin Shen, Zhishu Jiang, Jiaye Yang, Haibo Liu, Yichen Wan, Jiarui Zhang, Tingzhi Dai, Luodong Xu, Shuchen Wu, Guanqiang Qi, Chenxi Miao, Jiahui Liang, Weikang Li, Yang Li, Deguo Xia, and Jizhou Huang[1]. 2018. DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmenta- tion, and Evolution. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym âX). ACM, New York, NY, USA, 14 pages. https://doi.org/X.X 1 Introduction Immersive conversational interfaces are rapidly becoming the pri- mary entry point for diverse production domains, ranging from search engines to customer support and intelligent assistants [21,34, arXiv:2603.19248v1 [cs.CL] 25 Feb 2026 Conference acronym âX, June 03â05, 2018, Woodstock, NYXin Shen et al. SearchHistory Birthday GiftsGifts for HerComforting Friend Search What different roses mean ? SearchInterfaceInteractionChat History Hi! Are you looking to send flowers or learn about flower meanings? I can recommend something. Just asking casuallyâwhat do different rose colors mean? Whatâs suitable for a girlâs birthday? Red roses mean passionate love; pink roses mean gentle affection/liking; ... For a birthday with a sweet vibe, pink roses are a great choice. My friend has been having relationship issues latelyâhow should I comfort her? Thereâs a relationship expert online right nowâletâs invite them to join the call. Multi-Agent Interaction What should I say to my friend to comfort her without sounding dismissive or giving unsolicited advice? Figure 1: DuCCAE interface and interaction flow in production. Starting from a Baidu Search entry, the system maintains persona- consistent chat responses and supports escalation to real-time calling; complex requests trigger asynchronous collaboration and tool augmentation while preserving conversational continuity. 41 ,47,52]. Users in these environments demand a seamless blend of low-latency response and complex task execution [10,15,22]. Such interactions frequently involve intricate reasoning and tool usage grounded in multimodal signals (e.g., speech, video, and text) [13,35,38]. Processing these inputs under strict real-time budgets introduces a critical bottleneck, where the heavy-tail la- tency inherent in agentic planning breaks conversational flow and destabilizes persona consistency [15, 22, 50]. While recent agent and multi-agent paradigms have significantly improved capability via planning, routing, and tool invocation, they effectively exacerbate this latency bottleneck [22,33,46]. Long- horizon execution and tool latency are notoriously difficult to bound in production environments [15,48]. Consequently, âcapability-firstâ pipelines often achieve strong task performance yet degrade in- teraction quality due to silence, interruptions, and inconsistent persona, whereas âresponsiveness-firstâ conversational setups pre- serve fluency but struggle with reliable tool use and long-horizon task completion [13,30,32]. Bridging responsiveness and capa- bility in a single production system therefore requires a serving architecture that can keep the dialogue responsive while executing complex tasks asynchronously, and can integrate intermediate and final outcomes back into the same conversational context [26]. To address this challenge, we present DuCCAE, a hybrid engine for immersive conversation developed and deployed at Baidu [4,5]. As illustrated in Figure 1, DuCCAE instantiates this paradigm within the Baidu Search interface via an interactive digital human [35,40]. Crucially, the system utilizes a decoupled architecture that sepa- rates real-time video-based rendering from asynchronous agentic execution [12,49,55]. To achieve this, DuCCAE orchestrates five tightly integrated subsystemsâInfo, Conversation, Collaboration, Augmentation, and Evolutionâto manage the complex interplay be- tween multimodal perception, real-time response, and long-horizon tool use [30]. This design allows the digital human to sustain im- mediate, persona-consistent responsiveness during simple turns, while long-horizon tasks are offloaded to a background multi-agent collaboration layer [30,42]. A unified shared state bridges these two paths by retaining session context and execution traces, en- abling tool outputs and agent deliverables to be integrated back into the live video interaction seamlessly without breaking immer- sion [9,20,54]. Moreover, DuCCAE operates as an evolving service engine, where real interactions continuously drive automated eval- uation and post-training updates [14, 56]. To validate the effectiveness of this hybrid engine, we evaluate DuCCAE in a production setting using both online experiments and post-launch metrics. The key contributions to both the research and industrial communities are as follows: â˘We propose a decoupled framework that resolves the trade- off between responsiveness and capability. By synchronizing real-time and asynchronous streams, we ensure sub-second latency while supporting complex tool use. â˘We demonstrate that our evolved model outperforms zero- shot baselines with significantly larger parameters in both dispatch precision and task success rate. â˘Deployed to millions of users since June 2025, DuCCAE has driven substantial gains, achieving a 3Ăincrease in Day-7 User Retention (34.2%) and a Complex Task Completion Rate of 65.2%. 2 System Overview Figure 2 illustrates the architecture of DuCCAE, formulated as a hybrid orchestration framework designed to bridge the gap be- tween real-time responsiveness and agentic capability in immersive settings [1,9,46]. Unlike monolithic pipelines that block on tool execution, DuCCAE implements a Latency-Decoupled Architec- ture [31]. The core design philosophy is to separate the interaction loop (which demands strictly bounded latency for immersion) from the execution loop (which involves heavy-tail reasoning for com- plex tasks), utilizing a Unified Shared State as the synchronization Baidu DuCCAE SystemConference acronym âX, June 03â05, 2018, Woodstock, NY Info System PersonaKnowledgeOpinion Agent Memory User History User Profile Session User Memory User Memory Agent Memory Multimodal Processing Conversation System Query Understanding Responder Collaboration System Team Dispatcher ... doctorchefteachercoachlawyerengineer Augmentation System User Memory Agent Memory Resource pictures videos hot news ... Execution Unit MCP/A2A Tools Subagents ... Task Router Task 1Agent 1 Task 2Agent 2 Task 3Agent 3 Task NAgent N Task Context Task Execute Execution Result Execution Results Vision-Language Model Large Language Model User Interaction Video CallVoice CallTextInput Agentic Judge and RewardPost-TrainingContext Compression Evolution System Interaction Episode ... Judge LLM Judge Human Judge ... Conversation Collaboration Augmentation Info Systems Reward Model Filtered Data SFT / RL-FT Interaction Episode Query Q1 For this Q1: ... For this Q1: -call tool: search(...) -call tool: api(...) -... âRespond for Query 1 (long context) Summary User MemoryAgent Memory Update Memory Follow-up Q2 For this Q2: ... For this Q2: -call tool: search(...) -call tool: api(...) -... âRespond for Query 2 (long context) Summary User Memory Agent Memory Update Memory Interaction Episode FilteredData Agent2 Planner Executor Generator Evolving Updating Agent1 Agent3 Figure 2: DuCCAE: an evolving agentic service engine for immersive conversational interaction. Info System converts multimodal signals into policy-aware context and manages memory. Conversation System acts as a low-latency gatekeeper for intent routing and renders persona-consistent responses. Collaboration System supports multi-agent execution for long-horizon tasks via planning and tool use. Augmentation System empowers agents with external tools, retrieval resources, and execution protocols. Evolution System drives continuous improvement through episode-based judging and post-training. barrier [6,58]. This architecture instantiates the proposed decou- pled paradigm through five tightly integrated subsystems: Info, Conversation, Collaboration, Augmentation, and Evolution. 2.1 Dual-Track Dataflow and Synchronization The runtime behavior of DuCCAE is governed by a Dual-Track Execution Mechanism [29,44] that dynamically routes multi- modal streams based on computational complexity, as illustrated in Figure 3. Given an interaction episode, the system employs a semantic routing strategy to direct the execution flow through two coordinated paths: a Fast Track for immediate response and a Slow Track for asynchronous reasoning, which are eventually reunited via Event-Driven Synchronization. The Query Understanding module (detailed in Section 2.2.2) executes this critical routing selection by evaluating the intent complexity [43]. The Fast Track for Real-Time Interaction. The first path is the Fast Track, dedicated to maintaining conversational immersion by bypassing heavy reasoning modules. It triggers the Conversation System to directly resolve lightweight queries by leveraging pre- loaded agent knowledge and user context [57]. Such a streamlined process guarantees that persona-consistent feedback, including non-verbal cues or memory-grounded replies, is generated within a strict Time-to-First-Token (TTFT) budget of under 500ms [23]. The Slow Track for Asynchronous Reasoning. The second path is the Slow Track, which handles complex intents by offloading planning and tool invocation to the Collaboration System [11,39]. Conference acronym âX, June 03â05, 2018, Woodstock, NYXin Shen et al. Operating in the background, it executes long-horizon tasks without blocking the real-time interaction loop, thereby allowing the digital human [17,18,53] to maintain engagement while complex work proceeds asynchronously. Event-Driven Synchronization. These two tracks are synchronized via an Event-Driven Integration mechanism [6,58]. When the Slow Track produces intermediate artifacts or final results, it emits a state-update event. Upon capturing these signals, the Shared State prompts the real-time path to seamlessly incorporate the new in- formation into the ongoing video stream. Consequently, the user perceives a single and coherent dialogue flow rather than disjoint system outputs. 2.2 Subsystem Overview Having established the dual-track orchestration mechanism, we now detail the design rationale and specific responsibilities of each constituent subsystem, following the information flow from per- ception to evolution. 2.2.1 Info System: Multimodal Perception and State Construction. The Info System serves as the foundational perception layer, tasked with converting raw, unstructured interaction signals into a structured and policy-aware state space. To balance the criti- cal trade-off between perception depth and real-time latency, the architecture aligns all modalities into a unified textual representa- tion. Such a design enables the system to leverage powerful Large Language Models (LLMs) [3â5] for reasoning without incurring the computational overhead typically associated with processing high-dimensional video streams end-to-end. The system processes heterogeneous inputs through special- ized pipelines to extract textual semantic cues. For audio streams, a streaming Automatic Speech Recognition (ASR) [2,19] engine converts voice input into text with millisecond-level latency. Con- currently, handling video streams requires a distinct strategy to mitigate the heavy-tail latency inherent in large Vision-Language Models [4,5]. The system therefore employs a lightweight VLM dedicated to efficient semantic extraction. Instead of dense frame- by-frame inference, this module generates concise captions describ- ing key visual elements, including user actions, detected objects, and environmental tags. These structured text descriptions are then aligned with the ASR output to form a cohesive multimodal context. Beyond perception, the Info System orchestrates a dual-memory mechanism (as depicted in Figure 2) that strictly decouples user personalization from the agent execution context [37,57]. User Memory aggregates persistent signals to construct a stable user model, comprising static attributes in the User Profile, retrieval- augmented logs in the User History, and the immediate turn-taking context in the Session State. Fusion of these signals ensures that responses remain grounded in the specific persona and constraints of the user. Distinct from these personal attributes, Agent Memory functions as the unified cognitive storage for the digital human. As illustrated in Figure 3, it houses the agentâs static profiles (Persona and Knowledge Base) to guide consistent role-play, while primarily serving as the working memory to record the System-of-Thought (SoT) trajectory, which includes the current plan generated by the Collaboration System and intermediate tool traces [39,57]. Isolating the execution state allows the agent to manage complex, ephemeral task data without polluting the long-term user profile. Following this stratification, the Responder module orchestrates the output delivery. For immediate queries, it directly synthesizes the final response. Crucially, for complex requests requiring long- horizon reasoning, the Responder employs a âbridge-and-callbackâ strategy. Upon routing the request, it immediately generates a se- mantic bridging responseâsuch as the acknowledgement âIâve re- ceived your request...â shown in the Fast Track of Figure 3âto pre- vent âdead airâ and maintain conversational momentum [31]. Once the asynchronous workflow completes, the results are seamlessly reintegrated into the ongoing dialogue stream. 2.2.2 Conversation System: Latency-Aware Interaction and Routing. The Conversation System operates as the real-time interaction hub, strictly bounded by a latency budget to ensure immersive responsiveness. Functioning as the gatekeeper for the dual-track mechanism, it executes two critical processes: intent stratification via Query Understanding and stream-based rendering via the Re- sponder. The Query Understanding module acts as the decision engine, parsing the normalized Request Object to map user inputs onto a three-tiered complexity hierarchy [43,44]. This classification explicitly dictates the subsequent execution path: (1)Lightweight Queries (Tier-1): Interactions that can be resolved immediately using pre-loaded context and general knowledge (e.g., chit-chat, greetings, or memory-based QA). These are routed exclusively to the Fast Track for sub-second response. (2)Deterministic Tool Intents (Tier-2): Requests where the user explicitly seeks external information requiring discrete API calls (e.g., querying weather, stock prices, or calendar en- tries). These trigger specific tools within the Augmentation System while maintaining a simple execution flow. (3)Complex Domain Requests (Tier-3): High-order tasks necessitating intricate reasoning, multi-step planning, or pro- fessional expertise (e.g., medical consultation, travel itinerary planning, or legal advice). These leverage the Team Dis- patcher to recruit specialized agents within the Collaboration System for asynchronous collaboration. Following this stratification, the Responder module generates the appropriate user-facing output. For Tier-1 queries, it directly synthesizes a final response grounded in the active persona. Con- versely, for Tier-2 and Tier-3 requests that trigger asynchronous workflows, the Responder generates a semantic bridging response, such as a clarification question or a verbal acknowledgement (âI will check the latest reports for you...â), to maintain the conversational floor and mask the execution latency. 2.2.3 Collaboration System: Asynchronous Multi-Agent Execution. The Collaboration System serves as the execution engine for the âSlow Trackâ, handling Tier-2 and Tier-3 requests that exceed the ca- pabilities of immediate retrieval. To ensure reliability and efficiency under heavy-tail latency, the system implements a standardized PlanâExecuteâGenerate workflow [11, 39]. The process initiates with the Team Dispatcher, which func- tions as a semantic routing layer. The system maintains a library of Baidu DuCCAE SystemConference acronym âX, June 03â05, 2018, Woodstock, NY Iâm exhausted fromtodayâs match... Since New Year's is coming, can you plan a relaxing trip for me? Did you just finish a basketball match? You must be tired! I've received your request and am planning a recovery trip now... User Memory [Hobby] loves basketball [Diet] dislikes raw fish [Location] Beijing ... Agent Memory [Persona] Empathetic & Warm [Knowledge] Travel Planning, Geography, Recovery ... Fast Track Log Slow Track Log Yeah, exactly! It was a tough game, but I scored 20 points! Wow, 20 points! That's an MVP performance! Response for : âExhausted todayâs match â Response for : â New Year Travel Plan â Detected âExhaustedâRetrieve Memory [Hobby] Infer "Basketballâ Planner & Executor Activity: ArashiyamaOnsen (Hot Spring) Flight: Direct (3h 10m) Dinner: Premium Wagyu Yakiniku Chat HistoryDual-Track Execution To celebrate your victory and help you recover, I've chosen Japan (it's close to Beijing). I planned a Kyoto Onsen trip to soothe your muscles. And since you don't eat raw fish, I booked a celebratory Wagyu Beef dinner instead! Step 1 (Serial ¡ Decision) Step 2 (Parallel ¡ Efficiency) Step3 (Serial ¡ Constraint Fusion) Tool: Reasoning: Nearby destination suitable for short- term recovery trip. Result: Japan (Kyoto) â Branch A (Flight): Tool: Found: CA925 (Direct, 3h 10m) Branch B (Activity): Tool: Found: ArashiyamaOnsen (Hot Spring) Dest.Recommend(Origin="Beijing", Tag="Relaxing") Flight.Check(Beijing -> Osaka) Search.Activity(Loc="Kyoto",Tag="Relaxing) Dependency: Wait for Branch B (Location Confirmed). Constraint Trigger: User_Mem: Dislikes Raw Fish Action: ¡Recommend Sushi ¡Select Premium Wagyu Yakiniku Figure 3: Runtime Execution Dataflow of a Mixed-Intent Interaction Episode. Processing the input âExhausted... Plan a tripâ, the Fast Track (top) utilizes User Memory (âHobby: Basketballâ) for an immediate empathetic inquiry. Simultaneously, the Slow Track (bottom) decomposes the travel request, demonstrating: (1) Parallel Execution of independent flight and activity searches to optimize latency; and (2) Constraint Fusion, which filters dining options via the memory constraint (âDislikes Raw Fishâ) to select the compatible âWagyu Beefâ. specialized agents (e.g., medical experts, legal advisors, or data ana- lysts), each characterized by a specific model profile and knowledge base. Upon receiving a complex request, the Dispatcher employs a semantic matching mechanism to align the queryâs intent embed- ding with the most suitable agent profile. This dynamic recruitment ensures that specialized domain knowledge is applied precisely to the corresponding problem space. Once a specialized agent is instantiated, the workflow enters the Planner phase, where the primary objective is latency opti- mization through structural decomposition. The Planner analyzes the request to construct a task dependency graph, distinguishing between serial tasks and parallel tasks [6,36]. As visually demon- strated in the âSlow Track Logâ of Figure 3, the Planner schedules the flight check and activity search concurrently (Step 2) to opti- mize efficiency, while strictly enforcing sequential dependencies for dining reservations (Step 3). By identifying non-blocking sub- tasks, the Planner enables concurrent tool invocation, significantly accelerating the overall execution timeline. Following the plan, the Executor carries out the concrete ac- tions. It adheres to an iterative state update protocol: as each step is executed, the intermediate results (e.g., API returns or sub-agent replies) are written back into the shared Task Context. This mecha- nism is critical for serial tasks, as it ensures that subsequent steps have access to the data generated by their predecessors, maintaining logical continuity throughout the problem-solving chain. The workflow concludes with the Generator, which synthesizes the aggregated execution traces into a coherent user-facing deliver- able. This module is model-agnostic and modality-aware: it selects a Large Language Model (LLM) [4,5,51] for textual summaries or a Vision-Language Model (VLM) [3â5] when visual synthesis is required. The final precise output is then synchronized back to the shared state, ready to be presented by the digital human [17,18,53]. 2.2.4 Augmentation System: Retrieval, Protocols, and Tool Ecosys- tem. The Augmentation System extends the native capabilities of the core models by bridging them with external knowledge bases and executable environments. Rather than a static repository, it operates as a dynamic service layer that grounds model generation in reality and standardizes complex tool interactions. To ensure response fidelity and accelerate inference, the system implements a Retrieval-Augmented Generation (RAG) [7,8,45] mechanism over a heterogeneous resource layer. This layer aggre- gates proprietary internal databases, real-time âhot newsâ, and the structured histories from both User and Agent Memory. By retriev- ing relevant context prior to generation, the system achieves two critical objectives: Consistency and Efficiency. Retrieving historical memory ensures that the agentâs behavior aligns strictly with the userâs persona and past interactions, while retrieving structured knowledge allows for faster, high-confidence responses without hal- lucinations. This mechanism not only grounds the agent in verified data but also injects diversity into the conversation by surfacing rich, domain-specific content. To manage the complexity of diverse extensions, the system abstracts all external interactions through a Unified Execution Unit interface, adhering to strict industry protocols. Specifically, we implement the Model Context Protocol (MCP) to standardize how context is passed between the LLM and external tools, ensuring the model accurately understands tool states [20]. Furthermore, for multi-agent handshakes, we utilize an Agent-to-Agent (A2A) protocol that defines clear contracts for task delegation and result return. This standardization allows different specialized agents to collaborate seamlessly without format mismatches, treating tools and sub-agents as interchangeable functional blocks. The ecosystem supports a wide array of capabilities, ranging from information retrieval (e.g., Search, Document Parsing) to cre- ative media synthesis (e.g., Image and Music Generation). Crucially, Conference acronym âX, June 03â05, 2018, Woodstock, NYXin Shen et al. Table 1: Statistics of Du-Interact-Evo (Training) and Du-Interact (Testing) across Evolutionary Stages. System Ver. Evolution Stage Training Source Dataset SizeMethod DuCCAE-V1Cold StartN/A0Zero-shot Prompting DuCCAE-V2Evolution IDu-Interact-Evo-V115,000SFT (on V1 logs) DuCCAE-V3Evolution IIDu-Interact-Evo-V250,000SFT + RL (on V2 logs) BenchmarkEvaluationDu-Interact5,000Fixed Golden Set the deployment of Sub-Agents is governed by a Robust Orchestra- tion Engine. When executing tasksâparticularly during the parallel processing paths defined by the Plannerâthis engine enforces strict concurrency controls and validity checks. It prevents execution conflicts and manages failure states, ensuring that a stall in one sub- agent does not deadlock the entire workflow. This guarantees that complex, multi-tool chains are executed atomically and reliably. 2.2.5 Evolution System: Data-Driven Feedback and Adaptive Com- pression. The Evolution System functions as the engine for con- tinuous improvement, closing the loop between deployment and development [14,56]. Instead of static updates, it employs a peri- odic Data Flywheel mechanism that processes interaction episodes to refine both the systemâs efficiency and its intelligence. To construct high-quality training datasets, raw interaction logs first undergo Automated Assessment [16,25]. Model-based judges evaluate episodes based on three pragmatic criteria: (1) Next-Turn Engagement, measuring whether the systemâs response success- fully elicited a meaningful follow-up from the user (a key proxy for immersion); (2) Instruction Compliance, checking if tool in- vocations logically matched the userâs intent; and (3) Sentiment Alignment, analyzing the emotional valence of the userâs reaction. This automated phase produces a large-scale âSilver Datasetâ for Supervised Fine-Tuning (SFT). Subsequently, a subset is sampled for Human Verification, yielding a high-precision âGold Datasetâ reserved for training Reward Models and Reinforcement Learning (RL) benchmarks. Our system implements a two-stage Compression Strategy to manage the life-cycle of information, preventing context overflow while retaining critical signals. â˘Runtime Context Folding: To mitigate the heavy token overhead caused by verbose tool outputs, we adopt the Context- Folding mechanism inspired by [24,42]. During complex execution chains (in the Collaboration System), the agent isolates intermediate stepsâsuch as error traces or massive JSON payloadsâinto temporary branches. Upon task com- pletion, these branches are âfoldedâ into concise semantic summaries before being merged back into the main context. This drastically reduces context occupancy and latency for subsequent turns. â˘Episodic Memory Distillation: At the session level, raw conversation logs are too noisy for long-term storage. The system applies an abstraction policy to distill completed sessions into structured Knowledge Nuggets [37,57] (e.g., âUser prefers visual data over textâ or âUser is a vegetarianâ). These distilled artifacts are then committed to the long-term User Memory and Agent Memory, ensuring that the memory module grows in density rather than just volume. Finally, the processed assets drive a Modular Post-Training phase. The filtered âSilverâ and âGoldâ datasets are stratified to target specific subsystems, such as conversational pairs enrich the Conver- sation Systemâs persona adaptability, while folded execution traces refine the Collaboration Systemâs planning capabilities. By itera- tively retraining on these filtered experiences, DuCCAE evolves from a static system into an adaptive agent that aligns increasingly closer to user needs over time. 3 Experiments To rigorously evaluate the effectiveness of DuCCAE in a realistic in- dustrial setting, we conduct comprehensive evaluations comparing our evolved system against a wide spectrum of state-of-the-art open- source baselines. All experiments are conducted on Du-Interact, a standardized human-curated benchmark derived from the produc- tion environment of Baidu Search. 3.1 Datasets and Protocols Our data strategy distinguishes between the dynamic training stream and the static evaluation benchmark to ensure rigorous measurement of evolutionary progress. 3.1.1 Du-Interact-Evo: The Evolutionary Training Stream. The train- ing data for DuCCAE is not static; it is generated dynamically via the proposed Evolution System. We term this continuously grow- ing dataset Du-Interact-Evo. As shown in Table 1, the system underwent three evolutionary stages: Stage I (Cold Start): At the inception (V1), no domain-specific training data was available. The system relied on zero-shot prompt- ing of the foundation model using expert-defined rules. Stage I (Evolution I): We collected 3 months of interaction logs from the V1 system. These logs were filtered by the âData Flywheelâ pipeline (Automated Judging + Human Verification) to construct Du-Interact-Evo-V1, a high-quality Supervised Fine-Tuning (SFT) dataset containing 15,000 episodes. Stage I (Evolution I): Building on V2, we expanded the collec- tion to 50K episodes (Evo-V2). This stage introduced Reinforcement Learning (RL) signals, prioritizing sessions where users successfully completed complex tasks with high sentiment scores. 3.1.2 Du-Interact: The Golden Test Benchmark. To monitor the systemâs trajectory fairly, we construct Du-Interact, a fixed Golden Test Set containing 5,000 multi-turn interaction sessions sampled strictly from the V2 phase hard-negatives. Crucially, each session in Du-Interact is annotated by human experts with Ground Truth (GT) labels for: (1) correct intent routing logic, (2) optimal tool selection and parameter extraction, and (3) ideal response content. This ensures that V1, V2, and V3 are evaluated against the same high standard. Baidu DuCCAE SystemConference acronym âX, June 03â05, 2018, Woodstock, NY Table 2: Main Results on the Du-Interact Golden Benchmark. Backbone ModelSetup Task ExecutionDialogue QualityEfficiency Dispatch (í ííí í ) Success Rate (SR)Fidelity (%) Persona (1-5) Empathy (1-5)Avg. Latency Edge Class (3B) Qwen2.5-3B-InstructZero-shot52.4%28.5%45.2%2.52.41,250 ms Llama-3.2-3B-InstructZero-shot 51.8%26.2%44.1%2.42.31,280 ms Server Class (7B-11B) Qwen2.5-7B-InstructZero-shot61.5%37.2%54.8%2.82.81,480 ms Llama-3.2-11B-InstructZero-shot 58.2%35.8%53.5%2.72.71,520 ms Large Scale Class (30B-70B) Qwen2.5-32B-InstructZero-shot72.1%55.4%62.2%3.23.23,250 ms Llama-3.3-70B-InstructZero-shot75.5%61.1%64.4%3.43.35,800 ms DuCCAE (Ours) DuCCAE-V1Cold Start68.8%50.5%58.4%3.53.51,850 ms DuCCAE-V2Evolution I76.2%63.8%66.5%3.94.01,920 ms DuCCAE-V3Evolution I82.5%72.4%71.1%4.14.31,880 ms 3.2 Baselines and Setup 3.2.1 System Architecture. Our core system utilizes ERNIE-4.5- 21B-A3B [4] as the central reasoning backbone (Slow Track) and ERNIE-4.5-VL-28B-A3B [4] for visual perception. For fair compar- ison, all open-source baselines replace only the reasoning backbone while retaining the same visual front-end and the identical tool ecosystem [27, 28]. 3.2.2 Baseline Models. We categorize baselines into three groups to analyze the impact of model scale comprehensively: Edge Class (3B): We select Qwen2.5-3B-Instruct [51] and Llama-3.2- 3B-Instruct [28]. These models represent the ultra-low latency tier suitable for on-device deployment. Server Class (7B-11B): We select Qwen2.5-7B-Instruct [51] and Llama- 3.2-11B-Instruct [28]. These are the industry standards for cost- effective server-side inference. Large Scale Class (30B-70B): We select Qwen2.5-32B-Instruct [51] and Llama-3.3-70B-Instruct [28]. These serve as powerful âupper boundâ references for zero-shot reasoning capabilities. 3.3 Evaluation Metrics We employ a rigorous evaluation framework focusing on three crit- ical dimensions: Task Execution, Dialogue Quality, and Efficiency. (1) Task Execution (Hard Metrics). These metrics measure the systemâs functional reliability based on Human-Annotated Ground Truth (GT) in Du-Interact. Specifically, we report Dispatch Pre- cision (í ííí í ), which evaluates the accuracy of the Conversation System in routing user queries to the correct subsystem (Tier-1 Fast Track vs. Tier-2/3 Slow Track). Furthermore, we measure Suc- cess Rate (SR), defined as the percentage of sessions where the agent successfully completes the userâs intent. A session is marked as âSuccessâ only if the selected tools and extracted parameters perfectly match the GT. (2) Dialogue Quality (Hybrid Metrics). To assess response qual- ity, we utilize a hybrid approach combining objective GT match- ing and subjective LLM-as-a-Judge evaluation (powered by GPT- 4o) [16,25]. We calculate Response Fidelity, the percentage of re- sponses that accurately convey the key information points present in the GT response. Additionally, we assess Persona & Empathy on a 1â5 scale, evaluating whether the response maintains the digi- tal humanâs character and provides appropriate emotional support compared to the expert annotations. (3) Efficiency. We quantify system performance using Avg. La- tency, which represents the end-to-end duration from user input to final response rendering. For further details on the evaluation protocols and specific prompt configurations, please refer to Appendix Section D. 3.4 Experimental Results and Analysis Table 2 presents the holistic performance comparison on Du-Interact. The results reveal a clear hierarchy in capability and highlight the critical role of our evolutionary training pipeline. 3.4.1 Impact of Model Scale (Small vs. Large vs. DuCCAE-V1). The comparison across model scales offers significant insights into back- bone selection for industrial agents. While Edge and Server class models offer ultra-low latency (<1.5s), they fail to reliably handle the âThree-Tiered Complexityâ routing logic. For instance, Llama- 3.2-11B achieves only 35.8% Success Rate, often misinterpreting complex planning instructions. This confirms that<10B mod- els lack the reasoning depth for autonomous orchestration in our scenario. The Large Scale class (e.g., Llama-3.3-70B) demonstrates strong zero-shot reasoning (61.1% SR). However, their high latency (3.2s - 5.8s) makes them unsuitable for immersive real-time interac- tion, violating our strict response budget. Our backbone, ERNIE-4.5- 21B (DuCCAE-V1), strikes an optimal balance. In zero-shot settings, it outperforms the 8B class by a large margin while remaining significantly faster than the 70B model (1.8s vs 5.8s). 3.4.2 Impact of Evolutionary Data (V1 vs. V2/V3). The most com- pelling result is the performance leap achieved through training on Du-Interact-Evo, enabling a mid-sized model to outperform general-purpose giant models. Without domain adaptation, even the powerful Llama-3.3-70B is limited by the gap between general instructions and specific system protocols. DuCCAE-V2, fine-tuned on Du-Interact-Evo-V1, surges to a 63.8% Success Rate, surpass- ing the 70B baseline (61.1%). This demonstrates that system-specific alignment is more impactful than raw parameter scale. Conference acronym âX, June 03â05, 2018, Woodstock, NYXin Shen et al. V1 (Cold Start) V2 (Iter. I) V3 (Iter. I) 0 5 10 15 20 25 30 35 40 Day-7 Retention (%) 12.5 21.8 34.2 User Stickiness V1 (Cold Start) V2 (Iter. I) V3 (Iter. I) 0 20 40 60 80 Good Turn Rate (%) 52.4 65.8 75.6 Conversation Quality V1 (Cold Start) V2 (Iter. I) V3 (Iter. I) 0 2 4 6 8 10 12 14 Avg. Session Turns 4.2 7.8 12.5 Interaction Depth V1 (Cold Start) V2 (Iter. I) V3 (Iter. I) 0 10 20 30 40 50 60 70 Task Completion (%) 35.5 52.8 65.2 Agentic Capability Figure 4: Evolution of Key Business Metrics across System Iterations. Statistics are derived from large-scale online controlled experiments in Baidu Search. From V1 to V3, DuCCAE demonstrates consistent growth in: (a) User Stickiness; (b) Conversation Quality; (c) Interaction Depth; and (d) Agentic Capability. Table 3: Ablation on Visual Perception Strategies. Reported latency measures the visual perception overhead (pre-LLM). MethodParadigmResponse Fidelity Avg. Latency (%)â(ms)â Baseline-E2EVideo-Inâ Text-Out64.0%2,100 DuCCAE (Ours)Captionâ LLM71.1%480 Failure-Driven Evolution. Early iterations revealed three dom- inant failure patterns: intent mis-dispatch for cross-domain re- quests, parameter extraction errors in tool invocation, and long-tail latency in sequential planning. These cases were systematically harvested by the Evolution System and formed the core of Du- Interact-Evo, enabling targeted SFT/RL updates for the Collabo- ration and Conversation subsystems. Consequently, the improve- ments observed from V1 to V3 reflect not only algorithmic design but also a closed-loop learning process grounded in real user fail- ures. The V3 stage, utilizing the larger Du-Interact-Evo-V2 dataset with RL, pushes performance to robust production-grade (72.4% SR) and high Dialogue Quality (4.3/5.0). This validates that the Data Flywheel effectively captures and reinforces the âEmotional Digital Humanâ traits that generic models lack. Regarding efficiency, the slight latency increase in V2 reflects the higher computational cost of successfully executing complex planning tasks compared to V1âs early failures. However, V3 reduces latency back to 1,880 ms despite higher capability, validating the efficiency gains from our Runtime Context Folding and RL-driven path optimization. 3.4.3 Ablation Study: Visual Perception Strategy. To verify the ar- chitectural rationale of the Info System, we conduct an ablation study comparing our decoupled âVisual-to-Textâ strategy against a standard End-to-End (E2E) VLM paradigm [4]. As shown in Table 3, the E2E baseline suffers from a prohibitive latency of 2,100 ms due to dense video token processing. In contrast, our decoupled strat- egy reduces the visual perception latency by 77% to 480 ms. This sub-second speed is the decisive factor that enables the âFast Trackâ to function within the systemâs real-time constraints. While E2E models theoretically capture more visual details, they are prone to introducing irrelevant noise, which degrades the alignment with Ground Truth. By distilling visual signals into structured captions, our method filters out this noise, resulting in a higher Response Fidelity compared to the E2E baseline. 3.5 Online Commercial Impact Beyond offline benchmarks, the ultimate validation of DuCCAE lies in its scalable performance within the real-world production environment of Baidu Search. Figure 4 visualizes the evolutionary trajectory of four key business metrics across the systemâs three iterations. For detailed definitions and calculation protocols of these metrics, please refer to Appendix Section E. User Stickiness and Quality. As shown in Figure 4(a) and (b), the Day-7 Retention Rate nearly tripled from V1 (12.5%) to V3 (34.2%). This surge indicates that as the system evolved from a generic chatbot to an empathetic, personalized companion, users formed a stronger habit of returning. Concurrently, the Good Turn Rate (GTR) steadily improved to 75.6%, reflecting consistently high user satisfaction with single-turn responses. Immersion and Capability. Figure 4(c) highlights that the Average Session Turns increased significantly from 4.2 to 12.5. This 3Ăin- crease in session depth suggests that the âDual-Trackâ architecture successfully maintains conversational momentum, encouraging users to engage in longer, more immersive interactions. Most no- tably, Figure 4(d) confirms the effectiveness of our Collaboration System: the Complex Task Completion Rate jumped from 35.5% to 65.2%. This proves that DuCCAE has successfully transitioned from a conversational toy (V1) to a reliable service agent (V3) capable of handling intricate, long-horizon user requests. 4 Conclusion In this work, we presented DuCCAE, an industrial-grade hybrid engine that successfully reconciles the persistent trade-off between real-time immersion and long-horizon agentic reasoning. By im- plementing a Latency-Decoupled Architecture, the system secures sub-second responsiveness via a Fast Track while orchestrating asynchronous tool use via a Slow Track, synchronized seamlessly by a shared state. Beyond architecture, our results validate the effi- cacy of the Evolutionary Data Flywheel, demonstrating that domain- specific evolution allows cost-effective models to surpass signifi- cantly larger general-purpose baselines in complex task execution. Deployed at scale within Baidu Search, DuCCAE has proven its commercial viability, delivering substantial improvements in both long-term user retention and complex task completion. This work provides a proven blueprint for deploying scalable, empathetic, and capable agentic systems in high-traffic industrial environments. Baidu DuCCAE SystemConference acronym âX, June 03â05, 2018, Woodstock, NY References [1]Apoorva Adimulam, Rajesh Gupta, and Sumit Kumar. 2026. The Orchestration of Multi-Agent Systems: Architectures, Protocols, and Enterprise Adoption. arXiv preprint arXiv:2601.13671 (2026). [2]Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning. PMLR, 173â182. [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Report. CoRR abs/2502.13923 (2025). arXiv:2502.13923 doi:10.48550/ARXIV.2502.13923 [4] Baidu-ERNIE-Team. 2025. ERNIE 4.5 Technical Report. [5] Baidu-ERNIE-Team. 2026. ERNIE 5.0 Technical Report. [6] Edward Y. Chang and Longling Geng. 2025. ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning. CoRR abs/2505.12501 (2025). arXiv:2505.12501 doi:10.48550/ARXIV.2505.12501 [7]Wang Chen, Guanqiang Qi, Weikang Li, Yang Li, Deguo Xia, and Jizhou Huang. 2025. Pairs: Parametric-verified adaptive information retrieval and selection for efficient rag. arXiv preprint arXiv:2508.04057 (2025). [8]Wang Chen, Wenhan Yu, Guanqiang Qi, Weikang Li, Yang Li, Lei Sha, Deguo Xia, and Jizhou Huang. 2025. CMRAG: Co-modality-based visual document retrieval and question answering. arXiv preprint arXiv:2509.02123 (2025). [9]Konstantina Christakopoulou, Shibl Mourad, and Maja J. Mataric. 2024. Agents Thinking Fast and Slow: A Talker-Reasoner Architecture. CoRR abs/2410.08328 (2024). arXiv:2410.08328 doi:10.48550/ARXIV.2410.08328 [10]LluĂs C. Coll, Martin W. Lauer-Schmaltz, Philip J. Cash, John Paulin Hansen, and Anja A. Maier. 2025. Towards the "Digital Me": A vision of authentic Conversa- tional Agents powered by personal Human Digital Twins. CoRR abs/2506.23826 (2025). arXiv:2506.23826 doi:10.48550/ARXIV.2506.23826 [11]Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2025. Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks. In Forty-second Inter- national Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025 (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR / OpenReview.net. https://proceedings.mlr.press/v267/erdogan25a.html [12] Francesco Fabiano, Marianna Bergamaschi Ganapini, Andrea Loreggia, Nicholas Mattei, Keerthiram Murugesan, Vishal Pallagani, Francesca Rossi, Biplav Srivas- tava, and K. Brent Venable. 2025. Thinking Fast and Slow in Human and Machine Intelligence. Commun. ACM 68, 8 (2025), 72â79. doi:10.1145/3715709 [13] Hao Fei, Han Zhang, Bin Wang, Lizi Liao, Qian Liu, and Erik Cambria. 2024. EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot. CoRR abs/2406.15177 (2024). arXiv:2406.15177 doi:10.48550/ARXIV.2406.15177 [14] Tongtong Feng, Xin Wang, Yu-Gang Jiang, and Wenwu Zhu. 2025. Embodied AI: From LLMs to World Models [Feature]. IEEE Circuits and Systems Magazine 25, 4 (2025), 14â37. [15]Antonio A. Ginart, Naveen Kodali, Jason Lee, Caiming Xiong, Silvio Savarese, and John Emmons. 2024. Asynchronous Tool Usage for Real-Time Agents. CoRR abs/2410.21620 (2024). arXiv:2410.21620 doi:10.48550/ARXIV.2410.21620 [16] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. A Survey on LLM-as-a-Judge. CoRR abs/2411.15594 (2024). arXiv:2411.15594 doi:10.48550/ARXIV.2411.15594 [17]Jiazhi Guan, Zhiliang Xu, Hang Zhou, Kaisiyuan Wang, Shengyi He, Zhan- wang Zhang, Borong Liang, Haocheng Feng, Errui Ding, Jingtuo Liu, et al.2024. Resyncer: Rewiring style-based generator for unified audio-visually synced facial performer. In European Conference on Computer Vision. Springer, 348â367. [18]Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Hang Zhou, Shengyi He, Zhiliang Xu, Haocheng Feng, Errui Ding, Jingdong Wang, Hongtao Xie, et al.2024. Talk-act: Enhance textural-awareness for 2d speaking avatar reenactment with diffusion model. In SIGGRAPH Asia 2024 Conference Papers. 1â11. [19]Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 (2014). [20] Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions. CoRR abs/2503.23278 (2025). arXiv:2503.23278 doi:10.48550/ARXIV.2503.23278 [21]Hongbin Huang, Junwei Li, Tianxin Xie, Zhuang Li, Cekai Weng, Yaodong Yang, Yue Luo, Li Liu, Jing Tang, Zhijing Shao, and Zeyu Wang. 2025. Hi-Reco: High- Fidelity Real-Time Conversational Digital Humans. CoRR abs/2511.12662 (2025). arXiv:2511.12662 doi:10.48550/ARXIV.2511.12662 [22]Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. CoRR abs/2402.02716 (2024). arXiv:2402.02716 doi:10.48550/ARXIV.2402.02716 [23]Nikoleta Iliakopoulou, Jovan Stojkovic, Chloe Alverti, Tianyin Xu, Hubertus Franke, and Josep Torrellas. 2025. Chameleon: Adaptive caching and schedul- ing for many-adapter llm inference environments. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture. 217â231. [24] Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A. Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. 2025. ACON: Optimizing Context Compression for Long-horizon LLM Agents. CoRR abs/2510.00615 (2025). arXiv:2510.00615 doi:10.48550/ARXIV.2510.00615 [25] Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2025. From generation to judgment: Opportunities and challenges of llm-as- a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2757â2791. [26] Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhijiang Guo, Le Song, and Cheng-Lin Liu. 2025. From System 1 to System 2: A Survey of Reasoning Large Language Models. CoRR abs/2502.17419 (2025). arXiv:2502.17419 doi:10.48550/ARXIV.2502.17419 [27]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved Baselines with Visual Instruction Tuning. [28]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. In NeurIPS. [29]Yifan Lu, Rixin Liu, Jiayi Yuan, Xingqi Cui, Shenrun Zhang, Hongyi Liu, and Jiarong Xing. 2025.RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers. CoRR abs/2510.00202 (2025). arXiv:2510.00202 doi:10.48550/ARXIV.2510.00202 [30]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Bar- bieri, and Yuwei Fang. 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Asso- ciation for Computational Linguistics, 13851â13870. doi:10.18653/V1/2024.ACL- LONG.747 [31]Mykola Maslych, Mohammadreza Katebi, Christopher Peerapon Lee, Yahya Hmaiti, Amirpouya Ghasemaghaei, Christian Pumarada, Janneese Palmer, Este- ban Segarra Martinez, Marco Emporio, Warren Snipes, Ryan P. McMahan, and Joseph J. LaViola Jr. 2025. Mitigating Response Delays in Free-Form Conversa- tions with LLM-powered Intelligent Virtual Agents. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI 2025, Waterloo, ON, Canada, July 8-10, 2025, Jaisie Sin, Edith Law, Jim Wallace, Cosmin Munteanu, and Danai Korre (Eds.). ACM, 49:1â49:15. doi:10.1145/3719160.3736636 [32] Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. 2024. The Land- scape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey. CoRR abs/2404.11584 (2024). arXiv:2404.11584 doi:10.48550/ ARXIV.2404.11584 [33]Kai Mei, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. 2024. AIOS: LLM Agent Operating System. CoRR abs/2403.16971 (2024). arXiv:2403.16971 doi:10.48550/ARXIV.2403.16971 [34]Fengran Mo, Kelong Mao, Ziliang Zhao, Hongjin Qian, Haonan Chen, Yiruo Cheng, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, and Jian-Yun Nie. 2025. A Survey of Conversational Search. ACM Trans. Inf. Syst. 43, 6 (2025), 167:1â167:50. doi:10. 1145/3759453 [35]Se Park, Chae Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeonghun Yeo, and Yong Ro. 2024. Letâs Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 16334â16348. [36]Khush Patel, Siva Surendira, and Jithin George Shreyas Kapale. [n. d.]. THE SIX SIGMA AGENT: ACHIEVING ENTERPRISE-GRADE RELIABILITY IN LLM SYS- TEMS THROUGH CONSENSUS-DRIVEN DECOMPOSED EXECUTION. ([n. d.]). [37]Mathis Pink, Qinyuan Wu, Vy Ai Vo, Javier Turek, Jianing Mu, Alexander Huth, and Mariya Toneva. 2025. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents. CoRR abs/2502.06975 (2025). arXiv:2502.06975 doi:10.48550/ARXIV.2502.06975 [38]Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025. Computer Vision Foundation / IEEE, 24045â 24055. doi:10.1109/CVPR52734.2025.02239 [39]Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, Minghua Ma, Pu Zhao, Si Qin, Xiaoting Qin, Chao Du, Yong Xu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2023. TaskWeaver: A Code-First Agent Framework. CoRR abs/2311.17541 (2023). arXiv:2311.17541 doi:10.48550/ARXIV.2311.17541 Conference acronym âX, June 03â05, 2018, Woodstock, NYXin Shen et al. [40]Mashrur Rashik, Mahmood Jasim, Kostiantyn Kucher, Ali Sarvghad, and Narges Mahyar. 2024. Beyond Text and Speech in Conversational Agents: Mapping the Design Space of Avatars. In Designing Interactive Systems Conference, DIS 2024, IT University of Copenhagen, Denmark, July 1-5, 2024, Anna VallgĂĽrda, Li JĂśnsson, Jonas Fritsch, Sarah Fdili Alaoui, and Christopher A. Le Dantec (Eds.). ACM. doi:10.1145/3643834.3661563 [41] Ranjan Sapkota, Konstantinos I. Roumeliotis, and Manoj Karkee. 2026. AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges. Inf. Fusion 126 (2026), 103599. doi:10.1016/J.INFFUS.2025.103599 [42]Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. 2025. Scaling Long-Horizon LLM Agent via Context-Folding. CoRR abs/2510.11967 (2025). arXiv:2510.11967 doi:10.48550/ARXIV.2510.11967 [43] Clovis Varangot-Reille, Christophe Bouvard, Antoine Gourru, Mathieu Cian- cone, Marion Schaeffer, and François Jacquenet. 2025. Doing More with Lessâ Implementing Routing Strategies in Large Language Model-Based Systems: An Extended Survey. arXiv preprint arXiv:2502.00409 (2025). [44]Chen Wang, Xunzhuo Liu, Yuhan Liu, Yue Zhu, Xiangxi Mo, Junchen Jiang, and Huamin Chen. 2025. When to Reason: Semantic Router for vLLM. CoRR abs/2510.08731 (2025). arXiv:2510.08731 doi:10.48550/ARXIV.2510.08731 [45]Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al.2024. Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 17716â17736. [46]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. CoRR abs/2308.08155 (2023). arXiv:2308.08155 doi:10.48550/ARXIV.2308.08155 [47] You Xie, Tianpei Gu, Zenan Li, Chenxu Zhang, Guoxian Song, Xiaochen Zhao, Chao Liang, Jianwen Jiang, Hongyi Xu, and Linjie Luo. 2025. X-Streamer: Unified Human World Modeling with Audiovisual Interaction. CoRR abs/2509.21574 (2025). arXiv:2509.21574 doi:10.48550/ARXIV.2509.21574 [48] Jiayang Xu, Xiangjie Huang, Zijie Li, and Zili Meng. 2025. Make a Video Call with LLM: A Measurement Campaign over Five Mainstream Apps. CoRR abs/2510.00481 (2025). arXiv:2510.00481 doi:10.48550/ARXIV.2510.00481 [49] Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. 2025. StreamingVLM: Real-Time Understanding for Infinite Video Streams. CoRR abs/2510.09608 (2025). arXiv:2510.09608 doi:10.48550/ARXIV.2510. 09608 [50] Weikai Xu, Chengrui Huang, Shen Gao, and Shuo Shang. 2025. LLM-Based Agents for Tool Learning: A Survey. Data Sci. Eng. 10, 4 (2025), 533â563. doi:10. 1007/S41019-025-00296-9 [51]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 Technical Report. CoRR abs/2412.15115 (2024). arXiv:2412.15115 doi:10.48550/ARXIV.2412.15115 [52]Huck Yang, Andreas Stolcke, and Larry P. Heck. 2025. Spoken Conversational Agents with Large Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, Valentina Pyatkin and Andreas Vlachos (Eds.). Association for Computational Linguistics, Suzhou, China, 7â8. doi:10.18653/v1/2025.emnlp-tutorials.3 [53]Quanwei Yang, Jiazhi Guan, Kaisiyuan Wang, Lingyun Yu, Wenqing Chu, Hang Zhou, ZhiQiang Feng, Haocheng Feng, Errui Ding, Jingdong Wang, et al.2024. Showmaker: Creating high-fidelity 2d human video via fine-grained diffusion modeling. Advances in Neural Information Processing Systems 37 (2024), 51039â 51062. [54]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview. net/forum?id=WE_vluYUL-X [55]Haojie Yu, Zhaonian Wang, Yihan Pan, Meng Cheng, Hao Yang, Chao Wang, Tao Xie, Xiaoming Xu, Xiaoming Wei, and Xunliang Cai. 2025. LLIAâEnabling Low- Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models. arXiv preprint arXiv:2506.05806 (2025). [56] Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, and Dong Yu. 2025. Guided self-evolving llms with minimal human supervision. arXiv preprint arXiv:2512.02472 (2025). [57]Ruihong Zeng, Jinyuan Fang, Siwei Liu, and Zaiqiao Meng. 2024. On the Structural Memory of LLM Agents. CoRR abs/2412.15266 (2024). arXiv:2412.15266 doi:10. 48550/ARXIV.2412.15266 [58]Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Yilei Zhao, Ce Cui, Yang Liu, and Bo An. 2025. AgentOrchestra: Orchestrating hierarchical multi-agent intelligence with the Tool-Environment-Agent (TEA) protocol. (2025). Appendix This appendix is organized as follows: ⢠Limitations and Future Work (Section A) ⢠Related Work (Section B) ⢠Details of Du-Interact Benchmark (Section C) ⢠Prompt Engineering and Evaluation Protocols (Section D) ⢠Online Commercial Metrics and Protocols (Section E) ⢠Qualitative Case Studies (Section F) A Limitations and Future Work While DuCCAE has demonstrated substantial commercial success and architectural robustness in Baidu Search, we acknowledge several limitations that chart the course for our future research. 1. Perception Granularity vs. Latency. Currently, our Info System decouples visual perception by converting video streams into textual captions via a lightweight VLM. While this strategy effectively reduces latency by 77% and suppresses hallucinations, it inherently introduces an âInformation Bottleneckâ. Subtle visual cues (e.g., micro-expressions or complex spatial relationships) may be lost during the text conversion, potentially capping the upper bound of empathetic resonance. Future Work: We plan to explore Native Multimodal Fusion architectures that can process audio- visual tokens directly in the âFast Trackâ without intermediate text conversion, aiming to retain high-fidelity sensory details while maintaining the sub-500ms latency budget. 2. Inference Cost of Dual-Track Execution. The âLatency- Decoupled Architectureâ requires maintaining concurrent execu- tion contexts for both the conversational agent (Fast Track) and the reasoning planner (Slow Track). Although efficient for user experience, this imposes a significant GPU memory overhead per concurrent user (CCU), particularly given the 21B parameter size of our backbone model. Future Work: We are investigating Task- Specific Distillation to compress the 21B backbone into a mixture of smaller, specialized experts (<7B). This would allow us to of- fload specific routing or tool-calling tasks to lightweight models, reducing deployment costs without compromising the Success Rate. 3. Dependency on Proprietary Ecosystem. The current imple- mentation of DuCCAE benefits significantly from the proprietary infrastructure of Baidu, including the ERNIE foundation models and internal Search RAG APIs. This dependency presents a chal- lenge for the broader research community to fully reproduce our results. Future Work: To foster reproducibility, we aim to release a DuCCAE-Lite framework. This open-source version will abstract the orchestration logic and support plug-and-play compatibility with open-weights models (e.g., Llama/Qwen) and standardized tool protocols (MCP), enabling community benchmarking on the Du-Interact dataset. 4. Safety in Autonomous Tool Execution. As the Collaboration System evolves to handle increasingly complex tasks (e.g., trans- action execution), the risk of âAgentic Misalignmentâ increases. Current safety measures primarily focus on dialogue toxicity and hallucination. Future Work: We intend to integrate a Critic-in- the-Loop moduleâa dedicated verifier agent trained via Reinforce- ment Learning from Human Feedback (RLHF)âto simulate and Baidu DuCCAE SystemConference acronym âX, June 03â05, 2018, Woodstock, NY validate high-stakes tool actions before execution, ensuring stricter alignment with safety constraints in open-ended scenarios. B Related Work The development of immersive conversational agents stands at the intersection of real-time multimodal interaction and autonomous agent orchestration. Our work addresses the critical latency-capability trade-off inherent in this convergence. Recent research in digital human generation has predominantly focused on enhancing sensory fidelity through End-to-End (E2E) ar- chitectures. Systems such as Letâs Go Real Talk [35] and X-Streamer [47] integrate perception, reasoning, and generation into unified dif- ferentiable pipelines, achieving high audio-visual synchronization and emotional expressiveness. Similarly, Hi-Reco [21] advances the state-of-the-art in high-fidelity real-time rendering. However, these monolithic models often suffer from high inference latency and hallucinations when executing rigorous logical tasks, making them less suitable for service-oriented applications. In contrast, DuC- CAE adopts a decoupled modular architecture, prioritizing logical reliability and sub-second latency through lightweight text-based perception bottlenecks, ensuring robust tool execution without compromising conversational fluidity. The paradigm of LLM-based agents has evolved from single- turn interactions to complex multi-agent workflows. Frameworks like AutoGen [46] have standardized protocols for inter-agent col- laboration, enabling role-based problem solving. Concurrently, the concept of an LLM Agent Operating System (AIOS) [33] has emerged to address resource isolation and scheduling for concurrent agents. While these frameworks excel in offline task execution, they often lack mechanisms for real-time user interaction during execution. Our work aligns with the principles of Asynchronous Tool Usage [15], extending them into a dual-track orchestration engine that synchronizes asynchronous agentic workflows with synchronous video-based conversation, effectively bridging the gap between System 1 and System 2 processing [26]. Latency is a critical de- terminant of immersion in spoken dialogue systems. Extended silence disrupts turn-taking dynamics and degrades user trust. Re- cent studies [31] demonstrate the efficacy of conversational fillers and acknowledgement tokens in mitigating the perception of delay. DuCCAE systematizes this strategy via a dedicated Conversation System that generates semantic bridging responses, maintaining the conversational floor while the Collaboration System performs long-horizon planning in the background. C Details of Du-Interact Benchmark To ensure a rigorous evaluation of the systemâs robustness, we con- structed Du-Interact, a human-curated Golden Benchmark derived from real-world traffic in Baidu Search. Unlike random sampling, which is often dominated by trivial queries, Du-Interact is specifi- cally designed to stress-test the systemâs reasoning and planning capabilities. C.1 Sampling Strategy: Hard-Negative Mining The dataset consists of 5,000 multi-turn sessions sampled exclu- sively from the hard-negative pool of the V2 iteration. A session is included only if it meets at least one of the following criteria: â˘Long-Horizon Dependency: The session exceeds 8 turns and involves multiple context switches. ⢠Ambiguous Intent: The userâs request lacks explicit param- eters, requiring memory retrieval. ⢠Cross-Domain Tooling: The resolution requires coordinat- ing at least two distinct tools. This adversarial sampling strategy ensures that the high Success Rates reported in Table 2 reflect true agentic capability rather than over-fitting to simple patterns. C.2 Annotation Taxonomy Expert annotators labeled each session with three layers of Ground Truth (GT) to support the multi-dimensional evaluation metrics: (1)Routing GT: The strictly correct complexity tier (Tier-1/2/3) for every turn, used to calculate Dispatch Precision. (2)Execution GT: The optimal sequence of tool calls (function names) and precise argument values (slots). For complex planning, valid equivalent variants are also recorded. (3) Response GT: Key information points (Key-Value pairs) that must be present in the final response to ensure Response Fidelity. C.3 Quality Control Protocol To mitigate human error, we employed a Double-Blind Review process. Each session was initially annotated by two independent linguistic experts. Disagreements in routing or tool selection were flagged and adjudicated by a senior meta-annotator. The final dataset achieved an Inter-Annotator Agreement (Cohenâs Kappa) of í = 0.89, indicating high consistency. D Prompt Engineering and Evaluation Protocols To enhance the reproducibility and transparency of our work, this section details the specific instruction-tuning strategies employed in DuCCAE. Beyond standard prompt engineering, we implemented specific structural constraints to bridge the gap between the prob- abilistic nature of LLMs and the deterministic requirements of a production system. D.1 Orchestration via Structured Enforcement Figure 5 presents the master instruction used for the Router and Planner agents. Unlike open-ended conversational prompts, our design incorporates strict engineering constraints to ensure robust- ness in a high-traffic environment. The first among these is the enforcement of a rigid JSON schema, which is critical to bridging un- structured natural language with executable code. This constraint eliminates parsing errors in the downstream Dispatch Precision module and allows for seamless API argument extraction without the need for complex regular expressions. Furthermore, the prompt explicitly defines the operational bound- aries between Tier-1 (Chat), Tier-2 (Tool), and Tier-3 (Agent) modes directly within the context. This strategy significantly reduces rout- ing latency by preventing the model from over-analyzing simple Conference acronym âX, June 03â05, 2018, Woodstock, NYXin Shen et al. System Instruction: Router & Planner Agent ROLE You are DuCCAE, an intelligent digital human assistant embedded in Baidu Search. You can access external Tools (Search, Weather, Calendar) and a set of Sub-Agents (e.g., TravelPlanner, MedicalExpert). TASK Analyze the userâs latest query given the Session Context. 1) CLASSIFY Intent Complexity â˘Tier-1 (Fast Track) â chit-chat, greetings, or memory-based QA â output "mode": "chat" ⢠Tier-2 (Tool) â simple retrieval using tools â output "mode": "tool" ⢠Tier-3 (Complex) â long-horizon planning via sub- agents â output "mode": "agent" 2) EXECUTE According to Mode ⢠If "chat": generate empathetic response directly. ⢠If "tool/agent": decompose into executable steps. CONSTRAINTS ⢠Maintain Persona: Empathetic, Professional, Helpful ⢠JSON output only â no free-form reasoning text OUTPUT FORMAT "thought": "User requests travel plan â Tier-3", "mode": "agent", "routing_target": "TravelPlanner", "plan": [ "step": 1, "tool": "flight_search", "args": "dest": "Tokyo", "step": 2, "tool": "hotel_book", "args": "type": "Onsen" ] Figure 5: Core system prompt in DuCCAE-V3 for intent rout- ing and planning. The structured schema ensures strict com- pliance with the dual-track architecture and deterministic execution. queries, effectively short-circuiting the reasoning process for light- weight tasks. Additionally, by embedding persona constraints along- side functional instructions, we ensure that the system maintains a consistent tone even during internal routing decisions, preventing âroboticâ fallbacks during edge cases. D.2 LLM-as-a-Judge Configuration To assess the subjective metrics reported in Table 2 (specifically Persona Consistency and Empathy), we adopted GPT-4o as an im- partial evaluator. The evaluation prompt, shown in Figure 6, was LLM-as-a-Judge: Dialogue Quality Assessment TASK Evaluate the Model Response with respect to the User Query and Ground Truth (GT). INPUT ⢠User Query: [Input Query] ⢠Ground Truth: [Expert Annotation] ⢠Model Response: [DuCCAE Output] CRITERIA 1) Persona Consistency (1â5) ⢠5: Perfect match to target character ⢠1: Robotic / out-of-character 2) Empathy (1â5) ⢠5: Warm, emotionally supportive ⢠1: Cold, purely utilitarian 3) Response Fidelity (Hit/Miss) ⢠Conveys same key information as GT ⢠Ignore surface phrasing differences OUTPUT "persona_score": 5, "empathy_score": 4, "fidelity_hit": true, "reasoning": "Acknowledges emotion and keeps persona" Figure 6: Evaluation protocol for the LLM-as-a-Judge frame- work. The design enforces structured scoring aligned with expert annotations. meticulously engineered to mitigate common biases inherent in automated judging. Instead of relying on generic 1-5 scales, which often lead to cen- tral tendency bias, we provided concrete semantic anchors for the extreme scores from âWarm/Supportiveâ). This aligns the modelâs internal scoring logic with human expert standards. Crucially, the output format mandates aâreasoningâfield prior to the numerical scores. This requirement forces the evaluator to generate a Chain- of-Thought justification trace before assigning a value, a technique proven to improve the correlation between automated metrics and human judgment while reducing hallucinated evaluations. E Online Commercial Metrics and Protocols To validate the real-world impact of DuCCAE within Baidu Search, we monitored four critical business metrics during large-scale on- line controlled experiments. Unlike offline benchmarks which rely on static ground truth, these metrics quantify dynamic user be- havior. The rigorous definitions and calculation protocols for the metrics reported in Figure 4 are detailed below. Baidu DuCCAE SystemConference acronym âX, June 03â05, 2018, Woodstock, NY E.1 User Stickiness (Day-7 Retention) User retention serves as the primary indicator of long-term en- gagement. We define Day-7 Retention as the percentage of unique users active on DayíĄwho return to initiate at least one new valid interaction session on DayíĄ +7. The metric is formally calculated as: í ííĄíííĄííí 7 = |í íĄ âŠ í íĄ+7 | |í íĄ | Ă 100%,(1) whereí íĄ denotes the set of unique users who engaged with the system on day íĄ . E.2 Conversation Quality (Good Turn Rate) Given the sparsity of explicit user ratings in production, we assess conversation quality using the Good Turn Rate (GTR), a compos- ite metric derived from implicit feedback signals. A turn is classified as âGoodâ only if it satisfies satisfaction criteriaâspecifically, if the dwell time on the response card exceeds a dynamic length-based threshold or if an explicit positive interaction (click, share, like) occursâwhile triggering no negative signals. Negative indicators include immediate session termination, detected negative senti- ment in follow-up queries, or re-querying behavior. The metric is calculated daily as the ratio í ííí /í íĄííĄíí across all valid sessions. E.3 Interaction Depth (Avg. Session Turns) To quantify the depth of interaction, we measure the Average Ses- sion Turns across all users. For the purpose of this metric, a âSessionâ is rigorously defined as a continuous interaction sequence inter- rupted by no pause longer than 30 minutes. Any activity resuming after a timeout exceeding this threshold is logged as a new session. E.4 Agentic Capability (Online Task Completion) Measuring task completion in an open-ended online environment utilizes a strict Action-Based Proxy methodology, applied exclu- sively to sessions classified as Tier-2 (Tool) or Tier-3 (Complex Planning). A complex task is deemed âCompletedâ only if it meets specific terminal conditions: either a Service Conversion, where the user clicks on a functional service card generated by the agent, or Information Acceptance, characterized by the user accepting the provided plan without issuing corrections or reformulations within the subsequent two turns. F Qualitative Case Studies To validate our architectural decisions, we analyze representative interaction episodes sampled from the Du-Interact-Evo dataset (Fig- ure 7). While quantitative metrics confirm the systemâs overall efficacy, a qualitative examination of boundary conditions reveals the practical trade-offs inherent in deploying agentic systems at scale. Cases 1 through 3 demonstrate the efficiency of the Dual- Track routing mechanism. For high-frequency emotional queries (Case 1), the Fast Track bypasses reasoning modules to deliver responses within 450ms. For task-oriented queries, the system in- telligently distinguishes between deterministic tool usage (Case 2), which leverages optimized search patterns for speed, and complex multi-step planning (Case 3), which generates dependency graphs for reliability. This selective routing ensures that heavy reasoning resources are allocated only when strictly necessary. However, real-world complexity often challenges the systemâs boundaries, and analyzing failure modes provides critical engineer- ing insights. As shown in Case 4, the system encounters âPlanning Stagnationâ when user intents lack specific discriminators (e.g., âthat video from last monthâ). The vector retrieval returns numer- ous candidates with high semantic similarity, causing the Planner to stall. Crucially, the systemâs Safety Guardrails correctly identify this ambiguity constraint and trigger a clarification request instead of hallucinating a selection, prioritizing user trust over forced task completion. Beyond retrieval ambiguity, we observe two other significant categories of limitations. First is the trade-off between Perception Granularity and Latency. As discussed in Appendix A, our deci- sion to decouple visual perception via captioning reduces latency by 77% but introduces an information bottleneck. In failure sce- narios where users reference subtle visual cues (e.g., âIs this small spot on the apple safe?â), the lightweight captioning model often omits fine-grained details, leading to generic responses. Second is Contextual Drift in Asynchronous Execution. If a user initiates a long-horizon task (Slow Track) but rapidly switches context to a new topic (Fast Track) before the planner finishes, the delayed result may interrupt the new conversation flow. This highlights the necessity for a robust âTask Preemptionâ protocol to silently discard obsolete tasks when the session topic shifts, which remains a key focus for our future iterations. Conference acronym âX, June 03â05, 2018, Woodstock, NYXin Shen et al. Case 1 I'm totally drained... work was a nightmare today. I'm so sorry to hear that. Finance can be incredibly stressful. Why don't you sit back, and I'l play some of that jazz playlist you love to help you decompress? Info System User likes soothing jazz music User works in finance User Memory Multimodal Processing Sentiment: Negative/Tired Fast Track Log Case 2 What's the stock price of Baidu right now, and has it gone up since yesterday? Baidu is currently trading at $X.X. Yes, it has increased by x.x% compared to yesterday's closing price. Slow Track Log Augmentation System Search_Tool.query("Baidu stock price real-time comparison vs previous close" Tools Case 3 My wife's birthday is this Friday. Can you book a quiet Italian restaurant in Haidian for 7 PM, and also arrange for a bouquet of red roses to be delivered to the table? Let me arrange that special evening for you... All set! We could book a quiet table at X in Haidian for Friday at 7 PM. We could also order a bouquet of 12 red roses from a nearby florist to be waiting at your table. Slow Track Log Collaboration System Step 1 (Restaurant) Search & Book Italian restaurant in Haidian (Constraint: Quiet). èOutput: âXâ (Restaurant Name) Step 2 (Dependency) Wait for Step 1 to confirm address and time. Step 3 (Shopping) Search Florist & Schedule Delivery to "X ". Case4 Find that video about the funny cat I watched sometime last month. I found quite a few funny cat videos in your history from last month. Do you remember any specific details, like the cat's color or what it was doing, to help me find the right one? Slow Track Log Query Retrieval: topic="funny cat", time_range=âY-M" (Last Month) Result: Found 47 logs (Similarity > 0.85) â [High Ambiguity] Info System Plan: [Select_Video] Execute: FAILED. * State Update: Write Error: Missing_Unique_Constraint to Task Context. Generator: Generate_Clarification_Response Collaboration System Figure 7: Qualitative examples of DuCCAEâs Dual-Track execution. Note: Specific entities, numerical values, and PII have been replaced with placeholders (e.g., âXâ) to strictly preserve user privacy and commercial confidentiality.