Paper deep dive

SiriusDeliver: Automating Data Warehouse Delivery at Tencent

Haining Xie, Xiaokai Zhou, Jiaming Yang, Siqi Shen, Ziwei Wang, Yifeng Zheng, Tengyue Xu, Yipeng Shi, Zefang Zong, Yang Li, Peng Chen, Jie Jiang, Debiao He, Xiao Yan, Jiawei Jiang

Year: 2026Venue: arXiv preprintArea: cs.DBType: PreprintEmbeddings: 80

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Enterprise data warehouses (DWs) support business-critical analytics, but warehouse task delivery remains a complicated production process involving context retrieval, workflow configuration, code generation, platform submission, and failure diagnosis. Although large language models (LLMs) and coding agents have improved software development, they are insufficient for production DW delivery, which requires dependency-aware orchestration, lifecycle-aware artifact control, and continuous adaptation to evolving platform practices. We present SiriusDeliver, an end-to-end delivery automation agent for production warehouse task submission. SiriusDeliver integrates three components: a hierarchical delivery agent that orchestrates warehouse skills, an artifact lifecycle control module that verifies and revises artifacts before and after platform execution, and a trace-driven skill evolution mechanism that maintains reusable skills from delivery trajectories. We evaluate SiriusDeliver through offline datasets and large-scale production deployment on Tencent Cloud WeData. Offline experiments on real-world warehouse delivery cases show that SiriusDeliver improves delivery success and automation efficiency over representative baselines. During a two-month deployment across 6 business teams and 4 warehouse task types, SiriusDeliver served 3,600 monthly active users and supported 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. A one-month A/B test shows that SiriusDeliver reduces median delivery time from 228 to 23 minutes and engineer effort from 95 to 11 minutes, while maintaining comparable final delivery success.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

78,634 characters extracted from source content.

Expand or collapse full text

SiriusDeliver: Automating Data Warehouse Delivery at Tencent Haining Xie † , Xiaokai Zhou ‡ , Jiaming Yang † , Siqi Shen † , Ziwei Wang † , Yifeng Zheng † , Tengyue Xu † , Yipeng Shi † , Zefang Zong † , Yang Li † , Peng Chen † , Jie Jiang † , Debiao He ‡ , Xiao Yan ‡ , Jiawei Jiang ‡ † TEG, Tencent Inc. ‡ Wuhan University † hainingxie,besmingyang,siqishen,willziwang,yifengzheng,leooxu, portershi,willzong,thomasyngli,pengchen,zeus@tencent.com ‡ xiaokaizhou,hedebiao,yanxiaosunny,jiawei.jiang@whu.edu.cn Abstract Enterprise data warehouses (DWs) support business-critical analyt- ics, but warehouse task delivery remains a complicated production process involving context retrieval, workflow configuration, code generation, platform submission, and failure diagnosis. Although large language models (LLMs) and coding agents have improved software development, they are insufficient for production DW delivery, which requires dependency-aware orchestration, lifecycle- aware artifact control, and continuous adaptation to evolving plat- form practices. We present SiriusDeliver, an end-to-end delivery automation agent for production warehouse task submission. Sir- iusDeliver integrates three components: a hierarchical delivery agent that orchestrates warehouse skills, an artifact lifecycle con- trol module that verifies and revises artifacts before and after plat- form execution, and a trace-driven skill evolution mechanism that maintains reusable skills from delivery trajectories. We evaluate SiriusDeliver through offline datasets and large-scale production deployment on Tencent Cloud WeData. Offline experiments on real- world warehouse delivery cases show that SiriusDeliver improves delivery success and automation efficiency over representative base- lines. During a two-month deployment across 6 business teams and 4 warehouse task types, SiriusDeliver served 3,600 monthly active users and supported 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. A one-month A/B test shows that SiriusDeliver reduces median delivery time from 228 to 23 minutes and engineer effort from 95 to 11 minutes, while maintaining comparable final delivery success. Keywords Data Warehouse Task Delivery, Data Agent, Skill Evolution 1 Introduction Enterprise data warehouses (DWs) underpin business-critical ana- lytics by transforming raw business events into analytical tables. At Tencent, the production warehouse platform serves over 10,000 monthly active data engineers, manages millions of warehouse tasks, and maintains hundreds of thousands of production tables. Yet warehouse task delivery remains largely manual. As shown in Figure 1, engineers must retrieve context from metadata ser- vices, workflow platforms, historical tasks, logs, and troubleshoot- ing documents; generate code and configurations; validate depen- dencies; submit tasks; and repair failures. This fragmented process Conference’17, Washington, DC, USA 2026. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n Intent WeData Warehouse 6 Teams 3600+ MAUs 18,000+ Sessions Generate daily user feature tables from transaction logs. Artifact Generation ExecutionPlanning Artifact LifecycleControl Skill Orchestration Dependency Context Planning Metadata ... Workflow History Engineer Delivery Agent ConfigCodes Fail Failure Diagnosis Manual Repair & Resubmit Deliver Verify DiagnoseRevise Fail Deliver Skill Evolution Improve Resubmit Delivery Time 228 → 23 min AutoSubmission 73.5% Engineer Effort 95 → 11 min Human-driven DeliveryAgent-driven Automation Online Impact Figure 1: Traditional human-driven warehouse delivery (left) vs. our agent-driven automation (right). SiriusDeliver auto- mates the end-to-end delivery process, substantially reducing manual intervention in production warehouse systems. is labor-intensive and error-prone: missing dependencies, inconsis- tent schedules, invalid permissions, or inefficient queries can delay delivery or introduce silent data-quality risks. Evidence from our internal questionnaire survey, supplemented by in-depth follow- up interviews with experienced practitioners, suggests that SQL development accounts for less than 25% of end-to-end delivery time, while information retrieval and cross-platform coordination collectively constitute the primary sources of effort. Recent advances in large language models (LLMs) [1,7,9,14] and coding agents [2,10,15,25] have demonstrated strong capabilities in automated software development. However, production ware- house task delivery fundamentally differs from repository-level coding: it requires end-to-end coordination over data semantics, workflow dependencies, platform policies, execution feedback, and long-term operational maintenance. Without such DataOps inte- gration, general-purpose coding agents often produce plausible artifacts that remain incomplete, unsafe, or difficult to maintain in production warehouse environments. Our production experience shows that directly applying general- purpose code agents to this setting leaves three key challenges. (1) Multi-stage task composition. Warehouse task delivery spans multiple interdependent stages—from metadata retrieval and de- pendency resolution to code generation, configuration, and plat- form submission. For instance, importing operational data requires resolving source/target schemas, incremental keys, and schedule dependencies before task execution. In practice, one-shot LLM planning frequently omits or misorders such prerequisites, produc- ing invalid configurations or failed submissions. Existing tool-use arXiv:2608.09185v1 [cs.DB] 10 Aug 2026 Conference’17, July 2017, Washington, DC, USAXie et al. agents [16,19,20] rely on curated supervision or static tool defi- nitions, which fail to handle enterprise warehouses with complex, implicit, and team-specific dependency constraints. (2) Trustworthy artifact delivery. A deliverable warehouse bun- dle encompasses executable code, workflow topologies, schedules, permissions, and platform metadata. Syntactically valid artifacts may still contain invalid dependencies, schedule conflicts, or logic bugs that trigger silent data errors or execution failures. Existing code agents [28,30] perform reactive post-execution repair based solely on logs. In contrast, production delivery demands proactive pre-submission verification alongside post-execution diagnosis to guarantee end-to-end artifact safety. (3) Continuous platform adaptation. As platform APIs, data schemas, scheduling policies, and operational conventions con- tinually evolve, static delivery rules can rapidly become obsolete. Existing lifelong learning methods [22,26,31] optimize transient task-level behaviors rather than auditable, reusable skill updates. Furthermore, they overlook signals from successful trajectories, which can reveal potentially redundant context acquisition and reasoning steps that could streamline future execution. To address these challenges, we present SiriusDeliver, an end- to-end agent for production warehouse task delivery. For Chal- lenge (1), its delivery automation agent iteratively plans, routes, executes, and updates memory over hierarchical scenario, context, artifact, and platform skills, composing their outputs into work- flow specifications, executable artifacts, and task configurations. For Challenge (2), artifact lifecycle control validates and revises artifacts before and after execution by combining deterministic plat- form evidence, LLM reasoning, execution feedback, and retrieved diagnostic knowledge to detect risks and explain failures. For Chal- lenge (3), trace-driven skill evolution groups trajectories by skill and outcome into structured evidence and applies bounded LLM updates, using failures for repair and successes for compression. We evaluate SiriusDeliver through offline real-world cases and online production deployment. Offline, on 200 real-world delivery cases across 4 warehouse scenarios, SiriusDeliver improves the average end-to-end delivery success rate by 14.5 points (from 71.5% to 86.0%) over the strongest skill-augmented baseline and reduces token consumption by 30%, with ablations validating the contri- bution of each component. Online, a two-month deployment on Tencent Cloud WeData across 6 business teams and 4 warehouse scenarios served 3,600 monthly active users and supported 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. In a one-month A/B test, Sir- iusDeliver reduced delivery time from 228 to 23 minutes, engineer effort from 95 to 11 minutes, time to first artifact from 44 to 2.6 minutes, and manual intervention from 100% to 21%. To summarize, we make the following contributions: • We propose SiriusDeliver, a delivery automation agent that organizes warehouse skills hierarchically and generates platform- submittable artifact bundles from business requirements. •We design artifact lifecycle control with pre- and post-execution diagnosis to detect risks, explain failures, and revise artifacts. • We introduce trace-driven skill evolution, which turns delivery trajectories into bounded skill updates for failure repair and successful-run compression. Offline Delivery Agent Executor Manager Memory Intent Pre-execution diagnosis. Post-execution diagnosis. Trace-driven Skill Evolution Trajectory Grouping Validation and deployment Bounded Evolution Skill Library Update Memory Artifact Lifecycle Control Artifacts Business intent Delivery Agent Executor Manager Memory Intent Skill Library Deliver Data warehouse platform Tencent Cloud WeData Tencent Cloud WeDataWarehouse Approve Deliver Fail and Feedback Revise Validate Figure 2: Overview of key techniques in SiriusDeliver. •We evaluate SiriusDeliver on 200 real-world cases and in large- scale production deployment, achieving an 87.2% end-to-end success rate while reducing median delivery time and engineer effort by 89.9% and 88.4%, respectively. 2 Related Work Data warehouse platforms. Modern cloud-native data platforms (e.g., Snowflake [5], BigQuery [13], and Delta Lake [3]) provide high- performance storage and analytical query engines. However, these systems primarily expose low-level engine capabilities rather than automating end-to-end task delivery. Engineers must still manually interpret requirements, resolve dependencies, configure schedules, and diagnose execution failures. In contrast, SiriusDeliver oper- ates as an automation layer above warehouse platforms to close the requirement-to-submission loop using execution feedback. LLM-based NL-to-SQL. NL-to-SQL maps natural language to exe- cutable queries via schema linking, in-context learning, and exe- cution feedback [8,17,24,29]. While effective for query synthesis, NL-to-SQL solves only a subset of production warehouse delivery. A deployable warehouse task requires not only valid SQL, but also workflow topology, scheduling parameters, permission configs, and cross-stage failure handling. Thus, NL-to-SQL alone cannot satisfy full-lifecycle production submission requirements. LLM agents for code and data systems. LLM agents extend LLMs to repository editing and data analytics (e.g., SWE-agent [28], AutoCodeRover [30], TaskWeaver [18], and Data Interpreter [12]), while platform assistants (e.g., Databricks Assistant [6], Snowflake Cortex [23], and Gemini in BigQuery [11]) assist in query genera- tion. However, these systems mainly focus on code completion or interactive analysis. They do not address production task delivery, which demands dependency-aware orchestration, lifecycle artifact verification before/after platform submission, and trace-driven skill maintenance as platform practices evolve. 3 SiriusDeliver Figure 2 shows the workflow of SiriusDeliver. Given a warehouse requirement, SiriusDeliver first invokes the delivery automation agent to select warehouse skills, retrieve context, and generate an artifact bundle with workflow specifications, executable arti- facts, and task configurations. The bundle is then passed to artifact SiriusDeliver: Automating Data Warehouse Delivery at TencentConference’17, July 2017, Washington, DC, USA lifecycle control for pre-execution diagnosis, where evidence, depen- dencies, permissions, and configurations are verified before sub- mission. Only approved artifacts are submitted to the warehouse platform by the agent. After execution, post-execution diagnosis analyzes platform feedback and logs, explains failures, and revises the artifacts. The revised artifacts are returned to the agent for resubmission, closing the delivery loop. Throughout this process, SiriusDeliver records skill calls, intermediate artifacts, diagnostic results, execution feedback, and final outcomes as delivery trajecto- ries. Offline, trace-driven skill evolution transforms these trajectories into structured evidence, and proposes bounded skill updates for failure repair and successful-run compression. 3.1 Delivery Automation Agent Warehouse delivery starts from a business requirement, but exe- cutable artifacts depend on progressively acquired context, includ- ing data semantics, workflow states, historical configurations, and platform constraints. A one-pass LLM generation process cannot reliably determine which evidence is missing, which skills should be invoked, or when the delivery state is complete. General tool-use agents can invoke tools, but they usually do not explicitly track warehouse-specific prerequisites, dependencies, and submission readiness. To address this, we propose a delivery automation agent that maintains a structured delivery state and orchestrates ware- house skills to incrementally construct a candidate artifact bundle. Delivery objective. The agent aims to produce a candidate artifact bundle for downstream lifecycle validation. Definition 1 (Artifact bundle). Given a user requirement 푞, historical workflow configurations퐻, and platform context푀, the delivery automation agent outputs퐴=⟨퐺,퐶,푃⟩,where퐺is a workflow specification,퐶is a set of node-level executable artifacts, and 푃 is a set of task configurations. 퐺 describes the workflow structure, which can be a single node for single-task delivery or a DAG for workflow-level delivery.퐶 contains executable artifacts such as SQL, PySpark, Flink SQL, or Python programs.푃specifies schedules, dependencies, retry poli- cies, runtime parameters, resource hints, permissions, and other platform options. This allows code, workflow structure, and config- urations to be jointly validated before platform submission. Design insights. Our design follows four observations from pro- duction warehouse delivery. First, evidence is scenario-specific: synchronization tasks require connections, schemas, mappings, and incremental keys, whereas computation tasks require schemas, partition semantics, dependencies, and execution resources. Sec- ond, submission-critical fields must be clarified rather than inferred. Third, existing warehouse assets should be reused whenever possi- ble. Fourth, historical configurations provide grounded references for team-specific schedules, resources, retries, naming, and depen- dencies. Accordingly, the agent identifies the delivery scenario, resolves critical fields, reuses existing workflows or nodes, and retrieves relevant configurations before artifact generation. Agent and warehouse skills. The delivery automation agent is a closed-loop framework that maintains the delivery state and or- chestrates warehouse skills until all required artifacts are complete. Executor Yes Intent Router Scenario Context Artifact Platform Planner Missing? No Skill Library RetrieveInvoke Memory Terminator Artifacts No Short-term Context Long-term Knowledge Retrieve Memory UpdateYes Figure 3: Orchestration loop of the delivery automation agent. The manager plans and routes warehouse skills, the executor invokes them and updates memory, and the loop iterates until a platform-submittable artifact bundle is produced. A warehouse skill is a reusable executable capability with de- fined inputs, outputs, prerequisites, and side effects, implemented through metadata retrieval, deterministic scripts, LLM calls, historical- task search, or platform APIs. Besides task outputs, a skill may emit structured hints, such as the inferred scenario, unresolved prerequi- sites, and candidate next steps, to guide subsequent orchestration. To reduce planning complexity, SiriusDeliver organizes ware- house skills into four conceptual layers. Scenario skills identify the delivery scenario, such as computation, synchronization, or work- flow update. Context skills retrieve warehouse evidence, such as table metadata, workflow structure, and historical configurations. Artifact skills generate executable code, workflow specifications, and task configurations. Platform skills interact with the warehouse platform, including approved submission, status query, and feed- back collection. This hierarchy narrows the candidate skill space at each step and makes implicit dependencies easier to resolve. Orchestration. Figure 3 illustrates the iterative delivery orchestra- tion process. Given a user requirement, the agent checks whether the current delivery state contains sufficient information for the next step. It requests targeted clarification when critical inputs are missing; otherwise, it selects and executes warehouse skills based on the current state and historical context. Skill outputs are normalized and written to memory, enabling subsequent steps to use newly acquired metadata, workflow evidence, generated arti- facts, and platform feedback. The loop terminates when퐺,퐶, and 푃are complete, and the resulting candidate bundle is passed to artifact lifecycle control. After lifecycle approval, the agent invokes the corresponding platform skill to submit the bundle and collect execution feedback. The framework has three internal roles. •Manager tracks the delivery state and decides whether to request clarification, continue execution, or terminate with a complete candidate bundle. Its planner decomposes tasks, router selects skills, and terminator checks bundle completeness. •Executor resolves inputs from the requirement and memory, schedules prerequisite-dependent skills, invokes them, and records normalized outputs and execution traces. • Memory stores short-term session context, including metadata, workflow evidence, and skill outputs, as well as long-term knowl- edge from compressed trajectories, reusable configurations, and Conference’17, July 2017, Washington, DC, USAXie et al. Artifacts Evidence Grounding Hybrid Verification Pre-execution Diagnosis WeData Warehouse Failure Extraction Knowledge Retrieval Post-execution Diagnosis Memory Candidate Revision Approve Feedback Revise States & Traces Fail Deliver Root-cause- based Revision Figure 4: Artifact lifecycle control. Artifacts undergo pre- execution verification before submission, and post-execution diagnosis transforms platform feedback into repair sugges- tions and revised artifacts for subsequent execution. execution patterns. It also retains platform feedback and diagnos- tic traces for subsequent reasoning, lifecycle control, and skill evolution. 3.2 Artifact Lifecycle Control The delivery agent produces candidate rather than production- ready artifact bundles. Even syntactically valid bundles may con- tain silent semantic errors, invalid dependencies, permission or scheduling conflicts, and inefficient execution patterns. Direct sub- mission incurs costly trial and error, while pure LLM-based repair may overlook platform evidence, execution feedback, and histori- cal diagnostic knowledge. To address these risks, artifact lifecycle control validates and revises artifacts in two complementary stages, as shown in Figure 4: pre-execution diagnosis verifies artifacts be- fore submission using platform evidence, deterministic checks, and metadata-aware reasoning; post-execution diagnosis analyzes plat- form feedback, retrieves diagnostic knowledge, and revises failed artifacts for subsequent execution. Diagnosis coverage. Based on practical observations from ware- house development, we define diagnosis coverage for both pre- submission risks and post-execution failures, as summarized in Table 3. Pre-execution diagnosis targets defects that can be detected before platform submission from generated artifacts, task configu- rations, metadata, and pre-compilation evidence. For example, it detects brute-force scans caused by missing partition predicates and business-intent mismatches that may produce silent data er- rors. Post-execution diagnosis targets failures exposed by platform feedback after compilation, dry-run, or execution. For example, it diagnoses permission exceptions from access-denied logs and table-structure exceptions from partition conflicts. A diagnosis item is the basic unit of this coverage. Each item specifies: (1) the issue description and its risk; (2) judgment condi- tions for reporting the issue; (3) required evidence, such as code locations, compiler outputs, metadata, logs, or retrieved cases; and (4) the execution workflow for collecting evidence and producing the diagnosis. This structure prevents unsupported or speculative reports and makes diagnosis results comparable across tasks. Pre-execution diagnosis. To detect both execution failures and silent warehouse defects before costly production platform runs, we adopt a hybrid strategy that combines deterministic platform ev- idence with LLM-based reasoning. Deterministic evidence grounds each diagnosis in concrete, directly verifiable platform facts, such as current metadata, task configurations, declared dependencies, and pre-compilation results. LLM-based reasoning complements these facts by analyzing user intent, SQL semantics, and complex metadata-dependent risks that are difficult to reliably capture with rules alone. The resulting diagnostic process contains three steps. •Evidence grounding. We construct a canonical diagnosis input from codes, configurations, user requests, and conversation his- tory. Missing parameters are inferred only when supported by existing evidence, following the priority of code and configura- tions, conversation history, and code comments. If required fields remain unresolved, we ask targeted follow-up questions. • Hybrid verification. We combine statement-level pre-compilation with metadata-aware reasoning. The process decomposes SQL into atomic statements, collects pre-compilation evidence, ex- tracts table and field references, retrieves related metadata, and checks predefined diagnosis items. •Candidate revision. We report only defects that may cause execu- tion failures, data errors, or severe performance issues. Each issue includes its type, severity, code evidence, and repair suggestion. When sufficient evidence is available, we prompt an LLM with the original artifact, verified evidence, and repair suggestion to generate a revised artifact. Otherwise, we preserve the original artifact and report unresolved issues. Post-execution diagnosis. To convert platform failures into struc- tured repair evidence, we build on log-grounded diagnosis [4,21,27] by extracting failure signals from platform feedback and retriev- ing relevant diagnostic knowledge to guide artifact revision. This avoids direct repair from raw logs, reduces unsupported root causes, and keeps repair suggestions traceable. It contains three steps. •Failure extraction. We collect detailed execution logs, runtime errors, task states, and related platform metadata. We then extract concise, structured failure signals from platform feedback, such as the exception type, failed operator, error message, task stage, or platform component. These signals must be directly derived from platform feedback rather than inferred from the SQL alone. •Knowledge retrieval. We retrieve relevant cases, rules, and trou- bleshooting documents from a diagnostic knowledge base using the extracted failure signals. The retrieved evidence provides historical fixes and known platform behaviors that are difficult to infer from the current failure alone. • Root-cause-based revision. We diagnose the failure using the orig- inal artifact, extracted failure signals, and retrieved knowledge. Raw logs are used as auxiliary context, while localization focuses on the key failure signals. The output includes an exception cat- egory, a concise root-cause explanation, and an actionable repair suggestion. When the repair is sufficiently grounded, we prompt an LLM with the original artifact, localized evidence, and repair suggestion to generate a revised artifact for the next execution. 3.3 Trace-driven Skill Evolution Warehouse skills cannot remain static after deployment, because platform conventions, task patterns, and failure cases continuously change in production. Online delivery further produces reusable SiriusDeliver: Automating Data Warehouse Delivery at TencentConference’17, July 2017, Washington, DC, USA Grouping Compress Trajectory User: Sync user_behaviortable daily Agent: I will first check metadata... [Tool call] : get_table_metadata [Log]: 32 columns retrieved Agent: we can reuse template... ... Strucured Trajectory Session ID: c7f3e36d... Intent: Daily sync user_behavior Skills: <selected skills> Tools: <involved tools > Artifacts: <generatedartifacts > ... Metadata Task Type: ETL Sync Execution Shape: # Turns: 7 # Tool Calls: 4 # Skills Used: 5 Performance: # Tokens: 1.1M ... Scenario Context Artifact Platform Skill Library LLM 1. Trajectory Processing External Experience Retrieve Experience-guided Refinement Validation Pass Update Fail Revise 2. Skill Evolution Guide Self Figure 5: Trace-driven skill evolution. Raw trajectories are transformed into structured experience and metadata, which guide LLM-based skill refinement until validated updates are incorporated into the skill library. experience, including successful execution paths, failed attempts, diagnostic results, and platform feedback. However, these signals are scattered across long trajectories and cannot directly improve reusable warehouse skills. Manual maintenance is costly, while unrestricted LLM rewriting may introduce unstable changes to operational skills. To address this, we propose trace-driven skill evolution, an offline mechanism that converts delivery trajectories into bounded and reviewable skill refinements. As shown in Fig- ure 5, this module proceeds in three steps. Evidence preparation groups and compresses raw trajectories into structured evidence. Experience-guided refinement proposes bounded updates from tra- jectory evidence and historical update records. Validation and de- ployment verify candidate skill updates before human-reviewed release and rollback-enabled production deployment. Design insights. Our design follows three observations from pro- duction deployment. First, failures often recur around specific skills, scenarios, or platform constraints and should therefore be aggre- gated at the skill level. Second, successful trajectories also reveal reusable configurations, stable execution patterns, and redundant context acquisition. Third, unrestricted LLM rewriting may propa- gate regressions across downstream submissions. Therefore, Sirius- Deliver derives structured evidence from both outcomes, applies explicit, bounded refinements, and releases updates only after of- fline replay validation and mandatory human review. Evidence preparation. We convert long, multi-skill delivery ses- sions into structured evidence for skill-level maintenance. Each trajectory records the request, invoked skills and tools, intermedi- ate artifacts, diagnoses, platform feedback, and final outcome, and may contribute to multiple skill groups. We group trajectories by skill and outcome: failures reveal missing preconditions, unsafe assumptions, and weak recovery or validation, whereas successes reveal reusable context, shorter execution paths, and redundant calls. For each selected session, we retain these key interactions in a compact trajectory and pair it with metadata on turns, skills, tool calls, latency, and errors, while removing verbose reasoning and truncating long outputs. This representation preserves the evidence needed for evolution with lower noise and processing cost. Experience-guided refinement. This stage converts prepared evidence into bounded skill refinements. Directly rewriting a skill from raw traces is unstable, as the LLM may overfit to one failure or change unrelated behavior. We therefore retrieve two types of evo- lution experience before prompting the LLM. Self-guided experience retrieves previous updates of the same skill and their modification reasons, helping avoid repeated ineffective changes and maintain re- vision consistency. External-guided experience selectively retrieves relevant update records from related skills, providing transferable cross-skill repair or compression patterns. Given the structured trajectories and retrieved experience, the LLM proposes a candidate refinement with an update reason linked to supporting evidence. Repair-oriented refinements address re- curring failures, while compression-oriented refinements remove redundant context, steps, or tool calls from successful executions. An update budget strictly limits the number of modified fields and rewritten text in each iteration, keeping the evolution process re- viewable and preventing free-form skill rewrites. Validation and deployment. This stage prevents skill evolution from introducing regressions into production delivery. Candidate updates are not deployed immediately. We first run offline valida- tion on historical trajectories. The validation set includes targeted failed trajectories, successful trajectories that should remain unaf- fected, and held-out trajectories from related skills. A refinement is accepted only if it resolves the targeted failure or efficiency issue without introducing regressions on previously successful cases. After offline validation, the candidate refinement and its update reason are sent for human review. The reviewer may accept, re- ject, or edit the refinement before deployment. For each accepted update, we record the previous skill version, the new version, the update reason, and the validation result. If later monitoring detects degraded performance, the deployed skill can be rolled back to the previous version. This protocol makes production skill evolution trace-driven, offline replay-validated, and safely reversible. 4 Experimental Evaluation We evaluate SiriusDeliver through offline benchmarks and online production deployment by examining three questions:①whether it improves end-to-end warehouse delivery over representative baselines;②how artifact lifecycle control and trace-driven skill evolution affect reliability and efficiency; and③whether these gains translate into practical production benefits. 4.1 Experimental Settings Dataset. We evaluate SiriusDeliver in both offline and online set- tings. For offline evaluation, we construct a frozen benchmark of 200 production delivery cases from Tencent Cloud WeData, stratified equally across four task types (50 each): real-time synchronization, offline synchronization, real-time computation, and offline com- putation. Each case pairs a user requirement with the platform context needed for execution. The benchmark snapshots are frozen before evaluation, and all skill-evolution trajectories are strictly disjoint from evaluation cases to prevent data leakage (details in Appendix A.2). Due to proprietary schemas and platform metadata, these cases cannot be released; we instead provide DataClawEval, a public companion benchmark covering general data-engineering https://github.com/Dicemy/DataClawEval Conference’17, July 2017, Washington, DC, USAXie et al. Table 1: Offline end-to-end delivery performance and ablation on 200 WeData cases (50 per task type). The upper block compares three coding products under two configurations (+ skills vs. + SiriusDeliver); ∆ is the average-success gain of + SiriusDeliver. The lower block ablates the three SiriusDeliver components on the best product (Claude Code). Success is mean±95% CI (푛=200); turns, time, and tokens are means. Bold: best; underline: second-best. ProductConfiguration End-to-end success rate (%)↑ Avg. success↑Turns↓Time (min)↓Tokens (K)↓ RT SyncOff. SyncRT Comp.Off. Comp. Codex + skills72.068.062.056.064.5± 6.69.819.4151.6 + SiriusDeliver88.086.080.074.082.0± 5.36.111.898.4 ∆+16.0+18.0+18.0+18.0+17.5−3.7−7.6−53.2 CodeBuddy + skills76.072.066.060.068.5± 6.49.117.6137.2 + SiriusDeliver90.088.082.076.084.0± 5.15.710.891.5 ∆+14.0+16.0+16.0+16.0+15.5−3.4−6.8−45.7 Claude Code + skills80.074.068.064.071.5± 6.28.416.0124.0 + SiriusDeliver92.090.084.078.086.0± 4.85.410.286.9 ∆+12.0+16.0+16.0+14.0+14.5−3.0−5.8−37.1 Ablation of SiriusDeliver (on Claude Code) w/o hierarchical skill orch.72.072.080.074.074.5± 6.07.714.8118.7 w/o artifact lifecycle control86.084.074.068.078.0± 5.77.113.7111.8 w/o pre-execution diag.88.086.076.070.080.0± 5.56.712.9101.4 w/o post-execution diag.88.086.078.072.081.0± 5.45.911.4107.8 w/o trace-driven skill evolution90.088.082.076.084.0± 5.16.211.7112.3 capabilities. The online evaluation uses long-running real-world production deployment logs, as detailed in Section 4.3. Baselines and product configurations. We compare three repre- sentative agentic coding products: CodeBuddy [25], Claude Code [2], and Codex [15]. Unless otherwise specified, they use their native default backbones: GLM-5.2, Claude Opus 4.8, and GPT-5.5, respec- tively. For each product, + skills adds the same base warehouse skill library, including identical skill interfaces, input schemas, exe- cutable implementations, and platform tools; + SiriusDeliver ad- ditionally enables hierarchical skill orchestration, artifact lifecy- cle control, and trace-driven skill evolution. Within each product pair, the backbone, product version, skill library, platform tools, execution environment, and refinement budget are fixed, so the paired difference measures the incremental benefit of SiriusDe- liver. Because products retain different native backbones and inter- nal agent implementations, their absolute results represent product- conditioned systems rather than isolated comparisons of agent frameworks or backbone capabilities. Metrics. We use five metrics. End-to-end delivery success rate is the fraction of cases producing platform-accepted and executable ware- house tasks whose artifacts, dependencies, and configurations pass manual verification. First-pass success rate is the fraction completed by the initial submission without refinement. Turns counts agent reasoning and action rounds; Time measures wall-clock duration from receiving the requirement to the final submission result; and Tokens counts total LLM input and output tokens. For each product or backbone pair, we report the absolute percentage-point success improvement after enabling SiriusDeliver. Because products may use different tokenizers and internal inference procedures, turns, time, and tokens are interpreted primarily within matched pairs rather than by their absolute values across products. Implementation and controls. Each case permits one initial sub- mission and at most two refinement attempts based on platform feedback. For fair paired comparisons, configurations share task inputs, platform context, permissions, skill access, and refinement budgets. Agent states and temporary execution contexts are reset between cases to prevent cross-case information carryover. To pre- vent leakage, evolution trajectories are disjoint from evaluation cases and the resulting skill snapshot is frozen before evaluation. The backbone study uses the same 200 cases and CodeBuddy run- time, varying only the underlying model. Additional isolation and backbone controls are provided in Appendix A.1. 4.2 Offline Evaluation End-to-end performance. Table 1 reports the end-to-end deliv- ery performance on 200 offline cases. SiriusDeliver consistently outperforms all code-agent baselines across the four task types in both delivery effectiveness and efficiency. Compared with the best baseline, SiriusDeliver improves the average end-to-end delivery success rate by 14.5 points (from 71.5% to 86.0%), with the largest per-type improvement of 16.0 points on offline synchronization and real-time computation tasks. Meanwhile, SiriusDeliver reduces the average number of turns from 8.4 to 5.4, shortens the end-to- end delivery time from 16.0 to 10.2 minutes, and decreases token consumption by 30% (from 124.0K to 86.9K). These results show that SiriusDeliver improves reliability without relying on larger reasoning budgets. The gain comes from structured skill orches- tration, which reduces missing-context and wrong-skill-selection errors, and artifact lifecycle control, which prevents invalid artifacts from being repeatedly submitted and repaired through costly trial and error. Among the four task types, offline computation obtains the lowest success rate because it often involves complex SQL or PySpark semantics, multi-table dependencies, historical workflow reuse, partition management, and scheduling configurations. Nev- ertheless, SiriusDeliver still achieves a success rate of 78.0% on challenging offline computation and maintains consistently lower SiriusDeliver: Automating Data Warehouse Delivery at TencentConference’17, July 2017, Washington, DC, USA Table 2: Paired evaluation of SiriusDeliver across representative LLM backbones on the same 200 cases (50 per task type), all run under the same CodeBuddy runtime. ∆ Success is the absolute gain of + SiriusDeliver over + Skills. BackboneConfigurationSuccess (%)↑∆ Success (p)↑First-pass (%)↑Turns↓Time (min)↓Tokens (K)↓ DeepSeek V4 FlashAgent + Skills58.0–42.011.221.8162.0 Agent + SiriusDeliver74.0+16.056.07.614.6116.0 GPT-5.5Agent + Skills68.0–54.09.217.6130.0 Agent + SiriusDeliver80.0+12.064.06.412.2100.0 Claude Opus 4.8Agent + Skills70.0–55.09.017.4132.0 Agent + SiriusDeliver84.0+14.068.06.011.495.0 GLM-5.2Agent + Skills68.5–53.09.117.6137.2 Agent + SiriusDeliver84.0+15.568.05.710.891.5 0 1 2 3 4 5 Tokens (M) 3.87 2.88 ↓25.5% 0 50 100 150 200 Turns 163 134 ↓17.5% OriginalEvolved 70.9% 8.3% 20.8% WinTieLose Figure 6: Effect of trace-driven skill evolution. Left: token usage and reasoning turns before and after skill evolution. Right: pairwise artifact quality comparison (Win/Tie/Lose) between the original and evolved skill libraries. overall turns, time, and token cost than the baselines, showing its effectiveness in the most difficult offline delivery setting. Effect of artifact lifecycle control. We evaluate artifact lifecy- cle control at both the pre-execution and post-execution stages. Pre-execution diagnosis detects 75.9% of real artifact issues with a false-positive rate of 9.5%, while introducing an average latency of only 6.1 s. Post-execution diagnosis directly repairs 73.0% of failed executions within one diagnosis-guided revision, with an average latency of 24.2 s. Together with the component ablations in Table 1, these results show that lifecycle control Artifact lifecycle control improves delivery reliability by blocking risky artifacts before sub- mission and converting a large fraction of execution failures into successful repairs, with modest runtime overhead. Effect of skill evolution. We evaluate trace-driven skill evolution, split on offline production delivery cases into 50% training cases for deriving skill refinements and 50% held-out test cases for eval- uation. For each held-out case, we replay the same request with the original and evolved skill libraries, and measure token usage, reasoning turns, and artifact quality. Artifact quality is assessed by pairwise LLM-as-judge comparison between artifacts generated before and after skill evolution, using position swapping and multi- model voting to reduce order bias and model-specific variance. The detailed judging protocol provided in Appendix A.3. As shown in Figure 6, evolved skills reduce token usage by 25.5% and reasoning turns by 17.5% on held-out cases For artifact quality, evolved arti- facts are preferred or judged comparable in 79.2% of cases. These results suggest that skill evolution substantially reduces execution overhead without measurable degradation in final artifact quality. Effect of different LLM backbones. To examine whether the effectiveness of SiriusDeliver generalizes across different LLM backbones, we conduct a paired evaluation using a controlled agent runtime. We adopt CodeBuddy as the agent runtime for this study, since it supports pluggable custom model backbones, whereas Codex and Claude Code do not expose such an interface to the best of our knowledge. For each backbone, we compare two con- figurations: an agent equipped with the shared warehouse skill library and the same agent augmented with the three components of SiriusDeliver. The backbone is fixed within each pair, while the runtime, tools, prompts, context budget, refinement budget, and execution environment are kept identical. We evaluate repre- sentative enterprise-approved models spanning different capability levels and model families. As shown in Table 2, enabling SiriusDe- liver consistently improves end-to-end delivery success across all evaluated backbones. The paired success gains range from 12.0 to 16.0 percentage points. Notably, weaker backbones benefit more from SiriusDeliver (e.g., a 16.0-point gain on DeepSeek V4 Flash versus 12.0 points on the stronger GPT-5.5), as the deterministic platform evidence and structured orchestration compensate for lim- ited reasoning capability. Since all backbones are evaluated under the same CodeBuddy runtime, the reported numbers are not directly comparable to the product-conditioned results in Table 1, which use each product’s native runtime and backbone. SiriusDeliver also reduces the number of reasoning turns, wall-clock time, and token consumption for all backbones. These results indicate that the effectiveness of SiriusDeliver does not depend on a particular LLM backbone and that its structured delivery workflow provides complementary benefits across different model capability levels. Ablation studies. On the best-performing product, Claude Code, we remove one SiriusDeliver component at a time while holding the others fixed; Table 1 shows that all three components contribute, with the full model corresponding to the Claude Code + SiriusDe- liver row. Removing hierarchical skill orchestration causes the largest degradation, an 11.5-point success drop (86.0%→74.5%), concentrated on synchronization tasks where missing metadata, workflow context, or dependency evidence leads to invalid plans. Removing artifact lifecycle control reduces success by 8.0 points (86.0%→78.0%), mainly on computation tasks with latent correct- ness and performance risks; its pre- and post-execution sub-ablations both contribute to this drop. Removing trace-driven skill evolution causes a relatively smaller 2.0-point success drop but increases total token consumption from 86.9K to 112.3K, showing that evolution primarily improves efficiency by compressing redundant context acquisition and intermediate reasoning. Conference’17, July 2017, Washington, DC, USAXie et al. 6 Business teams 3,600 Monthly active users 18,240 Delivery sessions 2 Months 0%25%50%75%100% 41%27%19%13% Offline comp. Offline sync. Real-time comp. Real-time sync. Delivery success Autonomous submission Lifecycle trigger 0% 25% 50% 75% 100% 87.2% 73.5% 61.0% Figure 7: Online production deployment of SiriusDeliver. Left: deployment scale and workload composition. Right: production delivery metrics. 4.3 Online Deployment We deployed SiriusDeliver on Tencent Cloud WeData and evalu- ated it in a real production environment. SiriusDeliver provides a web interface where engineers submit warehouse requirements, inspect generated artifacts, and track delivery progress; an example is shown in Figure 13 in Appendix C.2. The deployment covers 6 business teams and 4 warehouse task types: offline computation, offline synchronization, real-time computation, and real-time syn- chronization. These tasks support core business scenarios such as WeChat Pay, advertising, and security risk control. Online results. As shown in Figure 7, during a two-month deploy- ment period, SiriusDeliver served 3,600 monthly active users and supported 18,240 delivery sessions. The four task types account for 41%, 27%, 19%, and 13% of online requests, respectively. Across all sessions, SiriusDeliver achieved an end-to-end delivery success rate of 87.2% and an autonomous submission rate of 73.5%, where autonomous submission means that artifact generation, validation, and submission are completed without manual modification of delivery artifacts. Artifact lifecycle control was triggered in 61% of sessions: pre-execution diagnosis detected 4,120 risky artifacts before submission, and post-execution diagnosis repaired 2,890 platform execution failures. Trace-driven skill evolution produced 128 human-approved skill updates, including 89 repair-oriented updates and 39 compression-oriented updates. Based on 3,180 user ratings, SiriusDeliver received an average score of 4.4 out of 5.0. A/B test. We conducted a one-month A/B test on real online deliv- ery requests. We recruited 70 experienced data warehouse engineers who regularly perform warehouse task delivery and randomized them at the user level into a control group and a treatment group, with 35 users in each group. The randomization was stratified by task type and business team. The control group followed the existing manual workflow, while the treatment group used Sirius- Deliver. During the experiment, the treatment and control groups completed 718 and 682 delivery tasks, respectively. Since experienced engineers already achieve high final success rates, the A/B test focuses on delivery cost. As shown in Figure 8, compared with the control group, SiriusDeliver reduced the me- dian end-to-end delivery time from 228 minutes to 23 minutes, corresponding to an 89.9% reduction with a 95% confidence interval of [86.4%, 92.1%]. It also reduced net engineer effort from 95 to 11 minutes, time to first executable artifact from 44 to 2.6 minutes, and manual intervention rate from 100% to 21%. The treatment group required 3.2 interaction rounds on average, indicating that most sessions were completed without repeated clarification. Meanwhile, Delivery time Engineer effort First-artifact latency 0 50 100 150 200 250 Minutes 228 95 44 23 11 2.6 ↓89.9% ↓88.4% ↓94.1% Human Ours Manual intervention Final delivery success 0% 25% 50% 75% 100% Rate 100% 21% 98.1% 97.6% Figure 8: Online A/B test comparing the manual workflow and SiriusDeliver. Left: delivery efficiency metrics. Right: human intervention and final delivery quality. final delivery success rates remained comparable between the treat- ment and control groups, 97.6 % vs. 98.1%, suggesting that efficiency gains did not compromise production submission quality. Production insights. The deployment yields three insights. First, gains in first-artifact latency, engineer effort, and manual interven- tion show that delivery automation should coordinate the end-to- end delivery process rather than focus on code generation alone. Second, lifecycle control is essential for production readiness: its ac- tivation in 61% of production sessions confirms the practical need to verify artifacts before submission and ground repairs in execution feedback. Third, the 128 approved updates show that warehouse skills are continually evolving operational assets; repair-oriented updates improve robustness, while compression-oriented updates reliably remove redundant context and interactions. Case studies. Appendix B presents four cases that complement the aggregate results with production traces. In the pre-execution case, SiriusDeliver combines SQL semantics with table metadata to block an excessive partition scan after discovering approximately 1,179 TB in the first matching hourly partition. In the post-execution case, it grounds the repair of a PySpark runtime failure in execution logs, artifact context, and retrieved knowledge, then generates a revised artifact for safe retry. The skill-evolution case abstracts recurring empty-result and file-conflict failures into reusable pre- ventive rules. The end-to-end case follows an offline computation task from intent routing through artifact generation, validation, execution, and feedback-driven resubmission. These cases demon- strate how SiriusDeliver integrates generation, lifecycle control, and skill evolution into a closed-loop production workflow. 5 Conclusion We presented SiriusDeliver, an end-to-end production warehouse delivery system that goes beyond general coding agents through dependency-aware skill orchestration, lifecycle-aware artifact vali- dation and repair, and trace-driven skill evolution. Offline evalua- tions demonstrate improved delivery performance, effective diag- nosis and repair, and reduced reasoning overhead. A two-month deployment on Tencent Cloud WeData served 3,600 monthly active users and supported 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. A production A/B test substantially reduced delivery time and en- gineer effort while maintaining comparable final delivery success, demonstrating that production warehouse automation requires co- ordinated planning, explicit artifact control, and experience-driven skill maintenance beyond code generation. SiriusDeliver: Automating Data Warehouse Delivery at TencentConference’17, July 2017, Washington, DC, USA References [1]Anthropic. 2024. Introducing Claude 3.5 Sonnet. https://w.anthropic.com/ index/claude-3-5-sonnet. Accessed: 2026-06-30. [2]Anthropic. 2026. Claude Code overview. https://docs.anthropic.com/en/docs/ claude-code/overview. Accessed: 2026-06-30. [3]Michael Armbrust, Tathagata Das, Sameer Paranjpye, Reynold Xin, Shixiong Zhu, Ali Ghodsi, Burak Yavuz, Mukul Murthy, Joseph Torres, Liwen Sun, Peter Boncz, Mostafa Mokhtar, Herman Van Hovell, Adrian Ionescu, Alicja Luszczak, Michal Switakowski, Takuya Ueshin, Xiao Li, Michal Szafranski, Pieter Senster, and Matei Zaharia. 2020. Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores. Proc. VLDB Endow. 13, 12 (2020), 3411–3424. [4] Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al.2024. Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference on Computer Systems. 674–688. [5]Benoît Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, Allison W. Lee, Ashish Motivala, Abdul Q. Munir, Steven Pelley, Peter Povinec, Greg Rahn, Spyridon Triantafyllis, and Philipp Unterbrunner. 2016. The Snowflake Elastic Data Warehouse. In Proceedings of the 2016 International Conference on Management of Data. ACM, 215–226. [6] Databricks. 2026. Databricks assistant: your context-aware ai assistant. https: //w.databricks.com/product/databricks-assistant. Accessed: 2026-06-30. [7]DeepSeek-AI. 2026. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948 [cs.CL] doi:10.1038/s41586-025-09422-z [8]Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jin- gren Zhou. 2023. Text-to-sql empowered by large language models: a benchmark evaluation. Proceedings of the VLDB Endowment (2023). [9]Gemini Team. 2024. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530 [cs.CL] https://arxiv.org/abs/ 2403.05530 [10] GitHub. 2026. GitHub Copilot: your AI pair programmer. https://github.com/ features/copilot. Accessed: 2026-06-30. [11] Google Cloud. 2026. Write queries with gemini Aassistance in bigquery. https: //cloud.google.com/bigquery/docs/write-sql-gemini. Accessed: 2026-06-30. [12] Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, et al.2025. Data interpreter: an llm agent for data science. In Findings of the Association for Computational Linguistics: ACL 2025. 19796–19821. [13]Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shiv- akumar, Matt Tolton, and Theo Vassilakis. 2010. Dremel: Interactive Analysis of Web-Scale Datasets. In Proceedings of the VLDB Endowment, Vol. 3. 330–339. [14] OpenAI. 2024. GPT-4 technical report. arXiv:2303.08774 [cs.CL] https://arxiv. org/abs/2303.08774 [15] OpenAI. 2025. OpenAI codex. https://openai.com/codex/. Accessed: 2026-06-30. [16]Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large Language Model Connected with Massive APIs. International Conference on Neural Information Processing Systems (2024). [17]Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: decomposed in- context learning of text-to-sql with self-correction. International Conference on Neural Information Processing Systems (2023). [18] Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, Minghua Ma, Pu Zhao, Si Qin, Xiaoting Qin, Chao Du, Yong Xu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2023. Taskweaver: a code-first agent framework. arXiv preprint arXiv:2311.17541 (2023). [19]Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: facilitating large language models to master 16000+ real-world APIs. International Conference on Learning Representations (2024). [20]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: language models can teach themselves to use tools. International Conference on Neural Information Processing Systems (2023). [21]Yu Shen, Shiyang Liu, Qihang He, Yihang Cheng, Haining Xie, Zhiming He, Huahua Fan, Xianzhi Tan, Teng Ma, Shaoquan Zhang, et al.2026. SiriusHelper: an LLM agent-based operations assistant for big data platforms. arXiv preprint arXiv:2605.00043 (2026). [22]Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. International Conference on Neural Information Processing Systems (2023). [23]Snowflake. 2026. Cortex analyst. https://docs.snowflake.com/en/user-guide/ snowflake-cortex/cortex-analyst. Accessed: 2026-06-30. [24]Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755 (2024). [25]Tencent Cloud. 2026. Tencent cloud code assistant CodeBuddy: product overview. https://w.codebuddy.cn. Accessed: 2026-06-30. [26] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An Open-Ended Embodied Agent with Large Language Models. Transactions on Machine Learning Research (2023). [27]Weiyuan Xu, Juntao Luo, Tao Huang, Kaixin Sui, Jie Geng, Qijun Ma, Isami Akasaka, Xiaoxue Shi, Jing Tang, and Peng Cai. 2025. LogSage: an LLM-based framework for CI/CD failure detection and remediation with industrial validation. IEEE/ACM International Conference on Automated Software Engineering (2025). [28]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: agent-computer interfaces enable automated software engineering. In International Conference on Neural Information Processing Systems. [29]Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3911–3921. [30]Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. [31]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. ExpeL: LLM agents are experiential learners. Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence (2024). Conference’17, July 2017, Washington, DC, USAXie et al. A Experimental Details A.1 Detailed Evaluation Controls Within each product pair, both configurations are initialized from the same base warehouse skill library. Agent states and tempo- rary execution contexts are reset between cases, and no test-case outcome or platform feedback is used to update the skills or exe- cution policy of subsequent cases. Historical trajectories used for skill evolution are disjoint from evaluation cases, and the resulting skill-library snapshot is frozen before evaluation. For the backbone study, all LLMs are accessed through direct APIs and integrated into the same CodeBuddy runtime on the same 200 cases. The prompt template, skill-library snapshot, tool access, context budget, refinement budget, and execution environment are fixed within each pair; only the underlying model differs. Backbone selection criteria are described in Section 4.2. A.2 Private Benchmark Construction and Public Companion Benchmark Case collection and stratification. The private benchmark was constructed from production warehouse delivery requests collected from Tencent Cloud WeData during the first half of 2026. The unit of evaluation is one delivery request together with the platform con- text required to complete it, including the relevant table schemas, workflow metadata, task configurations, historical references, and execution permissions. We retained requests for which the required platform context and the final delivery outcome were available, and excluded incomplete, cancelled, and duplicated requests. Eligible requests were grouped into four task types: real-time synchroniza- tion, offline synchronization, real-time computation, and offline computation. For each task type, we applied stratified selection and manually curated 50 representative cases, yielding 200 cases in total. To reduce selection bias, the curated cases were chosen to span dif- ferent difficulty levels, artifact types (e.g., SQL, PySpark, and Flink SQL), and failure modes, rather than favoring easily solved requests. The same fixed cases and platform-context snapshots were used by all compared methods throughout all evaluations. Deduplication and leakage control. Before evaluation, we re- moved exact-duplicate delivery requests so that no request appears more than once in the benchmark. The historical delivery trajecto- ries used for trace-driven skill evolution are disjoint from the 200 evaluation cases. The skill-library snapshot, task inputs, metadata, and platform context were frozen before evaluation, and no evalua- tion outcome or platform feedback was used to update the skills or execution policy of subsequent cases. Success adjudication. A case is counted as successful only when the generated artifact bundle is accepted by the platform, is exe- cutable in the designated environment, and satisfies the required artifact, dependency, and configuration constraints. The final out- puts were independently verified by three experienced warehouse engineers using a predefined checklist, and disagreements were re- solved through adjudication. For each task type and configuration, we report integer success counts together with the corresponding success rates for cross-method performance comparison. Public companion benchmark. The private benchmark cannot be publicly released because the cases contain proprietary schemas, workflow configurations, platform metadata, operational traces, and business-sensitive information. To support public research on related data-engineering agent capabilities, we separately provide DataClawEval, a public companion benchmark of 100 executable tasks across PySpark, MySQL, HiveSQL, PrestoSQL, and Flink SQL. Each task provides a sanitized request, an initialized data environ- ment, a ground-truth solution, an automated grader, and a Docker- based execution harness. DataClawEval complements rather than reproduces the private benchmark: it covers offline and real-time computation tasks, but does not model WeData-specific synchro- nization, scheduling, permission, platform-submission, or lifecycle- diagnosis workflows. The DataClawEval tasks do not contain any of the 200 private evaluation cases and are not used to produce the results reported in this paper under any setting. A.3 LLM-as-Judge Protocol for Skill Evolution We use a pairwise LLM-as-judge protocol to assess whether skill evolution affects artifact quality. For each case, we compare two artifacts generated from the same request: one using the original skill library and the other using the evolved skill library. The judge is asked to select the better artifact according to functional correctness, dependency consistency, configuration validity, and production readiness. To reduce position bias, each case is evaluated twice by the same judge with swapped artifact order. If the two judgments are consistent, we record the result as a win for the corresponding artifact. For example, if the judge selects artifact퐴in both the original order and the swapped order, the case is recorded as an퐴 win for this judge. If the two judgments are inconsistent, the case is recorded as inconsistent for subsequent analysis. We use three LLM judges: Gemini 3.8 Flash, GPT-5.5, and Claude Opus 4.8. Each judge independently produces one of three outcomes for each case:퐴win,퐵win, or inconsistent. We then aggregate the three judge outcomes by majority voting. If two or more judges select the same artifact, the result is recorded as a win for that artifact. For example, outcomes (퐴,퐴,퐵) are recorded as an퐴win. If no artifact receives at least two votes, or if the votes are dominated by inconsistent outcomes, the result is recorded as inconsistent. For example, outcomes (퐴,퐵, inconsistent) are recorded as inconsistent. B Case Studies B.1 Case Study of Pre-execution Diagnosis Figure 9 presents a case in which a syntactically valid SQL query remains unsafe for production execution. The query contains two overlapping lower-bound predicates onpartition_time, whose conjunction is equivalent topartition_time >= 2026062423. Predicate simplification alone does not reduce the effective scan range. Pre-execution diagnosis further retrieves table metadata and finds that the first matching hourly partition alone contains approximately 1,179 TB of data. Combined with the subsequent GROUP BYaggregation, this introduces a high risk of excessive shuffle, memory pressure, and OOM failure. Since no semantics- preserving rewrite can sufficiently reduce the scanned data, lifecycle control removes the redundant predicate but does not approve the artifact for direct submission. Instead, it recommends narrowing the time range, batching the query by time or business dimensions, or revising the table partition strategy. SiriusDeliver: Automating Data Warehouse Delivery at TencentConference’17, July 2017, Washington, DC, USA Table 3: Diagnosis coverage in SiriusDeliver. Pre-execution diagnosis detects risks before task submission, while post- execution diagnosis structures feedback from platform logs. StageIssue typeMain signalRisk or feedback Pre-execution Syntax errorSQL parser or compiler evidenceInvalid SQL or dialect-specific misuse Brute-force scanMissing or ineffective partition pruningExcessive scan cost and task delay Cartesian productMissing or incomplete join conditionsData explosion and resource waste Implicit conversionType mismatch in filters, joins, or expressionsSemantic drift or pruning failure Empty-table readEmpty source table or missing partitionSilent empty outputs or zero metrics Basic logic defectAggregation, join, null, or dedup logicSilent data distortion Where-clause logic defectBoundary, null, or predicate logicIncorrect filtering results Business-intent mismatchSQL semantics vs. user intentCorrect execution but wrong business meaning Post-execution Syntax exceptionParse, alias, function, join, or subquery errorsRoot cause for failed compilation/execution UDF exceptionUDF runtime, dependency, or registration errorsUDF repair or dependency fix Permission exceptionAccess-denied or unauthorized logsPermission application or policy check System environment exceptionTimeout, OOM, network, or service failuresRuntime or resource-level repair Data-source read exceptionMissing file, corrupted data, or source failureSource availability or format repair Table-structure exceptionMissing table, column, partition, or schema mismatchMetadata or schema repair Input: Candidate Artifact select context.sdk_version,count(0) as cnt from ams_log_iceberg::etl_pageview_mixer where partition_time>=2026062422 and partition_time>=2026062423 and site_set= 15 and action_info.request_info.ad_pull_mode=1 and site_info.site_id=80503886999934 group by context.sdk_version Status: Syntax-valid but not yet approved for submission. Evidence Grounding Predicate simplification: partition_time≥2026062422 ∧partition_time≥2026062423 ≡partition_time ≥2026062423 Platform metadata: The table is hourly partitioned, and the first matching partition alone contains approximately 1,179~TB of data. Execution operators: Large partition scan followed by a GROUP BY aggregation. Hybrid Verification •The redundant predicate does not restrict the effective scan range. •The first matching partition alone contains = 1,179 TB •GROUP BY increases shuffle and memory pressure. •No semantics-preserving rewrite can sufficiently reduce the scanned data. Candidate Revision Excessive Partition Scan (High Risk of OOM Failure) The redundant predicate is removed while preserving the original semantics. Because the resource risk remains unresolved, direct submission is blocked. The user is advised to narrow the time window, batch the query by time or business dimensions, or revise the table partition strategy. Case: Pre-execution Diagnosis for an Excessive Partition Scan Figure 9: Case study of pre-execution diagnosis for an ex- cessive partition scan. By combining SQL predicate analysis with table metadata, SiriusDeliver identifies a high risk of resource exhaustion, blocks direct submission, and returns a semantics-preserving revision with mitigation guidance. This case illustrates that syntax validation alone is insufficient for production warehouse delivery. Reliable diagnosis requires jointly reasoning over query semantics, table metadata, and execution characteristics, while avoiding unsupported rewrites when the un- derlying risk cannot be safely resolved. B.2 Case Study of Post-execution Diagnosis Figure 10 presents a case in which a PySpark task fails during runtime with anIndexOutOfBoundsException. The failure log re- ports an invalid access to index 106 from a container with size 0 during ORC data reading. Post-execution diagnosis first extracts localized failure signals from execution logs and retrieves relevant troubleshooting knowledge from historical cases. Based on the arti- fact context, failure signals, and retrieved knowledge, the diagnosis examines potential causes including corrupted ORC files, invalid reading logic, and empty input partitions.The evidence indicates that abnormal ORC data is the most likely root cause. The system then generates repair guidance to replace corrupted files, validate partition inputs, and add safeguards against empty data access. Us- ing the localized evidence and repair guidance, an LLM revises the original artifact for the next execution attempt. This case illustrates that execution logs alone are insufficient for reliable failure repair. Effective post-execution diagnosis requires combining platform feedback, artifact context, and historical knowl- edge to localize root causes and generate grounded revisions rather than directly modifying artifacts based on raw errors. B.3 Case Study of Trace-driven Skill Evolution Figure 11 presents a case in which execution traces are converted into reusable skill updates. The trajectories contain two recurring failures: an empty DataFrame beforeinsertInto()that later trig- gers aNoSuchElementException, and anupload_file(’etl_task.py’) operation that fails because the target file already exists. Rather than treating them as isolated fixes, trace-driven skill evolution abstracts them into auditable, reusable delivery rules across simi- lar tasks, including empty-result checks before writes, unique file naming, and platform-side existence checks before upload. When the evolved skill is reused, these failures are prevented before platform submission. The agent detects empty intermedi- ate results, revises the transformation before writing, and selects non-conflicting filenames after checking existing platform files. This case illustrates how SiriusDeliver turns failed trajectories Conference’17, July 2017, Washington, DC, USAXie et al. Case: Post-execution Diagnosisfor Failed PySparkTask Input: Platform Feedback ERROR Executor: Exception in task 1453.0 in stage 32.0 (TID 1453) Java.lang.IndexOutOfBoundsException: Index: 106. impl.RecordReaderInpl.next(RecordReaderInpl.java: 106) mapred.OrcRecordReader.next(OrcRecordReader. java: 215) FileScanRODSSanon$1.hasNextFileScanROD.scala:116) Failure Extraction •The task fails with an IndexOutOfBoundsException. •Index 106 is accessed from a container of size 0. •The failure occurs during PySparkORC data reading. •Raw logs are retained as auxiliary context. Knowledge Retrieval •Historical cases associate this failure with corrupted ORC files. •Similar errors may also arise from out-of-bound index access. •Empty or invalid partitions can produce zero-sized inputs. •Troubleshooting documents provide corresponding repair procedures. Root-cause-based Revision •Corrupted ORC files should be repaired or replaced. •Empty inputs and out-of-bound accesses should be explicitly guarded. •Partition fields should be validated to avoid invalid or empty partitions. •The original artifact is revised using the evidence and repair guidance. Output •A revised artifact is generated for the next execution attempt. •The output includes the exception category, root-cause explanation, and actionable repair guidance. Figure 10: Case study of post-execution diagnosis for a PyS- park runtime failure. By combining execution log analysis with diagnostic knowledge, SiriusDeliver identifies the root cause, provides actionable repair guidance, and generates a revised artifact for subsequent execution. into preventive execution knowledge, reducing repeated delivery failures through reusable skill refinement. B.4 Case Study of End-to-end Delivery Figure 12 presents an end-to-end delivery case for an offline compu- tation task. The user requests a daily detail table for cross-day Ray notebook executions, requiring day-boundary splitting, runtime calculation, minute-level GPU metrics, and insertion into the target partitiondt=’20260507’. The router identifies the request as a new offline SQL task, selects Spark-compatible SuperSQL, and orches- trates schema retrieval, preview-data acquisition, SQL generation, validation, resource recommendation, and platform operations. Based on the resolved context, SiriusDeliver generates a work- flow specification, SuperSQL code, and task configuration. Pre- execution diagnosis checks schema consistency, partition seman- tics, referenced fields, target mappings, and submission settings before the artifact is uploaded to WeData for trial execution. When the trial exposes a column-count mismatch and invalid arithmetic overSTRINGvalues, post-execution diagnosis removes the parti- tion column from theSELECTprojection, adds explicit casts, and resubmits the revised artifact. This case illustrates how SiriusDe- liver connects intent routing, context planning, artifact generation, lifecycle diagnosis, platform execution, and feedback-driven repair into a single production delivery loop. Case: Trajectory-Driven Skill Refinement and Reuse Input: A Group of Trajectories Case 1: DWS aggregation task failed. Error on insertInto() With empty data Tool Call: ‘spark.sql(...)’(0 rows returned) Log: NoSuchElementException: None.get Case 2 : DWD cleaning task submission failed. File name conflict on platform Tool Call:‘upload_file(‘etl_task.py’)’ Log: Error 8001 -File ‘etl_task.py’exists. Experience-guided Skill Refinement LLM Refinement Output(Key Changes): 1.Stronger Defensive Code Standard: ->Add mandatory empty DataFramechecks before write operations. 2.Unified Naming Strategy Rules: ->Enforce unique naming with timestamp or task-id suffix. 3.Pre-check Guidance: ->Add file existence check before upload. Next Execution With Updated skill (One-Shot Success) Case 1: DWS aggregation task success. Tool Call: spark.sql(...) (0 rows returned) Log: Pre-check, Empty DataFramedetected. Tool Call: revise_transformation(...) Tool Call: spark.sql(...) (128734 rows returned) Log: Write completed successfully. Case 2: DWD cleaning task submission success. Tool Call: platform.list_files()(result: ["etl_task.py", "daily_job.py", ...]) Tool Call: upload_file("etl_task_20260723.py") Log: Submission completed successfully. Figure 11: Case study of trajectory-driven skill refinement and reuse. C Details C.1 Agent and skills The delivery automation agent operates over a reusable library of warehouse skills. A skill is the basic executable unit for warehouse delivery, consisting of (1) a description of its functionality and applicable scenarios; (2) a specification of input arguments extracted from the user request or context; (3) execution code, which may involve LLM calls, platform APIs, retrieval modules, or deterministic scripts; and (4) optional dependencies that determine execution order. A skill may emit structured planning hints (e.g., inferred scenarios, remaining prerequisites, or downstream candidates) to support subsequent planning. C.2 Web Interface Figure 13 shows the production interface of SiriusDeliver, where engineers submit warehouse requirements, inspect generated arti- facts, and track delivery progress. The interface provides a unified workspace for initiating new delivery conversations, revisiting his- torical sessions, and selecting common warehouse task scenarios. Within each session, engineers can review the generated SQL, work- flow configurations, and diagnostic feedback, provide clarifications when required information is missing, and monitor the status of validation, submission, and subsequent execution. This interface op- erationalizes the human-in-the-loop delivery workflow by keeping requirement interaction, artifact inspection, and platform feedback within a single production-facing entry point. SiriusDeliver: Automating Data Warehouse Delivery at TencentConference’17, July 2017, Washington, DC, USA 1. Business Intent and Task Routing Business Requirement: Task Router: New offline SQL task using Spark-compatible SuperSQL. •Generate a daily detail table for cross-day Ray notebook executions. Split each execution at natural-day boundaries, calculate instance runtime, code runtime, and resource waiting time, attach minute-level GPU metrics, and write the results into partitiondt='20260507’ 3. ArtifactGenerationand Configuration Executable Artifact: SuperSQL is generated to split cross-day intervals, aggregate execution durations, resolve service-to-pod mappings, and join minute-level GPU metrics. Task Configuration: A new single-node offline task is configured with the Spark runtime, daily schedule, retry policy, execution resource, and static target partition. Candidate Bundle: Workflow specification + SuperSQL code + task configuration. INSERT OVERWRITE TABLE target PARTITION (dt='20260507') WITH daily_split_spansAS (...), trace_time_metricsAS (...) SELECT ... 4. Pre-execution Diagnosis Evidence Grounding: The generated code is checked against table schemas, preview data, partition semantics, and the task configuration. Hybrid Verification: Syntax, referenced fields, target mapping, and submission settings are jointly verified. Decision: Pass：Submit for a platform trial. Fail：Revise and repeat pre-execution diagnosis. 5. Platform Trial and State Polling Platform Submission: The code is uploaded to WeData, a new offline task is created from the validated configuration, and a trial execution is submitted. States & Traces: SiriusDeliver continuously polls the platform execution state after task submission. COMPLETED → Request publication confirmation. FAILED → Trigger post-execution diagnosis. RUNNING → Continue polling. 6. Post-execution Diagnosis Failure Extraction: Platform feedback reports a column-count mismatch and invalid arithmetic on STRING values. Knowledge Retrieval: Relevant historical cases, diagnostic rules, and troubleshooting knowledge are retrieved. Root-cause-based Revision: Remove dt from the SELECT projection, explicitly cast replicas and num_gpu, and resubmit the artifact. Subsequent Execution: The revised artifact is revalidated, resubmitted, and monitored again. 2. Context Planning and Skill Orchestration Context Acquisition: Schemas and previews are retrieved for five source tables and the target table. Field definitions and example values are combined into Light Schemas. Skill Orchestration: The delivery agent invokes metadata retrieval, SQL generation, resource recommendation, validation, and platform-operation skills according to their prerequisites. Resolved Context: Source fields, partition semantics, table mappings, execution identity, and target write mode. Case: End-to-end Delivery of an Offline Computation Task Figure 12: Case study of end-to-end delivery of an offline computation task. Figure 13: Production web interface of SiriusDeliver.

Instant research discovery