Paper deep dive

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

Shunfan Zheng, Dongsheng Shi, Yue Li, Xin Yi, Linlin Wang, Gerard de Melo

Year: 2026Venue: arXiv preprintArea: cs.DBType: PreprintEmbeddings: 47

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

45,731 characters extracted from source content.

Expand or collapse full text

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks Shunfan Zheng 1 , Dongsheng Shi 1 , Yue Li 1 , Xin Yi 1 , Linlin Wang 1 * , Gerard de Melo 2 1 East China Normal University 2 Hasso Plattner Institute/University of Potsdam sfzheng, dongsheng, yue_li, xinyi@stu.ecnu.edu.cn, llwang@cs.ecnu.edu.cn, gdm@demelo.org Abstract Large Language Models (LLMs) are transform- ing database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, cur- rent evaluation benchmarks remain dispropor- tionately fixated on Text-to-SQL tasks, neglect- ing the holistic Database Lifecycle-from ini- tial schema design to post-deployment main- tenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, ad- dressing the cognitive mismatch between am- biguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate bal- anced performance, specialized Text-to-SQL models suffer from “catastrophic forgetting” in non-coding phases like design and mainte- nance. DBLifeBench serves as a foundational step toward evaluating and building true full- stack database intelligence. 1 Introduction Large Language Models (LLMs) have demon- strated remarkable capabilities in processing struc- tured data, positioning them as potential successors to traditional database interfaces (Ruan et al., 2023; Kong et al., 2023). Beyond simple query genera- tion, LLMs have increasingly exhibited the poten- tial to serve as autonomous agents capable of plan- ning and executing complex tasks (Shi et al., 2026; Chen et al., 2026; Zhang et al., 2026b), including managing the entire database pipeline (Zhang et al., * Corresponding Author. idtitleauthor... Table 1: books idnameaddress Table 2: users ... idbook_iduser_id Table n: loans ... ... I want to build an SQLite database for a library management system. The database should be able to record information such as book loans ... ... Database: Library DBLifeBench Spider/BIRD Designa library database management system. Implementthe DB based on the designed structure. Text-to-SQLwhich members have not returned ...? You're a CPU agent; call the API to maintainthe DB. Text-to-SQL which members have not returned ... Missing 1 Missing 2 ComprehensiveEvaluation Single Task Evaluation Design ... Implement ... Missing N ... maintain ... ... ... Figure 1: Comparison between traditional Text-to- SQL-centric benchmarks and our holistic DBLifeBench. While existing benchmarks focus solely on query trans- lation, DBLifeBench covers the full spectrum of a DBA’s workflow. 2026a; Zhou et al., 2024b; Li et al., 2024). How- ever, as LLMs are increasingly deployed in com- plex, high-stakes environments, the need for a rig- orous, multifaceted evaluation framework has be- come paramount. High-quality benchmarks serve not only as evaluation tools but as a strategic “moat” that drives the evolution of model capabilities, pro- viding the reference baseline essential for building robust database products (Schmidt et al., 2025). Unfortunately, current benchmarks fail to fulfill this role. Benchmarks such as Spider (Yu et al., 2018) and BIRD (Li et al., 2023) predominantly focus on the Text-to-SQL task (Hong et al., 2024; Yaghmazadeh et al., 2017; Xu et al., 2017; Zelle and Mooney, 1996). This narrow focus renders them insufficient as reference baselines for enhanc- ing real-world database product capabilities. While valuable, relying solely on them creates a “tunnel vision” effect. Consider a real-world scenario: a database administrator (DBA) for a library system must first design a normalized schema to avoid re- dundancy, implement it with correct constraints, and continuously maintain it by analyzing error logs. A model that excels at generating SQL SE- LECT statements (Text-to-SQL) but fails to design arXiv:2608.03794v1 [cs.DB] 4 Aug 2026 a viable schema or debug a deadlock is insufficient for real-world deployment. Optimizing models purely for existing benchmarks yields diminishing returns for actual product utility, as metrics like Execution Accuracy (EX) fail to assess structural and operational reasoning capabilities. To address these limitations, we introduce DBLifeBench, a comprehensive benchmark de- signed to evaluate the “Full-stack Database Intel- ligence” of LLMs. DBLifeBench extends beyond query translation to cover five key phases of the database lifecycle: design, implementation, oper- ation, debugging, and maintenance. Each phase features tailored tasks and metrics, ensuring a mul- tidimensional assessment of whether a model can truly function as a database expert. Moreover, within the Operation phase, we iden- tify a fundamental cognitive gap in traditional Text- to-SQL: complex SQL logic is often too intricate to be mapped directly from a single natural language sentence. To mitigate this, we propose Progressive- Text2SQL. This task utilizes a Dynamic Reasoning Graph to decompose complex queries into inter- mediate logical steps, effectively simulating the iterative reasoning process of human experts. This structure not only improves model performance but also aligns evaluations with genuine cognitive workflows. In summary, our contributions are the follows: •We propose DBLifeBench, the first bench- mark to evaluate LLMs across five critical phases of the database lifecycle, moving be- yond the single-task paradigm. •Specifically,we introduce Progressive- Text2SQL, a task that leverages reasoning graphs to bridge the gap between natural language ambiguity and SQL complexity, offering a more robust evaluation of reasoning capabilities. •Our comprehensive analysis reveals the phe- nomenon of the “curse of specialization”, whereby models fine-tuned specifically for SQL generation exhibit significant perfor- mance degradation in broader database man- agement tasks. 2 The Proposed Benchmark 2.1 The Composition of DBLifeBench To comprehensively assess Database Intelligence, DBLifeBench maps the database lifecycle to dis- tinct cognitive capabilities of LLMs. We devise assessments across five phases: Design (P1: Abstraction & Modeling) This phase evaluates the model’s ability to translate ab- stract requirements into structured schemas. Given natural language requirementsQ, the model must construct a schemaS = (E,D,K), comprising entitiesE, data typesD, and foreign key dependen- ciesK. This tests the model’s high-level logical modeling capabilities. Implementation (P2: Syntax Precision) Mod- els must translate the conceptual schemaSinto executable SQL Data Definition Language (DDL) statementsY. Unlike semantic generation, this phase demands strict syntactic precision to de- fine tables, columns, and constraints that are valid within a specific database engine (e.g., SQLite). Operation (P3: Logic Translation & Reason- ing) This phase focuses on converting opera- tional needs into Data Manipulation Language (DML). It includes standard Text2SQL and our novel Progressive-Text2SQL. For a query or graph nodeQ i , the model generates satisfying SQLY i . This tests the model’s ability to map natural lan- guage intent to database logic. Debugging (P4: Diagnosis & Correction)Sim- ulating a developer’s workflow, the model receives a flawed SQL statementY flawed and an error mes- sage. It must identify the root cause-whether syn- tactic or semantic and generate a corrected state- mentY corrected . Maintenance (P5: System Management) This phase simulates the role of a Site Reliability En- gineer (SRE) or DBA. It consists of: (i) Triage (Assigner): Analyzing error logs to assign specific experts (e.g., IOExpert); (i) Resolution (Expert): Utilizing simulated API tools (e.g.,check_metric) to diagnose and fix system anomalies. This tests the model’s domain knowledge and tool-use capa- bilities. Figure 7 provides specific examples. 2.2 Benchmark Construction 2.2.1 Overview DBLifeBench integrates diverse data sources, in- cluding: S#1, existing open-source datasets; S#2, manually constructed data; S#3, authoritative text- books, and S#4, automatically generated data. The entire corpus includes 13 distinct databases, with an average of 7.8 tables per database, covering diverse database ### Requirements Analysis for the “schools” Table ...... Column name Data format Key typeDescription ...... Table 1: school Table 2: x ... Table n: x (1) Design Database (2) Implement Requirements exec (4) Debugging User (5) Maintenance (3) Operations SQL: SELECTSchool ID from school where ... Error Info: syntax error ... LLM Output: CREATE TABLE `school` (...) School-ID “SchoolID” True False Alerts Assigner IO Expert CPU Expert Memory Expert IO APIs: ➢io_match_diagnose_knowledge ➢... Input schools status type TEXT school id INTEGER address TEXT school id INT Primary Key ... schools status type TEXT author TEXT school id TEXT prediction ground truth Table Name Data Type Key Type Column What is the most popular school in ... SELECT School ID from school where ... Question (Text) Answer (SQL) Text2SQL CPU APIs Memory APIs UPDATE x (SELECT x) x Union x From x Progressive-Text2SQL (P-Text2SQL) Complex sql statements 1 SELECT 2 Update 3 (SELECT, Update,) 4 (SELECT, From,) 5 (SELECT, Union,) 6 (SELECT, Update, Union, From) Graph Construct Graph Node --- Edge Manual Verification Database Verification Graph Verify Error Detection Self- Correction Contextual Understanding Logical Reasoning Data Modeling Instruction Following Evaluation Capabilities Figure 2: An Overview of DBLifeBench. The left part illustrates the lifecycle-based evaluation, while the right part details the data construction process for the Progressive-Text2SQL task. domains such as finance, education, and sports. We utilize this corpus to construct five datasets (shown in Table 1) for the five core database scenarios introduced in Section 2.1. Specific examples are provided in Appendix A.1. PhaseDataset# ExamplesSources P1 Requirements265S#1; S#2 P2 P3 Text2SQL1,534S#1; S#3 P-Text2SQL1,149S#1; S#3; S#4 P4SQL-Debugging523S#1; S#2; S#4 P5Maintenance518S#1 Table 1: Dataset Statistics Requirements To ensure high-quality design prompts, we employed a human-AI collaborative loop. The process starts with AI reviewing the schema and generating a description. Three stu- dents then draft the initial requirements analysis based on the schema and AI description. AI ana- lyzes this version, suggests improvements, and the students revise it. This iterative process is repeated for three rounds. Text-to-SQL & Progressive-Text2SQL We in- corporate BIRD (Li et al., 2023) for standard eval- uation and construct a new Progressive-Text2SQL dataset to address the “reasoning hop” challenges found in complex queries. Using the Dynamic Pro- gressive Reasoning Graph method, we dynamically construct progressive reasoning graphs, detailed in Section 2.2.2. To ensure data diversity, we also manually include other statements like UPDATE and DELETE, while prior work focuses on SE- LECT (Yu et al., 2019; Lei et al., 2024). SQL-Debugging Consulting BigTable (Zhang et al., 2024), we collect erroneous SQL statements and their corresponding error messages during the execution of Text-to-SQL tasks. Based on this, we construct the SQL Debugging dataset. These erro- neous SQL statements cannot be executed properly in SQLite and require correction. MaintenanceBased on the DB-GPT framework (Zhou et al., 2024a), we divide tasks into assign- ment and expert execution. Detailed examples are provided in Figure 7. Automated techniques like regular expression matching and database entity recognition were used for filtering, followed by ex- ecution and validation in a database environment to ensure effectiveness. 2.2.2 Progressive-Text2SQL Standard Text-to-SQL tasks often suffer from a misalignment between the brevity of natural lan- guage and the complexity of the target SQL. To address this, Progressive-Text2SQL transforms the task into a structured reasoning process (Figure 2, right). SQL Reasoning Graph We define a reasoning graphG = (V, E)to serve as a cognitive scaffold for the model. NodesV =v 1 , . . . , v n represent sub-tasks (e.g., “filter by date” or “join tables”), where each node contains a localized natural lan- guage description and its corresponding SQL frag- ment. EdgesErepresent logical dependencies, guiding the model to build the final queryv n in- crementally. This structure mimics the “Chain-of- Thought (CoT)” process of human experts. Graph ConstructionThe data construction pro- cess is divided into three parts: (i) We collect ex- amples from Text2SQL that are more difficult and where the misalignment between the natural lan- guage description and SQL has been manually iden- tified. (i) To mitigate potential bias from relying on a single model, we construct the reasoning graph using a diverse set of large language models, in- cluding GPT-4o, Claude, Gemini, and DeepSeek- R1. Each model independently generates candidate graph structures based on complex SQL queries. We also provide a few manually written examples in the prompt to assist the LLM in understanding and generating the graph. The specific process is detailed in Appendix A.2. (i) Based on the constructed SQL reasoning graph, we then use the LLM to generate the corresponding natural lan- guage description for each SQL node in the graph. The natural language description and SQL from the original example are provided in the prompt. Data Validation We employ a two-stage valida- tion strategy. First, all SQL nodes are executed in an SQLite environment, and those failing to exe- cute are discarded. Following this, three graduate students with backgrounds in database systems in- dependently reconstruct graph edges using only the SQL and natural language descriptions for each node, without seeing the original model-generated edges. We then compare their annotations with the model-generated structure using Jaccard similarity. Only examples achieving more than 80% consen- sus are retained. To quantify inter-rater reliability (Artstein, 2017), we calculate Fleiss’ kappa, ob- taining a score of 0.8173, which indicates strong agreement. Annotators are trained with represen- tative edge construction examples and participate in weekly calibration discussions. Additionally, all annotation disagreements are logged and analyzed to identify common error patterns such as alias con- fusion and aggregation mismatches, which in turn inform future refinements to the data construction pipeline. 3 Evaluation Metrics As shown in Section 2.1, the evaluations in DBLifeBench vary across different phases, neces- sitating a tailored set of assessment metrics based on the unique characteristics of each task. (1) For the design phase: we assess whether the model’s designed database structure is reasonable in this phase. We establish three evaluation metrics: entity name accuracy (ACC i -Entity), entity data type accuracy (ACC i -Data), and foreign key depen- dency accuracy (ACC i -Key).idenotes the number of output tables, defaulting to 2. ACC i -Entity can computed by: ACC i -Entity = P T t=1 E t ∩ ˆ E t P T t=1 |E t | , (1) whereTis the total number of tables, ˆ E t represents the ground-truth set of entity names for tablet, andE t represents the set of entity names predicted by the model for tablet. Analogously, the same approach is followed for the other metrics. (2) For the implementation phase: in this phase, the model outputs the creation statements for database tables. We execute these statements and evaluate theTresulting tables using table-level accuracy (T-Level) and field-level accuracy (F- Level):                T-Level = 1 T T X t=1 I(N t , ˆ N t ) F-Level = P T t=1 F t ∩ ˆ F t P T t=1 ˆ F t (2) Here,N t represents the predicted table name for ta- bletandF t represents the set of fields for predicted tablet. ˆ N t and ˆ F t represent the corresponding ground truth. Finally,I(·)is an indicator function that yields 1 wheneverYand ˆ Yare semantically consistent (not necessarily identical in name), and 0 when Y and ˆ Y are not semantically consistent. (3) For the operation phase: in the case of the Text2SQL task, we adopt the widely used Execu- tion Accuracy (EX) metric, which calculates accu- racy by comparing the execution results ˆ V n of the n-th predicted SQL statement with the execution results V n of the ground truth SQL: EX = 1 N N X n=1 I(V n , ˆ V n )(3) For the Progressive-Text2SQL task, we apply EX for each node, and define the Graph Execution Accuracy (G-EX) as the average of the node accu- racies, i.e. G-EX = 1 G G X g=1 EX g ,(4) Model DesignImplementation ACC 2 -EntityACC i -DataACC i -KeyAVGT-levelF-levelAVG General GPT-4o76.9465.9780.9974.6385.7177.3581.53 GPT-4o-mini69.3055.8479.9568.3679.0371.3075.17 Llama364.9050.2073.1462.7470.4168.5971.00 Mistral41.9931.7239.1637.6256.0839.2347.66 DeepSeek33.4625.9637.5932.3442.8832.5537.72 Qwen2.566.3153.8067.7162.6180.3965.8173.10 ChatGLM-467.0852.4872.6264.0678.7169.2473.98 Specialized DeepSeek-Coder66.7856.6274.8466.0878.2864.1871.23 SQLCoder52.7940.6059.1950.8610.007.358.68 CodeQwen69.2459.4575.3368.0073.8661.8367.85 Llama3-sqlcoder45.9535.4546.4042.6038.4529.1733.81 Model OperationDebuggingMaintenance Text2SQLP-Text2SQLSingle-roundMulti-roundAssignerExpertAVG General GPT-4o56.1953.7968.3972.1779.3833.5056.44 GPT-4o-mini47.7254.3160.4461.8372.8833.5853.23 Llama330.5145.6338.0443.4259.3232.7761.55 Mistral19.8228.5523.6236.8246.473.5825.03 DeepSeek17.5426.8123.0927.8153.3931.2542.32 Qwen2.538.0139.0835.3842.1167.8032.2550.02 ChatGLM-435.9846.1330.7840.9846.8933.1740.03 Specialized DeepSeek-Coder39.3147.0041.5350.6945.4833.5039.49 SQLCoder24.6429.8525.7431.890.000.000.00 CodeQwen37.3541.2221.9923.3235.3132.2533.78 Llama3-sqlcoder33.2539.4348.1862.5246.0532.9239.48 Table 2: Main Results on DBLifeBench whereGis the number of reasoning graphs, and EX g represents the Execution Accuracy of theg-th graph. (4) For the debugging phase: we execute the cor- rected SQL generated by the model on the database and evaluate the model’s error correction ability based on the execution results. (5) For the maintenance phase: we evaluate the correctness of expert assignments (by the Assigner) and API calls (by the Expert). 4 Experiments 4.1 Baseline Models We evaluate the performance of two classes of ad- vanced baseline models in DBLifeBench. The first class consists of general models, including Open- AI’s GPT series (GPT-4o, GPT-4o-mini), Llama3, Mistral, DeepSeek, Qwen, and ChatGLM-4. The second class comprises specialized models that have been fine-tuned on SQL, code, and related data, including DeepSeek-Coder, SQLCoder, Code- Qwen, and Llama3-Coder. 4.2 Experimental Setup We run our extensive experiments using the SQLite database, consistent with previous work such as BIRD (Li et al., 2023). During inference, we con- sistently set the temperature to 0.3 and the top-p to 0.2 to ensure a balance between diversity and coherence in the generated outputs. 4.3 Main Results We conducted a comprehensive evaluation of 11 LLMs using our DBLifeBench, examining their performance across five different phases of the database lifecycle. The experimental results are presented in Table 2. GPT-4o and GPT-4o-mini generally outperform other models across almost all aspects, demonstrating their comprehensive database capabilities. However, certain models show varying performance in different phases of evaluation, underscoring the need for a more com- prehensive evaluation of different phases. For in- stance, ChatGLM-4 excels in the Design and Im- plementation phases, but performs poorly in the Operation and Maintenance phases. Some mod- els, such as SQLCoder, struggle significantly in the Implementation and Maintenance phases, likely due to their training data being heavily focused on Text2SQL tasks, which adversely affects their performance in other areas. Specialized models struggle with “Full-stack” tasks. DeepSeek, after fine-tuning with code data, shows significant improvements across the lifecy- cle evaluation, while Llama3, fine-tuned on SQL data, experiences a decline in the Design and Im- plementation phases. Task-specific fine-tuning can sometimes cause models to over-focus on particular patterns, impairing their generalizability to other types of data. In contrast, general-purpose models appear less affected by this overfitting and can bet- ter cope with variation. Additionally, the task itself may have scale dependencies: As shown in Section 5.5, when the number of input tables is low, the task is simpler, and general models might perform better since specialized models’ fine-tuned features may not offer significant advantages. Some models lack the ability to perform multi- round debugging. Models such as Llama3 and Mistral show significant improvements in multi- round debugging compared to single-round debug- ging, indicating that they are better at recognizing and correcting their previous mistakes. In contrast, DeepSeek lacks this capability, as its performance in the multi-round setting is consistent with that of single-round debugging, showing no improvement. Model performance is different in the two maintenance tasks. As Assigners, models like GPT-4o and Qwen are adept at discerning error types from logs, but others are not. This reflects dif- ferences in the models’ ability to perceive database contexts and the richness of their database knowl- edge. As Experts, all models exhibit similar perfor- mance and obtain low scores because they are not good at using database tools. 5 Analysis 5.1 Impact of Data Model Variations To assess the impact of schema variation on LLMs’ database performance, we conducted experiments using three schema variants: v1 (lowest normal- ization), v2, and v3 (highest normalization), all sourced from (Fürst et al., 2024) and migrated to SQLite. These are evaluated on Text2SQL and P-Text2SQL tasks. Table 5 shows LLM performance across schemas. v2 underperformed due to alias conflicts and redundant UNION operations from bridge ta- bles, which make queries more complex. In con- ModelText2SQLP-Text2SQL GPT-4o5.304.30 Llama39.307.57 Qwen2.513.425.80 CodeQwen13.425.80 DeepSeek7.916.83 DeepSeek-Coder11.515.95 Table 3: Coefficient of Variation Iter1Iter2Iter3 Iteration 3.0 3.2 3.4 3.6 3.8 4.0 4.2 Score Relevance Correctness Expression (a) Human Assessment #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 20 40 60 Iter3Iter2Iter1 (b) Model Performance Figure 3: (a) Human quality assessment of different versions of requirements analysis. (b) Performance of 11 models with different versions of requirements analysis, ordered as in Table 2. trast, v3 performed better by merging tables, elimi- nating aliases, and aligning schema semantics with natural language queries. This suggests that over- normalization should be avoided, with a focus on semantic alignment and minimizing multi-table joins. Additionally, as shown in Table 3, the coeffi- cient of variation (Reed et al., 2002) is smaller in the P-Text2SQL task, indicating that P-Text2SQL helps the model better adapt to database data model variations. 5.2 Iteration of Requirement Analysis The requirements analysis dataset underwent three rounds of iterative revisions. To assess the im- provements, we analyzed the dataset after each iteration from both human evaluation and model performance perspectives. Figure 3(a) shows that three master’s students assessed the dataset on relevance, correctness, and expression. The re- sults indicate significant improvement after each iteration, confirming enhanced clarity and accu- racy. Fleiss’ kappa coefficient (Fleiss et al., 1981) yielded a value of 0.8171, indicating strong consis- tency among annotations. Figure 3(b) shows that 11 baseline models performed better in the database design phase as the quality of the requirements analysis improved, suggesting that higher-quality analysis leads to better model performance and fewer evaluation errors. GPT-4o LLama3 DeepSeek Qwen2.5 codeqwen 0 10 20 30 40 50 60 70 Execution Accuracy (EX) Text2SQL Progressive-Text2SQL (a) EX Comparison GPT-4o LLama3 Qwen ChatGLM4 DeepSeek 40 60 80 100 120 140 Growth Rate (%) Simple Moderate Challenging (b) Growth Rate Comparison Figure 4: Comparison of the model’s EX results on Progressive-Text2SQL and Text2SQL tasks. The Progressive-Text2SQL settings slightly differ from the main experiment. We only compute the EX for the fi- nal node of each graph to ensure consistency with the Text2SQL data. 5.3 In-depth Analysis with P-Text2sql To explore whether the dynamic progressive reason- ing graph in Progressive-Text2SQL can better as- sist models in generating complex SQL queries, we compare the performance of models in Text2SQL and Progressive-Text2SQL tasks, as shown in Fig- ure 4(a). Graphs significantly improve the Execu- tion Accuracy (EX) of all baseline models, indicat- ing that the graph effectively simulates incremental user input, helping the model better understand and complete SQL generation tasks. Additionally, we compare the growth rate of EX for SQL generation tasks of varying difficulty when a graph is applied. As shown in Figure 4(b), as the task difficulty increases, the improvement in SQL generation due to the use of graphs be- comes more pronounced, with the most significant increase observed for tasks at the “Challenging” difficulty level. This is because more difficult SQL queries tend to be longer and harder to describe clearly using simple natural language. With graphs, however, the model is guided through a step-by- step reasoning process, helping it generate more accurate SQL queries. 5.4 Robustness Test of P-Text2SQL In the Progressive-Text2SQL dataset, we con- structed high-quality reasoning graphs, but real- world reasoning may contain errors. To better sim- ulate realistic disruptions and evaluate model ro- bustness, we define level-kperturbations as the targeted selection of nodes within the reasoning graph, where each selected node is subjected to one or more disruptive operations. These operations include syntactic corruption of SQL statements, se- 12345 Table Num 0 10 20 30 40 50 60 Design Metrics AVG (%) Llama3 Qwen2.5 DeepSeek Llama3-sqlcoder Qwen2.5-Coder DeepSeek-Coder Figure 5: Impact of the number of tables on LLMs in database design tasks. mantic misalignment with the schema, injection of contradictory logic, or complete removal of the node. This approach introduces realistic flaws that can affect downstream reasoning. As shown in Table 4, model performance declines as the per- turbation level increases, yet in most cases, still outperforms baseline Text2SQL models. This high- lights the importance of step-by-step structured reasoning and the critical role of graph integrity in SQL generation tasks. 5.5 Influence of Number of Tables for Database Design In the main experiment, the design phase evalua- tion requires the model to design two tables within the database. To further investigate how the num- ber of tables to be designed affects the model, we modify the input to include the description of the entire database, with the model being tasked to output the design for a specified number of tables, ranging from 1 to 5. We then evaluate the design capabilities of models under these different require- ments. As shown in Figure 5, as the number of ta- bles grows, different models exhibit varying perfor- mance. Among them, the general-purpose models show relatively stable and consistent performance, with little impact from the number of tables. In con- trast, models fine-tuned on code data exhibit more significant fluctuations in performance, which may be due to the impact of code fine-tuning on the gen- eral instruction-following capabilities of the model. Some specialized models, in most cases, perform worse with fewer input tables compared to when they have more tables. This suggests that when the number of input tables is small, specialized mod- els may struggle to fully utilize their fine-tuned features in such simpler tasks. Llama3MistralDeepSeekQwen2.5ChatGLM-4DeepSeek-CodersqlcoderCodeQwenLlama3-sqlcoder P-Text2SQL46.7630.128.1147.0146.5246.2736.3241.2944.03 Level-143.7130.0226.2446.4845.2743.8134.2741.1842.39 Level-2 41.4824.6522.8938.8137.6539.9829.5832.3436.90 Level-336.2922.1316.3633.3333.3336.0722.4525.8730.35 Level-433.2820.4016.6729.8531.5933.6820.7121.6027.36 Level-532.1419.6516.2329.8530.8532.6819.6519.9026.59 Text2SQL 28.8515.1713.6828.8628.8630.8516.6716.4224.88 Table 4: Robustness Test of Progressive-Text2SQL 5.6 New Perspective of Model Rankings Existing benchmarks are based on SQL generation tasks, such as Text-to-SQL, and do not address other types of database tasks. For further analy- sis, in Figure 6, we test the model’s performance on non-SQL generation tasks (e.g., design, mainte- nance). Among them, SQLCoder performs poorly on other tasks compared to its performance on SQL generation tasks, which is due to the fine-tuning on SQL data affecting its capabilities on other tasks. GPT-4o 56.19 GPT-4o-mini 47.72 Qwen2.5 38.01 Llama3-SQL 33.25 SQLCoder 24.64 DeepSeek 17.54 Model Comparison of SQL Tasks Model Comparison of Other Tasks Figure 6: Ranking of models on SQL generation tasks and other (non-SQL generation) tasks. 6 Related Work LLM Applications in Database Tasks Large language models (LLMs) have enabled signifi- cant advances in a variety of fields, including database management (Zhou et al., 2024b; Li et al., 2024). For example, LLMs have been employed for Text-to-SQL tasks, where they generate SQL queries based on natural language descriptions. A range of LLM-based Text-to-SQL methods have been proposed (Hong et al., 2024; Yaghmazadeh et al., 2017; Xu et al., 2017), showing substan- tial progress on this task. Additionally, LLMs have been used in database maintenance (Zhou et al., 2024a), a domain where recent work has con- tributed to advancements. As LLMs become more widely used in the database field, the challenge of evaluating their database capabilities has grown, necessitating a comprehensive benchmark. Existing Database BenchmarksSeveral bench- marks have been proposed to evaluate LLMs in the context of database-related tasks. SQLStorm (Schmidt et al., 2025), BIRD (Li et al., 2023) and Spider (Yu et al., 2018; Lei et al., 2024) are Text- to-SQL benchmarks designed to assess a model’s ability to generate SQL queries from natural lan- guage questions. They are widely used to evaluate the performance of general models in query gen- eration, primarily focusing on execution accuracy (Execution Accuracy, EX). Additionally, Database QA benchmarks (Zheng et al., 2025) have been in- troduced to assess the richness of models’ database domain knowledge. However, both benchmarks are limited to a single task and fail to compre- hensively assess other database-related tasks such as schema design, database debugging, or mainte- nance. We fill this gap by proposing a more com- prehensive evaluation benchmark that covers the entire database lifecycle (Gupta et al., 2011), in- cluding design, implementation, debugging, and maintenance. 7 Conclusion We presented DBLifeBench, a pioneering bench- mark that shifts the evaluation paradigm from iso- lated SQL generation to holistic database lifecycle management. By assessing capabilities across De- sign, Implementation, Operation, Debugging, and Maintenance, we exposed the limitations of cur- rent specialized models, which often sacrifice gen- eral reasoning for syntax proficiency. Additionally, our Progressive-Text2SQL task demonstrates that structured, step-by-step reasoning is key to solving complex database problems. DBLifeBench pro- vides the community with a rigorous standard to guide the development of the next generation of LLMs-true Autonomous Database Agents. Limitations DBLifeBench, while comprehensive, has some lim- itations. It is primarily designed for LLMs, which may not fully capture the performance of non-LLM approaches. Additionally, some tasks, particularly in design and maintenance, may be challenging to define and evaluate accurately. Lastly, the current benchmark focuses exclusively on textual modal- ities and does not address multimodal scenarios, such as interpreting visual performance charts. Fu- ture work will aim to incorporate multimodal inputs to broaden the scope of evaluation. References Ron Artstein. 2017. Inter-annotator agreement. Hand- book of linguistic annotation, pages 297–313. Yanyu Chen, Yue Li, Yongyi Cui, Dongsheng Shi, and Lichang Dai. 2026. Reinforcement learning for large language model selective evidence adoption from contaminated retrieval results. arXiv preprint arXiv:2607.20090. Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, et al. 1981. The measurement of interrater agreement. Statistical methods for rates and proportions, 2(212- 236):22–23. JonathanFürst,CatherineKosten,Farhad Nooralahzadeh, Yi Zhang, and Kurt Stockinger. 2024.Evaluating the data model robustness of text-to-sql systems based on real user queries. arXiv preprint arXiv:2402.08349. Pranshu Gupta, Ramon Mata-Toledo, and Morgan Mon- ger. 2011. Database development life cycle. Roma- nian Economic Business Review, 5:8–17. Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2024. Next-generation database interfaces: A survey of llm- based text-to-sql. arXiv preprint arXiv:2406.08426. Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shiwei Shi, Guoqing Du, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, and Rui Zhao. 2023. Tptu-v2: Boosting task planning and tool usage of large language model-based agents in real- world systems. Preprint, arXiv:2311.11315. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2024. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. Preprint, arXiv:2411.07763. Guoliang Li, Xuanhe Zhou, and Xinyang Zhao. 2024. Llm for data management. Proceedings of the VLDB Endowment, 17(12):4213–4216. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Preprint, arXiv:2305.03111. George F Reed, Freyja Lynn, and Bruce D Meade. 2002. Use of coefficient of variation in assessing variability of quantitative assays. Clinical and Vaccine Immunol- ogy, 9(6):1235–1239. Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. 2023. Tptu: Task planning and tool usage of large language model-based ai agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop. Tobias Schmidt, Viktor Leis, Peter Boncz, and Thomas Neumann. 2025. Sqlstorm: Taking database bench- marking into the llm era. Proceedings of the VLDB Endowment, 18(11):4144–4157. Dongsheng Shi, Yue Li, Xin Yi, Yongyi Cui, Huawei Feng, and Linlin Wang. 2026. Surgent: A surgical multi-agent assistance system across the periopera- tive workflow. arXiv preprint arXiv:2605.29368. Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sql- net: Generating structured queries from natural lan- guage without reinforcement learning. arXiv preprint arXiv:1711.04436. Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017.Sqlizer: query synthesis from natural language. Proc. ACM Program. Lang., 1(OOPSLA). Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. arXiv preprint arXiv:1909.05378. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3911–3921, Brussels, Bel- gium. Association for Computational Linguistics. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic pro- gramming. In Proceedings of the national conference on artificial intelligence, pages 1050–1055. Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, and Hangyu Mao. 2024. Benchmark- ing the text-to-sql capability of large language mod- els: A comprehensive evaluation. arXiv preprint arXiv:2403.02951. Kaituo Zhang, Mingzhi Hu, Hoang Anh Duy Le, Far- iha Kabir Torsha, Zhimeng Jiang, Minh Khai Bui, Chia-Yuan Chang, Yu-Neng Chuang, Zhen Xiong, Ying Lin, Guanchu Wang, and Na Zou. 2026a. A survey on evaluating quality and trustworthiness in llm-generated data. Preprint, arXiv:2601.17717. Kaituo Zhang, Zhen Xiong, Mingyu Zhong, Zhimeng Jiang, Zhouyuan Yuan, Zhecheng Li, and Ying Lin. 2026b. Are tools all we need? unveiling the tool-use tax in llm agents. arXiv preprint arXiv:2605.00136. Yihang Zheng, Bo Li, Zhenghao Lin, Yi Luo, Xuanhe Zhou, Chen Lin, Guoliang Li, and Jinsong Su. 2025. Revolutionizing database q&a with large language models: Comprehensive benchmark and evaluation. In Proceedings of the 31st ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining V. 2, pages 5960–5971. Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024a. Db-gpt: Large language model meets database. Data Science and Engineering, 9(1):102–111. Xuanhe Zhou, Xinyang Zhao, and Guoliang Li. 2024b. Llm-enhanced data management. arXiv preprint arXiv:2402.02643. A Appendix A.1 Examples 1 # Input 2 Requirements Analysis: 3 1. Purpose 4 2. Key Requirements 5 3. Non -Functional Requirements 6 ... 7 # Output 8 Table 1: frpm 9 CDSCode , TEXT , Primary Key 10 ... 11 Table 2: satscores 12 cds , TEXT , Primary Key 13 ... Listing 1: Example of Design. 1 # Input 2 results of design phase 3 # Output 4 Table 1: CREATE TABLE frpm ( 5 CDSCode TEXT PRIMARY KEY , 6 AcademicYear TEXT , 7 ... 8 ); 9 Table 2: CREATE TABLE satscores ( 10 cds TEXT PRIMARY KEY , 11 ... 12 ); Listing 2: Example of Implementation. 1 # Input 2 ## Simple DDL: 3 albums (AlbumId , Title , ArtistId) 4 artists (ArtistId , Name) 5 ... 6 ## Question: 7 Retrieve the album titles and the corresponding artist names. 8 # Output 9 SELECT Title , Name FROM albums INNER JOIN artists USING (ArtistId); Listing 3: Example of Text2SQL. 1 # Input 2 Simple DDL: ... 3 Nodes: 4 5 "id": 1, 6 "sql statement": "SELECT ID FROM instructor WHERE name ='David Brown '", 7 "nl": "Retrieve the ID of the instructor whose name is'David Brown'." 8 , 9 10 "id": 2, 11 "sql statement": "SELECT course_id , sec_id , semester , year FROM section WHERE building ='Smith' AND room_number ='101'", 12 "nl": "Retrieve the course ID , section ID , semester , and year for all sections held in room 101 of the 'Smith' building." 13 14 Edges: 15 16 "source": 1, 17 "target": 3 18 , 19 20 "source": 2, 21 "target": 3 22 23 "question": 24 Update the instructors of all courses held in room 101 of the'Smith' building to'David Brown'. 25 # Output 26 UPDATE teaches SET ID = (SELECT ID FROM instructor WHERE name ='David Brown ') WHERE (course_id , sec_id , semester , year) IN (SELECT course_id , sec_id , semester , year FROM section WHERE building ='Smith' AND room_number ='101') Listing 4: Example of P-Text2SQL. 1 # Input 2 ## Simple DDL: ... 3 ## Wrong SQL: 4 SELECT T1.Phone FROM satscores AS T1 INNER JOIN schools AS T2 ON T1.cds = T2.CDSCode WHERE T1.AvgScrRead > 1500 ORDER BY T1.NumTstTakr DESC LIMIT 1 5 ## Error Info: 6 OperationalError('no such column: T1. Phone') 7 8 # Output 9 SELECT T2.Phone FROM satscores AS T1 INNER JOIN schools AS T2 ON T1.cds = T2.CDSCode ORDER BY T1.NumGE1500 DESC LIMIT 1 Listing 5: Example of SQL-Debugging. 1 # Input 2 ## System: 3 You are a role assigner. Your task is to select experts with diverse identity to correctly analyze the root causes of the given alerts. 4 ## User: 5 - Role Description 6 You are the leader of a group of diagnosis experts. Currently , the database in your company may meet problems. The anomaly alert is: 7 During the inspection , it was found that from 2023 -10 -15 11:39:49 to 2023 -10 -15 11:40:49 , the database's CPU usage was above the threshold of ... 8 9 Now you need to select experts with diverse identity to correctly analyze the root causes of the given alert. The names of available experts are: 10 ['CpuExpert','MemoryExpert','IoExpert ','WorkloadExpert','QueryExpert', 1.Purpose The `frpm` table is designed to store data related to the Free and Reduced-Price Meal (FRPM) program for schools. ... 2. Key Requirements ... 3. Non-Functional Requirements... 4.... Requirements Analysis # Table 1: frpm frpmCDSCode, TEXT, Primary Key (otherfields ...) # Table 2: satscores satscorescds, TEXT, Primary Key ... Database Design ## System: You are a role assigner. Your task is to select experts with diverse identity to correctly analyze the root causes of the given alerts. ## User: Role Description Alert Information Output Rules Maintenance-Assign 1.CpuExpert 2. MemoryExpert 3. IoExpert Output ## Optional Expert: ['CpuExpert', 'MemoryExpert', 'IoExpert', 'WorkloadExpert', 'QueryExpert', 'WriteExpert', 'IndexExpert', 'ConfigurationExpert'] Output ## System: You are a CPU agent that can use the tool apisto check CPU usage, analyze the root causes of ... ## User: Alert Information Based on the above history, what will you, CpuExpert, do next? ## Optional APIs: -cpu_match_diagnose_knowledge(args...) -whether_is_abnormal_metric (args...), -... Maintenance-Expert Thought: Chain of Thought Action: whether_is_abnormal_metric Action Input: args... Expert Output Figure 7: Illustration of Design and Maintenance Model Text2SQLP-Text2SQL v1v2v3v1v2v3 GPT-4o19.2518.0020.5054.3349.1950.33 Llama313.7511.7514.7542.6740.4548.33 Mistral9.509.258.7530.0022.9830.00 DeepSeek11.7510.0012.0036.3330.7433.33 Qwen2.514.2514.0017.0049.3341.4249.67 ChatGLM-416.7515.0015.0037.6737.8641.00 DeepSeek-Coder15.0013.0017.2546.0044.0150.67 llama3-sqlcoder12.0011.7515.0049.0041.4253.00 sqlcoder-7b-28.758.008.7538.0032.0433.67 Qwen2.5-Coder-7B-Instruct9.008.2511.2536.3333.0138.00 Table 5: Comparison of the model’s performance across different schemas. 'WriteExpert','IndexExpert',' ConfigurationExpert'] 11 12 # Output 13 1. CpuExpert 14 2. MemoryExpert 15 3. IoExpert Listing 6: Example of Maintenance-Assign. 1 # Input 2 ## System: 3 You are a CPU agent that can use the tool apis to check CPU usage ( whether_is_abnormal_metric), analyze the root causes of high CPU usage using the metrics , queries and knowledge gained from ( cpu_match_diagnose_knowledge), and give optimization solutions (e.g., optimize_index_selection , enable_or_disable_nestloop_operator) . 4 ## User: 5 You are in a company whose databases meet an anomaly. The anomaly's start_time is $start_time and end_time is $end_time . The anomaly alert is: Alert ... 6 7 - During diagnosis , you have access to the following tools: 8 Optional APIs ... 9 10 Database history 11 12 Based on the above history , what will you , CpuExpert , do next? 13 # Output 14 Thought: Now that I have obtained the start and end time of the anomaly , I will check whether the CPU usage is abnormal during that time period using the` whether_is_abnormal_metric` tool. 15 16 Action: whether_is_abnormal_metric 17 Action Input: "start_time": "2023 -10 -15 21:01:49", "end_time": "2023 -10 -15 21:02:49", "metric_name": "cpu_usage " 18 19 Please execute the above action and provide the result. Listing 7: Example of Maintenance-Expert. A.2 Graph Construction of Progressive-Text2SQL The following is the prompt for constructing graph data in P-Text2SQL, with the Examples provided in Listing 4. 1 prompt = ( 2 "The goal is to construct a directed graph representation from a given sqlite SQL statement to represent the gradual advancement of functional implementation and eventually complete all requirements ." 3 "Each node in the graph represents a SQL statement , which is a subsequence of a given SQL statement , and the edge represents the execution order. " 4 "Subsequent nodes are the progression of the previous node. ( please attention the last node should be the original given SQL statement. )" 5 "This graph should faithfully reflect the topological execution order of SQL statements to achieve requirements. " 6 "The following includes three cases , including Sql , and Graph . " 7 8 "## Example 1 " 9 f"example_template_1 " 10 "## Example 2 " 11 f"example_template_2 " 12 "## Example 3 " 13 f"example_template_3 " 14 "Please build the graph according to the following Sql. Please note that just output the final graph. Do not include any other superfluous descriptions . " 15 "# Sql: " 16 f"raw_sql " 17 "# Graph : ") Listing 8: Prompt for Graph Construction.

Instant research discovery