Paper deep dive
ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language
Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, Fang Niu, Bofan Xie, Guanchen Ge, Leqi Zheng, Ziyang Liu, Xiannian Cao, Pengcheng Ge
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/12/2026, 2:41:12 AM
Summary
The paper introduces ProcArena, a benchmark for evaluating Large Language Models (LLMs) on Natural Language to PL/SQL development. It covers both Direct (single-turn) and Interactive (multi-turn) modes across nine development subscenarios in PostgreSQL and Oracle. The benchmark consists of 3,998 executable tasks over 157 databases, constructed using iterative logic enhancement and scenario-specific adapters. Evaluation of seven models shows that realistic NL-to-PL/SQL development remains challenging, with best average scores of 62.2% in Direct mode and 57.8% in Interactive mode.
Entities (10)
Relation Signals (12)
ProcArena → hasdatabasecount → 157
confidence 95% · over 157 databases
ProcArena → hassubscenariocount → 9
confidence 95% · spanning nine development subscenarios
ProcArena → hastaskcount → 3,998
confidence 95% · ProcArena comprises 3,998 executable tasks
ProcArena → supportsmode → Direct Mode
confidence 95% · ProcArena comprises 3,998 executable tasks... covering both Direct and Interactive modes.
ProcArena → supportsmode → Interactive Mode
confidence 95% · ProcArena comprises 3,998 executable tasks... covering both Direct and Interactive modes.
Tsinghua University → affiliatedwith → ProcArena
confidence 90% · Authors from Tsinghua University present ProcArena
Lenovo Group Limited → affiliatedwith → ProcArena
confidence 90% · Authors from Lenovo Group Limited present ProcArena
ProcArena → supportsdialect → PostgreSQL
confidence 90% · spanning nine development subscenarios in PostgreSQL and Oracle.
ProcArena → supportsdialect →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.
Tags
Links
- Source: https://arxiv.org/abs/2609.06527v1
- Canonical: https://arxiv.org/abs/2609.06527v1
Trouble viewing inline? Open PDF directly →
Full Text
79,687 characters extracted from source content.
Expand or collapse full text
ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language [Experiment, Analysis & Benchmark] Hang Zhang 1 , Chaokun Wang 1,∗ , Yuzhi Pan 1 , Ziyao Zhong 1 , Shuo Cao 1 , Yue Xue 1 , Zeyu Huang 1 , Xingwei Zhou 1 , Fang Niu 1 , Bofan Xie 1 , Guanchen Ge 1 , Leqi Zheng 1 , Ziyang Liu 1 , Xiannian Cao 2 , Pengcheng Ge 2 1 Tsinghua University, Beijing, China 2 Lenovo Group Limited, Beijing, China zhanghang24, panyz24, zhongzy25, caos24, y-xue24, huangzy24, zhouxw24, nf21, xbf25, ggc22, zhenglq24, liu-zy21@mails.tsinghua.edu.cn, chaokun@tsinghua.edu.cn, caoxn3, gepc1@lenovo.com ABSTRACT Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL pro- grams, attracting increasing attention from the database commu- nity. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and opti- mization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi- scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution- based benchmark covering both Direct and Interactive modes. Pro- cArena comprises 3,998 executable tasks over 157 databases, span- ning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic En- hancement and scenario-specific adapters, and derive paired In- teractive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver–User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particu- larly in interactive settings. We release the benchmark on the public website https://zeyuhuangzh.github.io/procarena-site/. PVLDB Reference Format: Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, Fang Niu, Bofan Xie, Guanchen Ge, Leqi Zheng, Ziyang Liu, Xiannian Cao, Pengcheng Ge. ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language [Experiment, Analysis & Benchmark]. PVLDB, 20(1): X-X, 2027. doi:X.X/X.X This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 20, No. 1 ISSN 2150-8097. doi:X.X/X.X PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/ZhanGHanG9991/ProcArena. 1 INTRODUCTION By integrating procedural constructs with SQL, PL/SQL 1 enables developers to implement data access and business logic directly within database systems, reducing communication overhead be- tween applications and databases [16]. It has therefore become an important component of modern database systems [3,6,8,9]. For example, Microsoft Azure SQL Database alone hosts over two bil- lion PL/SQL codes that serve billions of daily invocations [14,35]. However, developing PL/SQL codes is difficult in practice, as it re- quires knowledge of database schemas, business logic, and PL/SQL syntax [16,38]. Recently, large language models (LLMs) have been increasingly explored for automating the translation from natural language (NL) to PL/SQL (NL-to-PL/SQL) [50]. Despite this progress, systematically assessing LLMs for practical NL-to-PL/SQL development remains challenging, as no benchmark captures the diverse development scenarios and interactions in- volved in real-world PL/SQL programming. Prior work provides only partial foundations for such an assessment. PLForge [50] in- troduces NL-to-PL/SQL datasets, models, and execution-based eval- uation, but primarily studies direct PL/SQL code generation from complete requirements. NL-to-SQL benchmarks [21,27,45,49] ex- amine how models translate NL requests into SQL by grounding them in database schemas and contents, while interactive Text-to- SQL benchmarks such as BIRD-Interact [18] further incorporate interactions with users and database environments. However, both lines of work target declarative SQL rather than PL/SQL codes. General code-generation benchmarks [5,19,31,40] cover genera- tion and modification tasks in languages such as Python, Java, and C++, but do not capture key characteristics of PL/SQL development, including database environment and procedural logic. To the best of our knowledge, ProcArena is the first benchmark to evaluate LLMs on NL-to-PL/SQL across multiple development scenarios in both Direct and Interactive modes across PostgreSQL and Oracle. Figure 1 illustrates the Direct and Interactive task under from scratch development scnario. To construct the benchmark, we outline two main challenges. C1: How can we construct realistic 1 In this paper, we use PL/SQL to refer to procedural extensions of SQL, including Oracle PL/SQL [1], PostgreSQL PL/pgSQL [2], and related dialects. arXiv:2609.06527v1 [cs.CL] 6 Sep 2026 CREATE OR REPLACE PROCEDURE create_order(uid_in INT) LANGUAGE plpgsql AS$procedure$ DECLARE user_cart_id INT; new_order_id INT; order_amount DECIMAL := 0; BEGIN SELECTcart_id INTOuser_cart_id FROM carts WHEREuser_id = uid_in; INSERT INTO orders (user_id, total_amount) VALUES(uid_in, order_amount)RETURNING order_id INTOnew_order_id; FORrec IN SELECT* FROMcarts_item JOINproducts ON carts_item.product_id = products.product_id WHERE carts_item.cart_id = user_cart_id LOOP UPDATEproducts SET stk_num = stk_num -rec.cart_item_quantity WHEREproduct_id = rec.product_id; order_amount := order_amount + (rec.price * rec.cart_item_quantity); INSERT INTO orders_item (order_id, product_id, order_item_quantity) VALUES(new_order_id, rec.product_id, rec.cart_item_quantity); END LOOP; UPDATE ordersSET total_amount = order_amount WHERE order_id = new_order_id; DELETE FROM carts_itemWHERE cart_id = user_cart_id; END; $procedure$ Synthesis (5) From Scratch Document- Grounded Example- Guided Template Instantiation Code Composition Editing(4) Functional Modification Performance Optimization Code Repair Code Completion Direct (Single-Turn) Complete Requirement Create an Ora cle PL/SQL stored proced ure that terminates employment for transportation st aff who se years of service exceed twice the target service duration, but only proceed with terminations if the department maintain s adeq uate staff ing levels above the minimum requ irement. LLM Solver Complete requirement → PL/SQL code Interactive (Multi-Turn) Initial Request (Underspecif ied) I need a stored procedure to handle the checkout flow. Basically, when a user with a specificuser IDwa nts to buy something , the system needs to grab everything from theirsho pping cart , check theinventoryto make sure we have enough sto ck, an d then generate a ne worder. We also need to figure out thetotal amountcorrect ly and make sure t he cart is empty afterwards. The current proce s is a bit messy an d I want to make sure the data stays con sistent. Underspecified requirement → Interaction → PL/SQL code Which reading of the input scope is correct? 2.Only user_id; the active cart is inferred. Interaction 3 1 2 3 CREATE OR REPLACE PROCEDURE sp(para_Years_WorkingNUMBER, para_Min_StaffNUMBER) IS staff_countNUMBER; BEGIN SELECTCOUNT(*) INTOstaff_count FROM"transportation_staff" WHERE "Employment_Status" = 'Active'; IFstaff_count> para_Min_Staff THEN DELETE FROM "transportation_staff" WHERE "Years_Working" > para_Years_Working* 2; END IF; END; 1 2 FromScratch FromScratch ask_selector UPDATE products SET stk_num= stk_num-2 WHERE product_id= 101;SELECT stk_numFROM products WHERE product_id= 101; Ok, 1 row affected: stk_num| 8 Interaction 4 Multi-Turn Interaction execute_scratch get_schema() --what tables am I working with? table carts, columns=[cart_id(int|PK), user_id(int)] | table carts_item, columns=[cart_id, product_id, cart_item_quantity] | ...... I need to join the cart data with product details.Could you provide the specific foreign key relationship between thecarts_itemtable andtheproductstable? They are connected by the product ID. The products table contains the price information, and the shopping cart table contains the quantity for each item. Interaction 1 Interaction 2 get_schema ask_user LLM Solver submit 1. N o n e o f th e fo llo wing are co rrect. 2. The procedure takes only user_id; the active cart is looked up internally. 3. Th e p ro ced u re takes b oth user_idand an exp licit cart_id. 4. A list of product_idsand quantities, without reading the cart. Figure 1: ProcArena evaluates the same PL/SQL development target under two modes. Left: the nine subscenarios, five under Synthesis and four under Editing, and a Direct episode that maps a complete requirement to PL/SQL code in a single turn. Right: an Interactive episode that clarifies an underspecified request viaask_selectorandask_userturns with the User ,execute turns against the Database Environment before the code is submitted. and difficult NL-to-PL/SQL tasks across multiple PL/SQL de- velopment scenarios? Existing publicly available NL-to-PL/SQL datasets rarely cover these diverse development scenarios. They primarily focus on from-scratch generation with complete NL re- quirements, resulting in relatively limited scenario diversity and task difficulty. C2: How can we construct an executable inter- active environment for realistic multi-turn NL-to-PL/SQL development? Interactive NL-to-PL/SQL development requires the Solver to resolve incomplete requirements against both user intent and the database environment. A fixed transcript cannot assess whether the Solver appropriately seeks clarification or in- spects database information, while unrestricted interaction may alter database state or leak evaluation signals. The environment must therefore support controlled interactions between the Solver and both the User and the Database Environment under an exe- cutable protocol. For C1, we use iterative logic enhancement to add procedural logic, jointly update the PL/SQL code and NL requirement, and validate their executability and semantic consistency. We then ap- ply scenario-specific adapters to construct executable Direct tasks for nine subscenarios. For each subscenario, we rewrite the user requirement and prepare any required supporting artifact, such as a document, behavioral examples, a template, or existing code, while preserving the gold PL/SQL code. For C2, we apply knowledge injection and requirement perturbation to derive paired Interactive tasks from their Direct counterparts by introducing information omissions and resolvable contradictions while preserving the gold PL/SQL code. We maintain an elicitation ledger of the facts, def- initions, and resolutions available to the User Simulator during interaction. We further design the Solver–User Simulator protocol to coordinate user interactions with database environment. We propose ProcArena, which comprises 3,998 NL-to-PL/SQL tasks across nine PL/SQL development scenarios in PostgreSQL and Oracle. We evaluate 7 models across Direct and Interactive tasks. On the Direct and Interactive modes, the best average scores achieved by the models are 62.2% and 57.8%, respectively. The evaluation further compares generation across subscenarios and analyzes the construction components and User Simulator. In summary, we make the following contributions: (1)We propose ProcArena, the first benchmark to evaluate LLMs on NL-to-PL/SQL across multiple development sce- narios in both Direct and Interactive modes across Post- greSQL and Oracle. (2)We construct PL/SQL-specific interactive environments through knowledge integration and requirement pertur- bation, and propose the Solver–User Simulator protocol to coordinate Solver–User interactions and Database Environ- ment tools. (3)We conduct comprehensive experiments to evaluate seven models on 3,998 tasks across nine development scenarios in both Direct and Interactive modes, and further analyze the benchmark construction and interaction behavior. 2 PROBLEM FORMULATION An NL-to-PL/SQL task asks a Solver to produce PL/SQL code from an NL requirement, a database description, and an optional task attachment. We represent the task input as 푋=⟨푞, D, A⟩,(1) where푞states the NL requirement. The database descriptionD= ⟨D 푠 , D 푚 ⟩contains the database schemaD 푠 and its metadataD 푚 . A denotes the optional task attachment. LetPdenote the space of PL/SQL codes. Given a task input푋, the Solver ultimately produces a candidate PL/SQL code ˆ 푃 ∈ P. We model the Solver as S=⟨M, A⟩,(2) whereM is the LLM and A is its action space. 2 2.1 Direct NL-to-PL/SQL The direct NL-to-PL/SQL task provides a task푋 푑 =⟨푞 푑 ,D 푑 ,A 푑 ⟩to a SolverS 푑 =⟨M 푑 , A 푑 ⟩, whereA 푑 =submit. The LLM directly generates a candidate PL/SQL code ˆ 푃 푑 ∈ P from the task input: ˆ 푃 푑 =M 푑 (푋 푑 ).(3) Generation follows푞 푑 underD 푑 and may use the task attach- mentA 푑 . The Solver then submits ˆ 푃 푑 for evaluation. 2.2 Interactive NL-to-PL/SQL An interactive NL-to-PL/SQL task is a multi-turn collaboration among a SolverS 푖 , a User SimulatorU 푖 , and a database environ- mentE 푖 . TheS 푖 receives the initial task input fromU 푖 : 푋 푖 =⟨푞 푖 , D 푖 , A 푖 ⟩,(4) where푞 푖 may be underspecified. For a paired ProcArena task,D 푖 = D 푑 andA 푖 =A 푑 ; interactive construction changes the requirement view while preserving the database description and task attachment. TheS 푖 can clarify information through theU 푖 and inspect the database throughE 푖 . The interaction action space A 푖 ofS 푖 is A 푖 = A U ∪ A E ∪submit,(5) whereA U andA E denote the action spaces for interacting with U 푖 andE 푖 , respectively. The SolverS 푖 initiates each interaction turn. Starting from an empty interaction historyℎ 0 =∅, at an interaction turn푡, the LLM M 푖 selects an action based on the initial task input푋 푖 and the interaction history ℎ 푡−1 : 푎 푡 =M 푖 (푋 푖 ,ℎ 푡−1 ), 푎 푡 ∈ A 푖 .(6) For a nonterminal action, the corresponding interaction target returns a response 푟 푡 =R(푎 푡 ,ℎ 푡−1 ;U 푖 ,E 푖 ),(7) and the protocol updates the interaction history as ℎ 푡 =ℎ 푡−1 ⊕⟨푎 푡 ,푟 푡 ⟩,(8) where⊕denotes appending the current action–response pair to the interaction history. After zero or more nonterminal interaction turns, the LLM gen- erates a candidate PL/SQL code ˆ 푃 푖 ∈ P based on the initial task input and the interaction history: ˆ 푃 푖 =M 푖 (푋 푖 ,ℎ 푇 ),(9) where푇is the number of interaction turns. The SolverS 푖 then submits ˆ 푃 푖 for evaluation, which terminates the task. 3 BENCHMARK CONSTRUCTION In this section, we describe the construction of ProcArena tasks. Figure 2 illustrates the overall construction pipeline. We first con- struct the seed PL/SQL codes and enhance their procedural logic through an iterative logic enhancement method. We then rewrite the corresponding NL descriptions to preserve semantic consis- tency with the enhanced PL/SQL codes. For direct NL-to-PL/SQL task construction, we employ scenario adapters to transform the enhanced NL-to-PL/SQL pairs into direct NL-to-PL/SQL tasks span- ning nine development scenarios. For interactive NL-to-PL/SQL SeedSources Seed Task Scenario Adaptation Synthesis A1–A5Editing B1–B4 Directtask Knowledge Integration Domain Reasoning Commonsense Requirement Perturbation Information omission Contradiction injection Interactivetask fr o z en once, then r ead - only: g old c ode, c all suit e, initial s t a t e Task Logic enhancement Figure 2: The ProcArena construction pipeline. task construction, we apply knowledge injection and requirement perturbation to transform the direct NL-to-PL/SQL tasks into inter- active NL-to-PL/SQL tasks that necessitate interaction between the Solver and the user to achieve correct task completion. 3.1 Seed Task Construction We construct the seed NL-to-PL/SQL tasks from two sources. The first consists of the Hard and Webridge datasets from PLForge, whose underlying databases are derived from Spider. These datasets provide difficult NL-to-PL/SQL pairs in PL/pgSQL and Oracle PL/SQL, respectively. The second source is LogicCat [33], which contains databases and NL-to-SQL tasks that require more sophisticated logi- cal reasoning and domain knowledge. Using the LogicCat databases together with their original complex NL-to-SQL examples as prompts, we employ LLMs to generate corresponding seed NL-to-PL/SQL pairs. 3.1.1 Iterative Logic Enhancement. Due to the strong code genera- tion capabilities of state-of-the-art LLMs, we need to ensure that the NL-to-PL/SQL pairs in ProcArena are sufficiently challenging to effectively evaluate the capabilities of LLMs. In practice, users can extend a requirement by adding business rules. We leverage this characteristic and propose an iterative logic enhancement method, which let a held-out LLM to add one reasonable business rule in each round. Through iterative enhancement, we increase the proce- dural complexity of the PL/SQL code while ensuring that it remains grounded in realistic business scenarios. Concretely, the iterative procedure starts from an NL-to-PL/SQL pair in the source set and proceeds over multiple rounds. In each round, given the current NL-to-PL/SQL pair, the held-out LLM proposes a business rule, generates a PL/SQL code snippet that implements the rule, integrates the snippet into the current PL/SQL code, and updates the NL requirement to describe the resulting PL/SQL code.After푁rounds, we execute the final PL/SQL code on the database and use a held-out validation LLM to check whether 3 Algorithm 1: Iterative Logic Enhancement Input :Source pair⟨푞 0 ,푃 0 ⟩; database descriptionD; initial state 휎 0 ; held-out construction LLMM 푐 ; enhancement rounds 푁 ; repair budget 퐵 Output: Validated seed task 휏 + , or Fail 1 (푞,푃) ← (푞 0 ,푃 0 ) /* Add one rule per round */ 2 for 푛 ← 1 to 푁 do 3 ( ˆ 푞, ˆ 푃) ← Enhance(M 푐 ,푞,푃,D) 4 (푞,푃) ← ( ˆ 푞, ˆ 푃) 5 휂 ← Execute(푃,D,휎 0 ) 6 if Failed(휂) then /* Repair from execution feedback */ 7 for 푏 ← 1 to 퐵 do 8푃 ← Repair(M 푐 ,푞,푃,D,휂) 9휂 ← Execute(푃,D,휎 0 ) 10if Succeeded(휂) then 11break 12 if Failed(휂) or CheckAlignment(M 푐 ,푞,푃,D)= false then 13 return Fail 14 ˆ 퐶 ← GenerateCalls(M 푐 ,푞,푃,D) 15 퐶 ← KeepStateChanging( ˆ 퐶,푃,D,휎 0 ) 16 if 퐶=∅ then 17 return Fail 18 return 휏 + =⟨푞,D,∅,휎 0 ,푃,퐶⟩ the NL requirement and code remain semantically aligned. We then generate a call suite and retain only calls that change the database state. The validated pair and its execution assets are collectively referred to as the seed task for subsequent Direct and Interactive task construction. We illustrate the iterative logic enhancement method in Algorithm 1. 3.2 Direct Task Construction In this section, we first define the scenarios and then explain how to construct Direct tasks based on the seed tasks. 3.2.1 Development Scenarios. We collect 1,535 unique posts from Stack Overflow and Database Administrators Stack Exchange by searching for keywords such as PL/SQL and PL/pgSQL. We then analyze the types of user requests in these posts and find that most could be broadly categorized into two categories. In 48.7% of the posts, users have no existing PL/SQL code and need to generate code from scratch. In another 38.4%, users have PL/SQL code but need to modify, optimize, fix, or complete it. We refer to these two categories as Synthesis and Editing, respectively. Based on the analysis, we further divide Synthesis and Editing into five and four subscenarios, respectively. For Synthesis, we define five subscenarios according to the infor- mation or code artifact provided with the task: •A1: From-Scratch Synthesis. The user provides an NL require- ment and the database description and requests complete target code from scratch. •A2: Document-Grounded Synthesis. The user also provides a design or requirement document that the code must follow. •A3: Example-Guided Synthesis. The user provides examples that illustrate the expected behavior of the target code. • A4: Template Instantiation. The user provides a generic code template and requests a target implementation that fills its task-specific bindings. •A5: Code Composition. The user provides complete, callable PL/SQL components and requests new target code that coordinates their calls, parameters, and procedural logic. For Editing, we define four subscenarios based on the type of requested code change: •B1: Functional Modification. The user requests changes that add, remove, or modify business behavior in existing code. •B2: Performance Optimization. The user requests better exe- cution performance while preserving the behavior of exist- ing PL/SQL code. •B3: Code Repair. The user reports a defect in PL/SQL code and requests a repair that restores the intended behavior. •B4: Code Completion. The user provides incomplete PL/SQL code and requests completion of a coherent missing block. 3.2.2 Scenario-Specific Adapters. We use scenario-specific adapters driven by held-out LLM to transform a validated seed task into a Direct task for each subscenario. The validated seed task defines the target behavior. Each scenario adapter creates the starting point for one development subscenario by rewriting the user requirement and adding a task attachment when needed. The attachment may contain a document, examples, a template, reusable components, or existing PL/SQL code. Let휏 + denote the validated seed task. The scenario contract퐺 푗 defines the preconditions under which adapter 퐹 푗 applies and the acceptance criteria that its output must satisfy. For subscenario푗, the adapter constructs the complete Direct task 퐹 푗 (휏 + ;퐺 푗 )=휏 푑,푗 =⟨푞 푑,푗 ,D,A 푗 ,휎 0 ,푃,퐶⟩.(10) Here,푞 푑,푗 denotes the scenario-specific requirement, andA 푗 de- notes the task attachment. The remaining elements correspond to the database description, initial database state, reference PL/SQL code, and call suite, all of which remain unchanged. Each scenario adapter constructs a task by rewriting the original requirement into a scenario-specific requirement and, when needed, generating a task attachment, while leaving all other task elements unchanged. The resulting NL requirement and attachment must contain all information necessary to complete the task and remain consistent with the database description. If candidate content con- tains conflicting statements, the adapter resolves the conflict and retains a single consistent statement. To construct the scenario-specific requirement and attachment, the adapter combines LLM-guided rewriting with scenario-specific artifact preparation. The LLM handles content that requires se- mantic rewriting. For Synthesis subscenarios that require an at- tachment, the adapter places the appropriate supporting artifact in the attachment: a document, behavioral examples, the provided generic template, or callable components. For Editing subscenarios, deterministic transformations derive code attachments from the reference PL/SQL code without modifying the reference code itself. 4 These transformations revert a business rule, introduce a behavior- preserving slowdown, inject a defect, or remove a coherent code block. After each attempt, we validate the candidate against its scenario contract. If validation fails, the LLM receives the failed checks as feedback for the next attempt. Every candidate must pass three common checks to ensure that the constructed Direct task is valid, non-trivial, and semantically aligned with the reference seed task.①Artifact validity verifies that the attachment has the type and structure required by the target subscenario.②Task non-triviality ensures that the candidate neither exposes the reference code nor leaves no meaningful work for the Solver.③Reference alignment verifies semantic consistency with the reference code through execution. In addition to these common checks, each candidate must pass the scenario-specific checks defined in Table 1. After an adapter passes all checks, we materialize the resulting Direct task. The Solver receives 푋 푑,푗 =⟨푞 푑,푗 ,D,A 푗 ⟩.(11) This input contains only the scenario-specific requirement, data- base description, and task attachment. We keep the gold PL/SQL code, call suite, and initial database state hidden for offline evalua- tion. In Direct, the Solver is allowed to submit only once and cannot callask_user,ask_selector, or any environment tool before sub- mission. We construct Direct tasks for all nine subscenarios. 3.3 Interactive Task Construction 3.3.1 Interactive Requirement Construction. We construct Interac- tive tasks from the Direct tasks in Section 3.2 through two stages. In real-world development, user requirements are often not fully explicit: users may express their intent using domain-specific knowl- edge, logical reasoning, or commonsense knowledge, and may also omit necessary information or provide conflicting constraints. We model these realistic forms of underspecification through Knowl- edge Injection and Requirement Perturbation. Figure 3 illustrates both stages on one airline-operations task, together with the Itera- tive Logic Enhancement step that produces the underlying Direct requirement. Both stages modify only the requirement and preserve the task attachment, reference PL/SQL code, call suite, and initial database state. We apply this construction to eight subscenarios; B2 remains Direct-only because it uses runtime-based evaluation. Knowledge Injection transforms a complete Direct require- ment by replacing selected explicit statements with formulations that rely on implicit knowledge. It uses three operators: •Domain Knowledge replaces an expanded definition or for- mula with a domain term. •Logical Reasoning replaces procedural steps with an invari- ant or constraint. •Commonsense Knowledge removes information that users infer from common conventions in the given scenario. All three operators preserve the scenario contract and the reference PL/SQL code. For all three operators, we extract the knowledge needed to interpret each rewrite and store it in the online knowledge ledger Λ 퐾 푗 . Each entry contains a domain definition, reasoning logic, or TypeOriginal sentence and controlled change Logic Enhancement Knowledge Injection Information Omission Contradiction Injection Base: Add a wine and set every vintage to the requested year. Logic: Add the wine (skip duplicates), then assign vintages by priority: premium, target, or Napa Valley → requested year; otherwise → previous year. Original: Otherwise, set the vintage to one year before the requested year. Injected: Otherwise, apply the prior-vintage rule. [Glossary: prior-vintage rule = requested year − 1.] Original: For wines marked “now,” increase Cases by exactly 1; treat NULL as 0. Omitted: For wines marked “now,” increase Cases. [The increment and NULL rule are hidden.] Intended: The premium-score threshold is 90. Injected: The main rule says score ≥ 90, but a closing note says score ≥ 80. Figure 3: Controlled requirement edits on one example task. Each row pairs the original sentence with the rewritten one produced by a single operator, with the changed span high- lighted: Iterative Logic Enhancement, Knowledge Injection, Information Omission, and Contradiction Injection. Algorithm 2: Knowledge Injection Input : Task 휏 푑,푗 ; contract 퐺 푗 ; knowledge budget 휅; attempt budget 퐴 Output: Interactive requirement candidate 푞 퐾 푖,푗 and ledger Λ 퐾 푗 , or Fail 1Π ← EditableProse(푞 푑,푗 ,퐺 푗 );fb←∅ 2 ifΠ=∅ then 3 return Fail 4 for 푎 ← 1 to 퐴 do 5 ( ˆ Π, ˆ Λ 퐾 ) ← LLMInject(Π,푃,D,퐺 푗 ,휅,fb) 6fb← KnowledgeGates( ˆ Π, ˆ Λ 퐾 ;Π,푃,D,퐺 푗 ) 7 iffb=∅ then 8푞 퐾 푖,푗 ← ApplyRewrite(푞 푑,푗 ,Π, ˆ Π) 9return 푞 퐾 푖,푗 , ˆ Λ 퐾 10 return Fail commonsense knowledge and is available to the Solver during interaction. Algorithm 2 summarizes the Knowledge Injection process. For each rewritten requirement fragment produced by Knowledge Injec- tion, the algorithm records its knowledge type, the corresponding source fact, the rewritten text, and any queryable knowledge en- try. It then validates the rewritten fragment by checking whether it preserves the scenario characteristics, remains consistent with the gold PL/SQL code, avoids schema-identifier leakage, provides sufficient ledger coverage, and can be traced back to the source fact. If a rewritten fragment fails any validation check, a new rewrite is generated within the fixed attempt budget. Requirement Perturbation applies two operators to푞 퐾 푖,푗 after Knowledge Injection. •Information Omission removes necessary but recoverable information from the visible requirement. 5 Table 1: Scenario-specific adapters. A1–A5 cover Synthesis and B1–B4 cover Editing. All adapters preserve the database description, reference code, call suite, and initial database state. Each constructed user input is written as requirement; attachment. The last column lists the checks applied after the three common checks. SubscenarioApplicable seed taskConstructed user inputAdditional acceptance criteria A1: From-ScratchAny validated seed task.Complete behavioral requirement; no attachment.❶The task has no attachment.❷The requirement fully specifies the target behavior. A2: Document-GroundedThe target behavior depends on facts that can form a separate document. Task request without the document facts; document stating them. ❶ Some implementation that ignores a document fact differs from the reference under the call suite.❷The document exposes no code. A3: Example-GuidedThe call suite contains inputs that illustrate the target behavior. Task request; calls selected from the call suite with their observed behavior. ❶The reference code reproduces every example.❷A leakage check rejects PL/SQL implementation text. A4: Template InstantiationThe target admits a reusable skeleton and task-specific bindings. Task-specific bindings; reusable skeleton with unfilled placeholders. ❶ A placeholder-coverage check matches every binding to the template.❷ The check rejects a target-specific body. A5: Code CompositionExisting callable components can implement the target when coordinated. Composition request; callable PL/SQL components without the coordination code. ❶ The components compile and are callable.❷ A structural check confirms the target coordination code is absent. B1: Functional ModificationReference code has a business rule that functional reversion can invert. Original rule as the requested change; reference code with that rule reverted. ❶ At least one call distinguishes the unchanged code from the reference behavior. B2: Performance OptimizationA behavior-preserving slowdown operator applies to the target. Optimization request; behavior-preserving slowdown of the reference code. ❶ The slower code matches the reference outputs and state.❷ Its warm median runtime exceeds the reference by the contract margin. B3: Code RepairA defect operator applies to the working code.Repair request; reference code with an injected defect. ❶At least one call exposes the defect: an exception, or a mismatched output or persistent state. B4: Code CompletionThe reference code contains a coherent removable block. Completion request; reference code with a coherent block removed and the site marked. ❶A structural check validates the removed block.❷Compilation or execution confirms the incomplete code fails the target behavior. •Contradiction Injection introduces conflicts within the re- quirement or between the requirement and the database environment. Information Omission removes an explicit but recoverable require- ment detail from the visible requirement. We regard the removed detail as necessary if its omission permits an implementation that satisfies the remaining requirement but behaves differently from the reference PL/SQL code when both are executed on the same call suite and initial database state. For each omitted detail, the require- ment ledger records the original statement, where it appears in the source requirement, who can provide the missing information, and how the detail can be restored during interaction. Contradiction Injection introduces conflicting information that cannot be resolved from the visible requirement alone. We consider two forms of contradiction. An Intra-requirement contradiction introduces two incompatible statements into the same requirement, such as different thresholds. Because both statements are visible, the Solver can detect the con- flict directly from the requirement, but must interact with the user to determine which statement the user actually intends. A Requirement-environment contradiction introduces a statement that conflicts with the actual database schema or metadata. For ex- ample, the requirement may refer to a nonexistent table or column, or specify an incorrect data type or constraint. The Solver can use environment tools to detect such mismatches. Environment tools can detect the mismatch, but cannot determine the user’s intended correction. The conflict therefore requires interaction with the user for resolution. Both perturbation operators modify only the visible requirement while preserving the database schema, metadata, gold PL/SQL code, call suite, and initial database state. Each injected conflict has a unique resolution that is consistent with the gold PL/SQL code. For requirement-environment contradictions, we additionally require that the mismatch can be detected from the database environment. 3.3.2 Task Materialization and Interaction Protocol. Task Mate- rialization. After Knowledge Injection and Requirement Pertur- bation, we package the resulting requirement and its supporting assets into an Interactive task: 휏 푖,푗 =⟨푞 푖,푗 ,D,A 푗 ,휎 0 ,푃,퐶,Λ 푖,푗 ⟩.(12) The Solver receives only the requirement, database description, and task attachment. The router assigns each ledger entry to its desig- nated information holder. The pipeline retains only tasks whose ledger entries identify the corresponding construction operators. Interaction Protocol. The User Simulator represents the user in each interactive episode and responds to the Solver’s questions and clarification requests. Since the interaction proceeds over multiple turns in free-form natural language, we implement the simulator with a held-out LLM. The Solver–User Simulator protocol governs how the Solver interacts with the simulator. It defines the actions available to the Solver, the task information that the simulator may access for each action, and the responses that the simulator may provide. By restricting the simulator to user-facing informa- tion, the protocol prevents hidden evaluation assets from leaking into the interaction. Within the simulator, the requirement holder and knowledge holder handle requests that match entries in their corresponding ledgers. During task construction, we create an NL description of the gold PL/SQL behavior for the fallback channel, which answers questions without a matching ledger entry. Separately, environment tools expose the database environment defined in Section 2.2, allowing the Solver to inspect task-relevant database information through the prescribed tool interface. 6 Table 2: Nine actions available to the Solver in an Inter- active episode. Every issued action consumes budget units, even if malformed or failed. ActionEnv.ArgumentsReturned observationCost ask_userUser questionNatural-language answer2 ask_selectorUser optionsIndex of the intended option2 get_schemaDB–Schema of the whole database1 describe_tableDB tableMetadata of one table0.5 sample_rowsDB table, limitSample rows of one table0.5 compile_plsqlDB codeCompiler diagnostics0.5 execute_scratchDB sqlExecution result1 reset_scratchDB–Scratch context restored0.5 submitEval code-0 Action Space. The router sends each action to the component selected by the Solver, and the offline evaluator remains outside the interaction. The Solver is the only action initiator in the proto- col. It may inspect the environment, request information through ask_user, ask the user to select among candidate interpretations throughask_selector, or end the episode throughsubmit. Table 2 summarizes the nine concrete actions, their target components, inputs, returned observations, and interaction costs. The two user- facing actions return a natural-language answer and the index of the intended option, respectively. The analysis groups malformed or unrecognized names undermalformed/other; this category falls outside the action space. The task attachmentAremains visible throughout the episode. Every environment action and everyask_userorask_selector action consumes the interaction budget;submitis cost-free but ter- minates the episode immediately and returns no execution-accuracy feedback, preventing the Solver from iteratively refining its solution against the hidden oracle. The protocol provides two user-facing actions.ask_userroutes the question to the holder responsible for the matched ledger entry, falling back to a default channel when no entry matches. ask_selectorlets the Solver choose a holder and supply candi- date interpretations; the holder returns the index consistent with its ledger. When candidates mix requirement facts with injected knowledge, the Solver may query the two holders separately to resolve each source independently. Resolving contradictions between the requirement and the data- base requires two complementary information sources. The Solver first calls a schema or metadata action to identify the objective mismatch, such as a referenced column that does not exist, then usesask_userorask_selectorto obtain the user’s intended res- olution. Environment actions expose database facts only and never infer user intent, ensuring a clean separation between objective state and subjective preference. 4 PROCARENA STATISTICS ProcArena spans nine Direct subscenarios and eight paired Inter- active subscenarios across PostgreSQL and Oracle. This breadth follows from the task itself: NL-to-PL/SQL development spans syn- thesis and editing, complete and underspecified requirements, and two dialects. Varying the scenario, the mode, and the dialect there- fore keeps evaluation grounded in the full task rather than in one narrow setting. This section reports the resulting corpus: Table 3 Table 3: ProcArena and prior NL-to-PL/SQL datasets. Tasks reports all evaluation instances, and Interactive reports the Interactive subset. Stmt and Cyclo report the mean statement count and cyclomatic complexity per reference procedure. Scale & coveragePer-task difficulty DatasetTasksDBsDialectsScen.InteractiveStmtCyclo PLForge Simple3001301102.83.0 PLForge Hard30011311010.76.8 ProcBench3001231103.33.5 WeBridge2094011029.59.4 ProcArena3,998 157291,87932.118.6 summarizes its scale, coverage, and difficulty; Table 4 breaks it down by subscenario; and Table 5 quantifies the transformations that construct Interactive requirements. 4.1 Comparison with Prior Datasets Table 3 compares ProcArena with four prior NL-to-PL/SQL datasets: PLForge Simple, PLForge Hard, ProcBench, and WeBridge [16,50]. ProcArena expands both scale and coverage. The prior test splits contain 209–300 Direct tasks, one dialect, one scenario, and no Interactive episodes. ProcArena contains 3,998 tasks across two dialects and nine subscenarios, including 1,879 Interactive episodes. ProcArena procedures also contain more statements and have higher cyclomatic complexity on average. They average 32.1 state- ments and 18.6 cyclomatic complexity, compared with 10.7 state- ments and 6.8 cyclomatic complexity for PLForge Hard and 29.5 and 9.4 for WeBridge. ProcArena therefore covers more settings and has higher average procedure complexity. 4.2 Corpus Composition Table 4 breaks down the corpus by subscenario. Direct task counts range from 111 to 120 per dialect. ProcArena contains 1,060 Post- greSQL and 1,059 Oracle Direct tasks. Each Direct task has a paired Interactive episode except B2, which has no Interactive task because it uses a runtime score after semantic-equivalence checking. This pairing yields 940 PostgreSQL and 939 Oracle Interactive episodes. The corpus spans 157 databases; B3 uses 110, while A1 and B2 each use 76. The Input rows report the mean size of the requirement and task attachment. NL tokens include the requirement and any textual attachment, such as the A2 design document and A3 behavioral examples. Code tokens count PL/SQL code in the attachment. A1– A3 have no code attachment, so their Code-token cells are shown as dashes. The Reference procedure rows summarize each target procedure and its call suite: statements, lines, parameters, calls, and tables written. Across the corpus, reference procedures average 39.0 lines, 4.0 parameters, 6.3 calls, and 4.9 tables written. 4.3 Interactive Requirement Transformations Table 5 reports six transformation types for Interactive requirement construction. Section 3.3.1 groups them into Knowledge, Contradic- tion, and Omission. Knowledge replaces explicit facts with domain 7 Table 4: ProcArena corpus by subscenario. All gives the per-subscenario mean for task counts, the corpus-wide distinct count for databases, and the per-applicable- task mean for other metrics. A dash denotes a missing Interactive task or a subscenario without a code attach- ment. A. SynthesisB. Editing MetricA1A2A3A4A5B1B2B3B4All Scale & coverage Direct, PG114119118119111119120120120117.8 Direct, Oracle115116120116118118120120116117.7 Interactive, PG114119118119111119–120120117.5 Interactive, Oracle115116120116118118–120116117.4 Databases76858587889076110107157 Input (per task, avg) NL tokens609.5898.5764.31125.6496.4385.5149.3732.2330.9609.6 Code tokens–519.4334.3446.5429.1410.2244.7402.3 Reference procedure (per task, avg) Statements36.931.243.640.722.741.915.026.730.532.1 Lines39.845.454.439.245.154.513.332.726.739.0 Parameters4.04.04.13.64.93.93.93.94.14.0 Calls푘6.46.16.46.46.76.45.26.36.56.3 Tables written5.14.85.85.94.25.82.34.65.24.9 Table 5: Interactive requirement transformations per task and subscenario. B2 has no Interactive task. Avg reports the mean across Interactive tasks. A. SynthesisB. Editing Fact kindA1A2A3A4A5B1B3B4Avg Knowledge Jargon (terminology)1.893.012.051.001.561.810.001.071.54 Reasoning0.840.780.860.430.760.840.000.850.67 Commonsense0.840.790.820.370.740.790.000.830.65 Contradiction Requirement0.680.690.680.670.680.690.680.680.68 Environment0.320.310.320.330.320.310.330.320.32 Omission Omitted facts2.002.602.001.991.581.942.001.001.89 Transformed facts6.568.196.734.785.646.383.004.755.75 terminology, reasoning constraints, or commonsense conventions. Domain definitions enter the elicitation ledger, while reasoning and commonsense rewrites remain in the offline trace. Contradiction introduces incompatible readings within the requirement or be- tween the requirement and the environment. Information Omission removes a registered fact from the visible requirement and stores it in the elicitation ledger. ProcArena applies 5.75 transformations per Interactive task on average. A4 has fewer reasoning and commonsense rewrites than the other non-B3 subscenarios, while B3 has zero Knowledge trans- formations. The construction pipeline changes only the require- ment, so a subscenario can retain transformations even when its attachment contains much of the target code. 5 EXPERIMENTS We evaluate seven language models on ProcArena, across nine Di- rect subscenarios, eight paired Interactive subscenarios, and two SQL dialects. The strongest model reaches 62.2% execution accuracy in the Direct mode and 57.8% in the Interactive mode. Sections 5.2 through 5.5 report what the models achieve and how they fail. Sec- tion 5.6 then checks that the task construction and the interactive environment earn their place. 5.1 Experimental Setup Evaluation Modes. ProcArena presents all nine subscenarios in the Direct mode and eight of them in the Interactive mode. B2 Performance Optimization is Direct-only, because its score is exe- cution time rather than state equality. In the Direct mode the Solver receives the complete requirement and submits one procedure. In the Interactive mode it receives an underspecified requirement. It may ask the User Simulator for a fact withask_user, confirm a can- didate reading withask_selector, and inspect the database with six environment tools before it submits. Each paired task shares its database description, task attachment, gold procedure, call suite, and initial state. Evaluation Metrics. Execution accuracy (EX) is the metric for the eight paired subscenarios in both modes. We adapt PLForge’s execution-match principle [50] to the stateful behavior of proce- dural code.Exec(푃,퐶,휎 0 )installs푃from the initial state휎 0 , runs the ordered call suite퐶, and returns its outputs together with the final persistent state. For a candidate ˆ 푃, the task-level execution indicator is EX( ˆ 푃;푃,퐶,휎 0 )= 1 Exec( ˆ 푃,퐶,휎 0 )= Exec(푃,퐶,휎 0 ) .(13) Execution accuracy averages this indicator over tasks. B2 is scored differently, because its input is a degraded version of the gold procedure and already satisfies Equation 13. Semantic equivalence is therefore a gate rather than a score. An admitted candidate is measured against the gold procedure by two ratios, acceleration푇 푃 /푇 ˆ 푃 and work푊 푃 /푊 ˆ 푃 , where푇is a median wall-clock time and푊a deterministic work counter. Both are oriented so that 1.0 means the candidate matched the gold procedure and larger is better. Section 5.3 gives the measurement protocol. For the Interactive mode we additionally report the number of interaction turns and the total tokens per episode. Retention is defined in Section 5.4, where it is first used. Baseline Models. We evaluate seven language models: GPT-5.5, GPT-5.4 Mini, and GPT-5.2 [36], Gemini-3.1 Pro [12], GLM-5.3 and GLM-5.2 [11], and DeepSeek V4 Pro [7]. Implementation Details. Every task runs on a freshly restored database, so no episode observes the writes of another. ProcArena ships tasks in two dialects, PL/pgSQL on PostgreSQL 12.22 and Oracle PL/SQL on Oracle Database 21c Express Edition. A sin- gle fixed model, DeepSeek V4 Flash, drives the User Simulator across every Solver, so an EX difference between two Solvers never reflects a difference in the simulator. We set reasoning effort to high for every model that exposes it, and we run each task once. The Solver may take at most 60 recorded steps. Its tool budget is 6+2|Λ|+3 units, whereΛis the task’s elicitation ledger.ask_user andask_selectorcost two units each, environment tools cost between 0.5 and one unit, andsubmitis free. Both the budget and the price list are shown to the Solver, and every tool result reports the units spent so far. 8 Table 6: Interaction lowers accuracy on Synthesis and raises it on Editing. Execution accuracy (%) per subscenario and dialect, with Direct above the shaded band and Interactive below it. Subscenarios run along each category’s ordering axis of Section 3.2. Within each mode block, the best value per column is in bold and the second best is underlined. PostgreSQLOracle A. SynthesisB. EditingOverallA. SynthesisB. EditingOverall ModelA1A2A3A4A5B1B3B4EXA1A2A3A4A5B1B3B4EX Direct DeepSeek V4 Pro39.536.127.144.533.340.319.241.735.246.134.541.744.830.544.115.050.938.3 GLM-5.244.745.433.951.337.849.631.753.343.547.040.548.350.928.045.832.551.743.0 GLM-5.362.352.150.059.757.761.329.265.854.752.247.452.549.128.052.546.754.347.8 GPT-5.4 Mini57.952.950.046.251.443.741.760.050.463.555.265.856.033.944.945.059.552.9 GPT-5.264.952.944.160.558.647.138.365.053.862.6 55.263.360.334.752.549.262.955.1 GPT-5.568.458.055.170.664.058.8 63.3 69.2 63.4 64.3 55.2 66.760.336.459.368.367.259.7 Gemini-3.1 Pro67.558.8 55.1 74.859.564.754.269.263.064.3 55.264.2 62.1 36.4 62.7 80.865.561.4 Interactive DeepSeek V4 Pro5.310.95.110.112.628.629.233.317.029.652.628.335.319.529.77.542.230.5 GLM-5.225.451.321.214.325.247.952.542.535.233.051.727.537.116.933.133.341.434.2 GLM-5.353.5 73.949.2 26.131.560.555.053.350.551.372.449.245.723.753.452.553.450.2 GPT-5.4 Mini30.750.446.621.018.042.954.252.539.842.654.346.740.522.037.345.052.642.6 GPT-5.247.468.155.131.130.658.060.860.851.741.769.046.749.122.042.444.262.147.1 GPT-5.537.752.138.1 49.6 44.171.471.778.355.657.4 79.353.357.8 29.754.270.864.758.4 Gemini-3.1 Pro39.561.331.443.744.1 79.8 74.275.056.451.375.054.252.628.063.6 76.7 71.6 59.1 5.2 Overall Results Table 6 reports EX for every model on every subscenario, in both dialects. The Direct block sits above the shaded band and the In- teractive block below it. The two blocks use the same paired tasks and the same row order, so a reader compares them cell by cell. B2 is not in this table; Section 5.3 reports it. Gemini-3.1 Pro is the strongest model in both modes, at 62.2% and 57.8% pooled over the two dialects. GPT-5.5 follows at 61.6% and 57.0%. DeepSeek V4 Pro is last in both, at 36.8% and 23.7%. Even in the Direct mode, where the requirement is complete and no interaction is needed, the best model is wrong on more than a third of the tasks. The two dialects behave alike, and their differences are small and not consistently signed. In the Direct mode Gemini-3.1 Pro scores 63.0% on PostgreSQL and 61.4% on Oracle, while GPT- 5.4 Mini scores 50.4% and 52.9%. A5 Orchestration is the hardest subscenario for every model. No model exceeds 49.8% on it in the Direct mode or 36.7% in the Interactive mode. A5 asks the Solver to compose existing procedures into one transactional workflow while respecting their signatures. The two categories respond to interaction differently. In the Direct mode Synthesis and Editing are close: 59.8% against 66.2% for Gemini-3.1 Pro, and 59.9% against 64.4% for GPT-5.5. In the Interactive mode they separate, to 48.1% against 73.5% for Gemini- 3.1 Pro. Two subscenarios even score higher in the Interactive mode. A2 rises from 57.0% to 68.1% for Gemini-3.1 Pro and from 49.8% to 73.2% for GLM-5.3, and B3 rises from 67.5% to 75.5% for Gemini-3.1 Pro. Two properties of the tasks account for this. An Editing task hands the Solver the procedure it must change, and that attachment already fixes most of what requirement perturbation removed, so less has to be recovered by asking. A2 and B3 are the two subsce- narios whose difficulty is grounding rather than specification: A2 must map a design document onto the real schema, and B3 must reproduce a reported defect. Both gain from the database tools the Interactive mode provides, and here that gain outweighs the miss- ing requirement facts. A Synthesis task offers no such anchor, so it absorbs the full cost of an underspecified requirement. 5.3 Performance Optimization Figure 4 reports B2, the one subscenario that execution-state equiv- alence cannot rank; the figure labels the gold procedure as the reference. B2 hands the Solver a degraded version of the gold pro- cedure and asks for a faster one that behaves the same way. The degradation applies real inefficiency patterns: a set-basedUPDATE becomes a cursor loop with one statement per row, a join becomes an N+1 lookup, and an aggregate is recomputed inside a loop. The Solver is told that the procedure is correct but slow, and is not told where the inefficiency is. Seven models answer 120 tasks in each dialect, which gives 1,680 measurements. Acceleration and work are defined in Section 5.1; what remains is how they are measured. Every table is scaled to the row count recorded with the task, inside a transaction that is rolled back afterwards. Each procedure is then timed five times, with the gold procedure, the degraded input and the candidate taking turns in rotating order, so that cache warmth favors none of them. The median of the five runs is reported. We take the ratio between a procedure’s own slowest and fastest run as the measurement noise, whose median is 1.66–2.05 on PostgreSQL and 1.42–1.48 on Oracle. Results are aggregated with the harmonic mean, which penalizes a regression instead of letting one large win hide it. Panel (a) bins the tasks by speedup. Every model matches or beats the gold procedure on most tasks in both dialects. GPT-5.4 Mini places 36% of its PostgreSQL tasks in the 2–10×band. GLM-5.2 has the largest slower block, 14% on PostgreSQL and 17% on Oracle. Panel (b) reports the two harmonic means. Five models sit just above the gold procedure on both. GPT-5.4 Mini has the highest 9 (a) Distribution of speedup over the reference(b) Acceleration and work, harmonic means(c) Answers rejected by the EX Figure 4: Most candidates match or beat the gold procedure, but two models buy their speed with far more database work. Each model contributes one bar or marker pair per dialect, PostgreSQL then Oracle. Panel (a) stacks the faster speedup bins above the axis and the slower ones below. Panel (b) uses a log axis on which 1.0 matches the gold procedure, and its connector spans the acceleration and work means. In panel (c), a zero marks a model with no rejected task. acceleration, 1.51, but a work mean of 0.18. GLM-5.2 reaches 0.02 on work, and is the only model whose acceleration falls below the gold procedure, at 0.72. Panel (c) counts the answers the semantic gate rejects, split by compile failure, wrong final state, and runtime error. GPT-5.5, Gemini-3.1 Pro and GPT-5.2 have none in either dialect. GLM-5.2’s rejections are mostly a wrong final state, and DeepSeek V4 Pro’s are mostly compile errors, seven of its eight on Oracle. Only three of the 1,680 answers raised a runtime error. Panel (a) counts tasks and panel (b) weighs them, which is why they disagree. GLM-5.2’s two worst PostgreSQL answers per- form 4,445×and 1,058×the gold procedure’s operations. GPT-5.4 Mini’s worst performs 516×, and its measured speedup is 0.97, close enough to the gold procedure to disappear into the matched band of panel (a). An answer can therefore look correct on the clock while doing hundreds of times the database work. On a scaled test table those extra operations cost little wall-clock time; on production volumes they would not. The same two models also account for most of the rejected answers, and GLM-5.2’s are changes of behav- ior rather than compile failures. We read this as code that is locally plausible without a model of what the database will actually do, which is why B2 reports work beside time. 5.4 Interaction Analysis This subsection asks how much accuracy survives requirement resolution, what that costs, and what the cost is spent on. Figure 5 plots Direct and Interactive EX as bars with their ratio as a line, and mean turns as bars with mean total tokens as a line, where tokens are the input plus output usage reported for an episode. Figure 6 gives each model’s share of recorded actions over the nine tools, plus one bucket for any action name outside the protocol. Figure 7 gives action counts by recorded step for four models spanning the Interactive EX range. It plots counts rather than shares, so the outline of each stack is that model’s episode attrition. All three pool the paired PostgreSQL and Oracle tasks. DS-V4 GLM-5.2 GPT-5.4M GPT-5.2 GLM-5.3 GPT-5.5 Gemini 3.1 Pro (a) EX and retention 0 20 40 60 EX (%) DS-V4 GLM-5.2 GPT-5.4M GPT-5.2 GLM-5.3 GPT-5.5 Gemini 3.1 Pro (b) Interaction cost 0 10 20 Mean turns 60 80 100 Retention (%) 50 100 Mean tokens (k) Direct EXInteractive EXRetentionMean turnsMean tokens Figure 5: Interaction cost does not predict retention. Panel (a): Direct and Interactive EX as bars, their ratio as a line. Panel (b): mean turns as bars, mean tokens per episode as a line. Retention is Interactive EX divided by Direct EX. It measures how much of a model’s own generation ability survives requirement resolution. Raw Interactive EX cannot measure this, because a model with weak generation and strong elicitation scores like a model with the opposite profile. GLM-5.3 retains the most, 98.2%, and DeepSeek V4 Pro the least, 64.5%. The other five fall between 79.4% and 92.6%. Cost does not follow. GPT-5.5 spends the fewest tokens, 38.5k per episode, and the second fewest turns, 7.1, yet retains 92.6%. GLM-5.2 spends the most of both, 123.8k tokens over 19.8 turns, and retains 80.2%. DeepSeek V4 Pro spends the fewest turns, 6.9, and retains the least. Figure 6 shows what the turns are spent on, and the mod- els split into two strategies. GLM-5.3 spends 54.5% of its actions onexecute_scratch, GLM-5.2 40.0%, and Gemini-3.1 Pro 39.1%: these models iterate against a private copy of the database. GPT- 5.5 spends 7.2% there, and 18.0% onsample_rowsand 18.1% on compile_plsqlinstead: it reads the data and compiles the code 10 get_schema describe_table sample_rows compile_plsql execute_scratch reset_scratch ask_user ask_selector submit malformed/other Trace tool Gemini 3.1 Pro GPT-5.5 GLM-5.3 GPT-5.2 GPT-5.4M GLM-5.2 DS-V4-Pro 9.302.132.979.8239.051.1114.4012.029.200.00 13.545.3217.9718.147.220.0815.088.3614.300.00 6.601.329.675.5854.453.267.954.496.630.07 6.269.2517.4214.3533.314.106.802.436.080.00 9.685.4422.1420.5613.760.5612.404.8310.630.00 4.9911.4514.516.9740.043.086.467.135.080.29 9.9912.063.5915.5418.232.4819.192.5116.010.39 0 10 20 30 40 Action share (%) Figure 6: Models split between iterating against the database and inspecting it. Share of all recorded ac- tions per model. Tool names are verbatim from the traces; names outside the protocol are grouped as malformed/other. rather than running it. The two channels to the User Simula- tor split the models the same way. DeepSeek V4 Pro asks the most, 19.2% of its actions onask_user, and confirms the least, 2.5% onask_selector. Gemini-3.1 Pro inverts this, at 12.0% on ask_selector, the highest in the figure. Action names outside the protocol never exceed 0.4%, so tool-call formatting affects none of these numbers. Figure 7 places the same behavior in time. Gemini-3.1 Pro con- centratesask_userin steps two to five and then stops asking. GLM-5.3’sexecute_scratchband runs to step 25, and its episodes are the longest in the figure. DeepSeek V4 Pro’s episodes are nearly over by step ten. Both strategies work. GLM-5.3 iterates against the database and GPT-5.5 barely does, yet they are the two models that retain the most, at 98.2% and 92.6%, and GPT-5.5 reaches its figure on less than half the tokens. Neither the volume of interaction nor the volume of questions predicts retention. What fails is the absence of a strategy rather than the choice between them: DeepSeek V4 Pro takes the fewest turns, spends the largest share of them asking, confirms the fewest readings, and retains the least. Retention is also unstable when the denominator is small, so GLM-5.3’s 98.2% should be read against its Direct EX of 51.3%. 5.5 Error Analysis Figure 8 reports Interactive failures by type, as a percentage of the tasks each model ran, so a cell reads directly against the Interactive EX columns of Table 6. Schema is a wrong table or column, Con- trol flow a wrong branch or loop, State effect a wrong persistent write, Exception a remaining runtime call failure, and Contract an installation or interface mismatch. We assign these types after eval- uation with deterministic rules. Schema and Contract come from installation, interface, argument, signature, component-use, and database diagnostics, and a remaining runtime call failure becomes Exception. For a state mismatch, a lexical comparison of branch, 5101520 (a) Gemini 3.1 Pro 0 500 1000 1500 510152025 (b) GLM-5.3 510152025 (c) GPT-5.4M 0 500 1000 1500 510152025 (d) DS-V4-Pro Recorded tool step Action count get_schema describe_table sample_rows compile_plsql execute_scratch reset_scratch ask_user ask_selector submit malformed/other Figure 7: Models follow visibly different interaction trajectories. Action counts by recorded step, two mod- els per row with shared axes. The bands use the tool names of Figure 6. loop and return structure between the candidate and the gold pro- cedure separates Control flow from State effect. This comparison is used only in post-hoc analysis and is never shown to the Solver. Control flow is the largest category for six of the seven models, between 17.6% and 31.2%. State effect is second, between 7.9% and 15.4%. Exception is negligible everywhere, at most 2.9%. GLM-5.2 has the highest rate in both dominant categories, 31.2% and 15.4%. Gemini-3.1 Pro and GPT-5.5 have the lowest rates in almost every category. DeepSeek V4 Pro has a different profile: its Schema rate is 20.1% and its Contract rate is 27.9%, several times any other model’s, while its control-flow rate is the lowest at 17.6%. Models rarely crash. They compile, they run, and they compute the wrong thing. Two of the five categories have no analogue in declarative SQL, because a query has no branch or loop to get wrong and no persistent write to get wrong. Together they account for between 25.5% and 46.6% of the tasks each model ran. DeepSeek V4 Pro is the one model that fails before the logic rather than in it. Its wrong tables and wrong signatures are consistent with its low retention in Section 5.4, and with the fact that it asks many questions but confirms few readings. 5.6 Benchmark Validation To verify the reliability of the User Simulator, we conduct a hu- man audit of itsask_userreplies (Figure 9). We sample 180 turns stratified over solver models and sub-scenarios, one third from the LOC pool; auditors blind to model identity judged each reply against the task’s oracle evidence for acceptability, failure reason, and answer leakage. Overall, 85.6% of replies are acceptable and the residual failures scatter over six categories with no dominant mode (a); LOC-pool replies pass at a similar rate, with only 3.3% leaking answer content (b). The simulator thus provides reliable feedback with negligible answer leakage. 6 RELATED WORK NL-to-PL/SQL. To the best of our knowledge, PLForge [50] repre- sents the first study specifically targeting NL-to-PL/SQL, achieving 11 Schema Control flow State effect Exception Contract Classified failure type Gemini 3.1 Pro GPT-5.5 GLM-5.3 GPT-5.2 GPT-5.4M GLM-5.2 DS-V4-Pro 5.718.910.11.16.4 7.118.310.10.76.9 0.322.413.71.611.7 1.425.111.60.711.8 4.426.714.61.211.9 3.531.215.42.912.3 20.117.67.92.827.9 0 5 10 15 20 25 Failures / tasks (%) Figure 8: Control flow and persistent state dominate, and neither exists in declarative SQL. Failures by error type in the Interactive mode, as a percentage of the tasks each model ran. Table 7: Every construction operation lowers EX on its own, and the full pipeline lowers it most. Base is EX (%); indented rows are paired percentage-point changes against the same tasks. ConditionGPT-5.5Gemini-3.1 ProGPT-5.2GLM-5.2 Direct task86.6782.9272.9273.33 + Knowledge injection-16.27-9.09-13.40-25.36 + Contradiction injection-10.92-13.87-10.92-15.13 + Requirement omission-10.82-3.90-7.79-25.54 Direct task + all three methods-23.35-17.18-22.91-39.21 SOTA performance through continued pre-training on a PL/SQL- centric corpus. However, PLForge relies on a template-based method for dataset generation, which may restrict syntactic diversity and limit its ability to reflect the complex distribution of real-world workloads. Although real-world procedural workloads also contain UDFs and triggers [14], both PLForge and the current version of ProcArena focus on SPs. ProcArena instead broadens evaluation along a different dimension by covering multiple SP development scenarios in both direct and interactive settings. Text-to-SQL. Recent progress on Text-to-SQL is increasingly driven by language models, involving two main directions: su- pervised PLM-based encoder-decoder parsers and LLM-based ap- proaches utilizing prompting and workflows. PLM-based encoder- decoder parsers typically map serialized inputs to SQL. Key enhance- ments include injecting graph structures for schema-aware learning [4,26], utilizing intermediate representations to decouple schema linking [10,13,24], and applying constrained decoding to ensure syntactic validity [44]. These methods establish reliable baselines for encoding and constraining schema information. LLM-based Text- to-SQL methods have shifted Text-to-SQL toward prompt-centric and agentic paradigms [15,29,32], including in-context learning (ICL), fine-tuning (FT), and agentic workflows. ICL exploits prompt- ing via decomposition and execution-guided refinement [15,29,39], improving schema grounding through retrieval and dynamic few- shot construction [28,41,48]. FT approaches adapt models for Pass 154 (85.6%) 2.8% 1.7% 2.8% 3.3% 2.8% 1.1% 0%20%40%60%80%100% Outcome (n = 61)Fail 13.1% Failure reasons (n = 8) Pass 86.9% 12.5%12.5% Wrong boundary 50.0% 12.5%12.5% (a) Outcome and failure breakdown (b) LOC audit summary Wrong fact Incomplete answer Unsupported assertion Wrong boundary Irrelevant answer Other Figure 9: Human audit (a) Pass rate and failure reasons over all auditedask_userreplies. (b) The same audit restricted to the sampled LOC pool. controllability, leveraging domain-oriented training [23,25,37,46] and structural constraints [42,51]. Agentic workflows treat syn- thesis as an iterative process involving planning and feedback, exemplified by MCTS-based Alpha-SQL [22], multi-agent collab- oration [47], and user-in-the-loop clarification [34,52]. Although BIRD-INTERACT [18] introduces dynamic and stateful multi-turn evaluation for Text-to-SQL, it remains focused on SQL statements and does not evaluate dialogue-based alignment and revision of procedural PL/SQL specifications. Code Language Models. Existing code language models [17,20, 30,43] rely on massive polyglot training data to achieve broad cov- erage. However, this general-purpose objective often leads to subop- timal performance on specialized, low-resource languages such as PL/SQL. The scarcity of high-quality PL/SQL data in public reposito- ries remains a major obstacle to PL/SQL-specific model adaptation. Rather than proposing a new model-specialization method, Pro- cArena complements existing model-centric research with a model- agnostic, executable benchmark for evaluating procedural database programming across multiple development and interaction settings. 7 CONCLUSION We proposed ProcArena, to our knowledge the first benchmark to evaluate LLMs on NL-to-PL/SQL across multiple development sce- narios in both Direct and Interactive modes: an executable suite of 3,998 tasks spanning nine Direct subscenarios and eight paired In- teractive subscenarios in two dialects. Iterative Logic Enhancement, live validation, and scenario-specific adapters constructed verifiable Direct tasks, from which Knowledge Integration and Requirement Perturbation derived paired Interactive tasks whose executable tar- get an isolated Offline Evaluator preserved. Across seven models, the strongest reached 62.2% execution accuracy in the Direct mode and 57.8% in the Interactive mode, interaction cost accuracy on Synthesis and recovered it on Editing, and failures concentrated in control flow and persistent state, the two error types that declara- tive SQL does not have. We released the benchmark, the interaction traces, and the construction pipeline for reproducible evaluation. 12 REFERENCES [1] 2025. Oracle Help Center. https://docs.oracle.com/en/ [2] 2025. PostgreSQL: Documentation. https://w.postgresql.org/docs/ [3] Itzik Ben-Gan, Dejan Sarka, Roger Wolter, Greg Low, Ed Katibah, and Isaac Kunen. 2009. Inside Microsoft SQL Server 2008 T-SQL Programming. Microsoft Press. [4] Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non- Local Relations. ArXiv abs/2106.01093 (2021). [5]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [6]Vinicius Ferreira de Sousa, Cláudio de Souza Baptista, André Luiz Firmino Alves, and Hugo Feitosa de Figueirêdo. 2025. Effectiveness of small and large language models for PL/SQL bad smell detection. In Simpósio Brasileiro de Banco de Dados (SBBD). SBC, 399–412. [7]DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Con- text Intelligence. arXiv preprint arXiv:2606.19348 (2026). [8]Korry Douglas and Susan Douglas. 2003. PostgreSQL: a comprehensive guide to building, programming, and administering PostgresSQL databases. SAMS publish- ing. [9]Steven Feuerstein and Bill Pribyl. 2005. Oracle pl/sql Programming. " O’Reilly Media, Inc.". [10] Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John Robert Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL Easier to Infer from Natural Language Specifications. In Conference on Empirical Methods in Natural Language Processing. [11]GLM-5 Team. 2026. GLM-5: from Vibe Coding to Agentic Engineering. arXiv preprint arXiv:2602.15763 (2026). [12]Google DeepMind. 2025. Gemini 3. https://deepmind.google/models/gemini/. Accessed: 2026-04-18. [13]Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and D. Zhang. 2019. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation. ArXiv abs/1905.08205 (2019). [14] Surabhi Gupta and Karthik Ramachandra. 2021. Procedural Extensions of SQL: Understanding their usage in the wild. Proceedings of the VLDB Endowment 14, 8 (2021), 1378–1391. [15] Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2024. Next-Generation Database Interfaces: A Survey of LLM-Based Text-to-SQL. IEEE Transactions on Knowledge and Data Engineering 37 (2024), 7328–7345. [16] Gansen Hu, Zhaoguo Wang, Chuzhe Tang, Jiahuan Shen, Zhiyuan Dong, Sheng Yao, and Haibo Chen. 2024. WeBridge: Synthesizing Stored Procedures for Large- Scale Real-World Web Applications. Proceedings of the ACM on Management of Data 2, 1 (2024), 1–29. [17]Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al.2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). [18] Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, et al.2026. BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation via Lens of Dynamic Interactions. In International Con- ference on Learning Representations, Vol. 2026. 44285–44333. [19] Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025. 58791–58831. [20]Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A sur- vey on large language models for code generation. arXiv preprint arXiv:2406.00515 (2024). [21]Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. In The Thirteenth International Conference on Learning Representations. [22]Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. 2025. Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search. In Forty-second International Conference on Machine Learning. [23]Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025. OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale. Proc. VLDB Endow. 18, 11 (July 2025), 4695–4709. https://doi.org/10.14778/3749646. 3749723 [24]Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In AAAI Conference on Artificial Intelligence. [25]Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. CodeS: Towards Building Open-source Language Models for Text-to-SQL. Proceedings of the ACM on Management of Data 2 (2024), 1 – 28. [26]Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023. Graphix-T5: Mixing Pre- Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing. ArXiv abs/2301.07507 (2023). [27] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM already serve as a database interface? a big bench for large-scale data- base grounded text-to-SQLs. In Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook, NY, USA, Article 1835, 28 pages. [28]Xiuwen Li, Qifeng Cai, Yang Shu, Chenjuan Guo, and Bin Yang. 2025. AID-SQL: Adaptive In-Context Learning of Text-to-SQL with Difficulty-Aware Instruction and Retrieval-Augmented Generation. 2025 IEEE 41st International Conference on Data Engineering (ICDE) (2025), 3945–3957. [29]Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yu xin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2024. A Survey of Text-to-SQL in the Era of LLMs: Where Are We, and Where Are We Going? IEEE Transactions on Knowledge and Data Engineering 37 (2024), 5735–5754. [30]Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al.2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 (2024). [31] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al.2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021). [32] Yuyu Luo, Guoliang Li, Ju Fan, Chengliang Chai, and Nan Tang. 2025. Natural Language to SQL: State of the Art and Open Problems. Proc. VLDB Endow. 18 (2025), 5466–5471. [33] Xutao Mao, Dixuan Zhang, Yifan Li, Jiaming Hou, Rui Li, YunLong Li, Aoze Zheng, Zhiqiang Zhang, Luo Zhewei, Hongying Zan, et al.2026. LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, Vol. 40. 29958–29966. [34] Aritra Mazumder, Parth Desai, Fuheng Zhao, and Anna Fariha. 2026. Atom- SQL: Interactive Disambiguation of NL-to-SQL via User-Guided Atom-Level Alignment. Proceedings of the VLDB Endowment 19 (2026). [35]Microsoft. n.d.. Azure SQL Database. https://azure.microsoft.com/en-us/services/ sql-database/. Accessed: 2026-01-16. [36] OpenAI. 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/. Accessed: 2026-04-18. [37] Minjun Park, Yongju Seong, Myoseop Sim, Kyungkoo Min, and Stanley Jungkyu Choi. 2026. ReSQL: Self-Improving Framework for Reasoning-Aware Text-to-SQL Dataset Generation. In Findings of the Association for Computational Linguistics: ACL 2026. 33582–33602. [38]Andrew Pavlo. 2017. What Are We Doing With Our Lives? Nobody Cares About Our Concurrency Control Research. In Proceedings of the 2017 ACM International Conference on Management of Data. 3. https://doi.org/10.1145/3035918.3056096 [39]Mohammadreza Pourreza and Davood Rafiei. 2024. Din-sql: Decomposed in- context learning of text-to-sql with self-correction. Advances in Neural Informa- tion Processing Systems 36 (2024). [40] Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. 2025. mHumanEval-a multilingual benchmark to evaluate large language models for code generation. In Proceedings of the 2025 Conference of the Nations of the Amer- icas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 11432–11461. [41]Tonghui Ren, Yuankai Fan, Zhenying He, Ren Huang, Jiaqi Dai, Can Huang, Yinan Jing, Kai Zhang, Yifan Yang, and Xiaoyang Sean Wang. 2024. PURPLE: Making a Large Language Model a Better SQL Writer. 2024 IEEE 40th ICDE (2024), 15–28. [42]Tonghui Ren, Chen Ke, Yuankai Fan, Yinan Jing, Zhenying He, Kai Zhang, and Xiaoyang Sean Wang. 2025. The Power of Constraints in Natural Language to SQL Translation. Proc. VLDB Endow. 18 (2025), 2097–2111. [43]Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). [44]Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 9895–9901. [45]Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A Survey on Employing Large Language Models for Text-to-SQL Tasks. ACM Comput. Surv. 58, 2, Article 54 (Sept. 2025), 37 pages. https://doi.org/10.1145/ 3737873 13 [46]Marko Sterbentz, Kevin Cushing, Cameron Barrie, and Kristian J Hammond. 2026. RingSQL: Generating Synthetic Data with Schema-Independent Templates for Text-to-SQL Reasoning Models. arXiv preprint arXiv:2601.05451 (2026). [47] Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian-Wen Zhang, Zhao Yan, and Zhoujun Li. 2023. MAC-SQL: A Multi-Agent Collaborative Frame- work for Text-to-SQL. In International Conference on Computational Linguistics. [48]Xiangjin Xie, Guangwei Xu, Lingyan Zhao, and Rui Guo. 2025. OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment. Proceedings of the ACM on Management of Data 3 (2025), 1 – 24. [49]Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al.2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3911–3921. [50]Hang Zhang, Chaokun Wang, Hongwei Li, Cheng Wu, Songyao Wang, Yabin Liu, Gengyuan Shi, and Ziyang Liu. 2025. PLForge: Enhancing Language Models for Natural Language to Procedural Extensions of SQL. Proc. ACM Manag. Data 3, 6, Article 348 (Dec. 2025), 28 pages. https://doi.org/10.1145/3769813 [51] Qinggang Zhang, Hao Chen, Junnan Dong, Shengyuan Chen, Feiran Huang, and Xiao Huang. 2025. Structure-Guided Large Language Models for Text-to-SQL Generation. In Forty-second International Conference on Machine Learning. [52]Fuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divyakant Agrawal, and A. El Abbadi. 2024. Sphinteract: Resolving Ambiguities in NL2SQL Through User Interaction. Proc. VLDB Endow. 18 (2024), 1145–1158. 14