Paper deep dive
ProgramTab: Boosting Table Reasoning of LLMs via Programmatic Paradigm
Pei Guo, Enjie Liu, Yunzhi Tan, Mochi Gao, Jianxin Zhang, Ruichao Zhong, Juntao Li, Bo Hu, Zang Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/18/2026, 2:14:47 PM
Summary
The paper introduces ProgramTab, a framework that enhances Large Language Model (LLM) performance on table-based reasoning tasks by using a programmatic paradigm. It addresses limitations of text-to-SQL approaches on unstructured web tables by employing Python code for data preprocessing and normalization, followed by SQL generation for information extraction. The framework utilizes in-context learning to perform row retrieval, column extraction, data redefinition, SQL generation with Chain-of-Thought, and final answer generation, outperforming existing baselines on datasets like WikiTQ and TabFact.
Entities (8)
Relation Signals (7)
ProgramTab → uses → SQL
confidence 95% · SQL generation and executed to obtain the most valuable information.
ProgramTab → uses → Python
confidence 95% · ProgramTab framework... perform tabular data preprocessing with Python code
ProgramTab → improves → Table-based reasoning
confidence 92% · ProgramTab: Boosting Table Reasoning of LLMs via Programmatic Paradigm
ProgramTab → employs → In-Context Learning
confidence 90% · guides LLMs employing in-context learning
ProgramTab → evaluatedon → WikiTQ
confidence 90% · We validate our ProgramTab framework by conducting experiments on... WikiTQ
ProgramTab → evaluatedon → TabFact
confidence 90% · We validate our ProgramTab framework by conducting experiments on... TabFact
ProgramTab → uses → Chain-of-Thought
confidence 88% · SQL queries are generated using chain-of-thought (CoT)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Table-based reasoning with large language models (LLMs), which requires reasoning based on natural language questions and structured tabular data, has gained widespread attention. However, a series of issues still constrain the application of this task. The previous approaches suffered from significant performance degradation when faced with large tables due to the difficulty of long text modeling and the limitation of input length for LLMs. The text-to-SQL approach is used to efficiently extract key information from tables and generate smaller sub-tables. However, tabular data, especially web tables, often lack the necessary structure and consistency, making them unsuitable for performing mathematical logic operations using SQL queries. We propose the ProgramTab framework, which guides LLMs employing in-context learning to perform tabular data preprocessing with Python code, as well as the momentous contents extraction with row and column extraction and SQL generation. The experiment results on table reasoning datasets demonstrate that the ProgramTab framework effectively deals with table-based reasoning tasks and outperforms all LLM-based baselines.
Tags
Links
- Source: https://arxiv.org/abs/2607.11207v1
- Canonical: https://arxiv.org/abs/2607.11207v1
Trouble viewing inline? Open PDF directly →
Full Text
53,179 characters extracted from source content.
Expand or collapse full text
ProgramTab: Boosting Table Reasoning of LLMs via Programmatic Paradigm Pei Guo, Enjie Liu, Yunzhi Tan * , Mochi Gao, Jianxin Zhang, Ruichao Zhong, Juntao Li, Bo Hu * , Zang Li ♠ Big Data and AI Platform Department, Tencent, China ♢ Institute of Computer Science and Technology, Soochow University, China anthonyguo, karolinaliu, boristan, mochigao, harryyfhu, gavinzli@tencent.com; 20204027008@stu.suda.edu.cn; rzhongab@connect.ust.hk; ljt@suda.edu.cn Abstract Table-based reasoning with large language models (LLMs), which requires reasoning based on natural language questions and struc- tured tabular data, has gained widespread atten- tion. However, a series of issues still constrain the application of this task. The previous ap- proaches suffered from significant performance degradation when faced with large tables due to the difficulty of long text modeling and the lim- itation of input length for LLMs. The text-to- SQL approach is used to efficiently extract key information from tables and generate smaller sub-tables. However, tabular data, especially web tables, often lack the necessary structure and consistency, making them unsuitable for performing mathematical logic operations us- ing SQL queries. We propose the ProgramTab framework, which guides LLMs employing in- context learning to perform tabular data pre- processing with Python code, as well as the momentous contents extraction with row and column extraction and SQL generation. The experiment results on table reasoning datasets demonstrate that the ProgramTab framework ef- fectively deals with table-based reasoning tasks and outperforms all LLM-based baselines. 1 Introduction Tables, as a popular form of data representation, play a significant role in everyday work and life. Analysis and reasoning based on tabular data have emerged as a hot topic in natural language process- ing, attracting wide attention from academia and industry. The main downstream tasks of tabular rea- soning include table-based fact verification (Chen et al., 2020; Aly et al., 2021) and table-based ques- tion answering (Panupong and Percy, 2015; Cho et al., 2019). The challenges of these tasks lie in how to enable language models to comprehend table data content, including text, numbers, etc., * Corresponding Author dateopponentresult september 6, 1981 at los angeles rams w 27-20 september 13, 1981 at cleveland browns w 9-3 september 27, 1981miami dolphinsl 10-16 ......... december 20, 1981 pittsburgh steelers w 21-20 Title: 1981 Houston Oilers season Figure 1: An example of a table in WikiTQ dataset. establish their connection with user queries, and ex- ecute efficient logical reasoning and computations. Recently, LLMs (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2022) have significantly trans- formed the landscape of natural language process- ing tasks with their impressive understanding and generation capabilities. Instead of fine-tuning the pre-trained models, sufficiently utilizing the in- context learning of LLMs to solve complex tab- ular data reasoning has been a mainstream direc- tion (Chen, 2023; Cheng et al., 2023; Ye et al., 2023; Wang et al., 2024). However, current meth- ods still face several limitations. Firstly, most of the work (Cheng et al., 2023; Ye et al., 2023; Wang et al., 2024) treats the entire table as an input, which is unsuitable for tables containing large amounts of data. When the number of tokens in a table exceeds the maximum input limitation of LLMs, the content of the table will be truncated, leading to information loss and consequently affecting the performance of LLMs. This has been verified in the work of (Chen, 2023). To mitigate the length constraint of inputs, the common approach is to utilize a programmatic language, such as generat- ing SQL queries to retrieve the most relevant rows and column data (Ye et al., 2023; Nahid and Rafiei, 2024b; Zhang et al., 2024c,a). However, table data, especially the web table is usually provided as 1 arXiv:2607.11207v1 [cs.CL] 13 Jul 2026 strings and often lacks the necessary structure and consistency, requiring conversion to the ap- propriate format and data types for mathemati- cal logic operations to avoid calculation errors. It requires SQL to preprocess the data while ex- tracting the relevant information, which increases the complexity of generating SQL (The results in Table 3 can confirm this opinion). For example, as shown in Figure 1, when the question is about the number of games the Houston Oilers won in 1981 season, the ‘w’ and ‘l’ symbols from the result cell aren’t provided as a single column and need to be extracted, which is defined as "lack of necessary structure". Regarding the absence of consistency, the structure at the "year" column in Figure 2 is inconsistent, such as "1931" and "spring 1932". To address the above challenges, we introduce the ProgramTab framework, which executes with program languages (Python and SQL) to flexibly handle the table contents based on the questions. Specifically, as shown in Figure 2, (1) we utilize the embedding model to compute the relevant scores of each line of tables with the questions and resort the lines in descending order. In the following steps, the top K lines with higher relevant scores are extracted as instances to replace the complete tables. With the most relevant lines as input, (2) LLMs are prompted to select the columns related to the questions, (3) generate the Python code to preprocess the table data, including unifying the data format and defining the data type for each column. After that, (4) SQL queries are generated using chain-of-thought (CoT) (Wei et al., 2023) and executed to obtain the most valuable information. Finally, (5) LLMs process this information and produce the final answers. We validate our ProgramTab framework by con- ducting experiments on two challenging table rea- soning datasets: WikiTQ (Panupong and Percy, 2015) and TabFact (Chen et al., 2020). With three LLM backbones, our evaluation demonstrated that ProgramTab achieves excellent performance on table-based reasoning benchmarks, and outper- forms all the other baselines with different LLM backbones. Besides, ProgramTab is not limited by the input length of table data, which obtains a sig- nificant efficiency and effectiveness improvement compared with other strong baselines. 2 Related Work In this section, we introduce the related approaches of table-based reasoning: fine-tuning-based and prompting-based table reasoning. 2.1 Fine-tuning-based Table reasoning Table-based understanding and reasoning tasks are significant in data analysis systems. Many ap- proaches focus on constructing pre-trained lan- guage models and fine-tuning them to address these tasks (Zhang et al., 2020; Patnaik et al., 2024). Among them, mask language models (MLM) are widely adopted. For example, TaPas (Herzig et al., 2020) acquires BERT (Devlin et al., 2019) to parse table information via pre-training. PASTA (Gu et al., 2022) pre-trains DeBERTaV3 (He et al., 2021) to perform six types of common sen- tence–table cloze tasks. Besides, TAPEX (Liu et al., 2022) employs the BART (Lewis et al., 2020) model to learn the neural SQL executors over a syn- thetic corpus. OmniTab (Jiang et al., 2022) lever- ages retrieval to pair relevant natural sentences with mask-based pre-training and synthesizes natural language questions by converting sampled SQL from tables. Inner Table Retrieval (ITR) (Lin et al., 2023) extracts sub-tables to preserve the most rele- vant information for the questions. 2.2 Prompting-based Table Reasoning Recently, LLMs (Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023) have gained widespread attention due to their powerful understanding and generation capabilities. Given a few augmenting few-shot examples relevant to the tasks, the LLMs can tackle various reasoning tasks (Fu et al., 2023; Zhang et al., 2023). A few approaches also employ LLMs to tackle table reasoning tasks with few-shot prompts. TableCoT (Tai et al., 2023) systemati- cally explores the performance of LLMs on table reasoning tasks and finds that LLMs are excellent at solving such tasks, especially combined with CoT approach. Besides, rather than generating general text, additional programmatic text, such as Python programs (Chen et al., 2022; Gao et al., 2023), and Text-to-SQL (Rajkumar et al., 2022) approaches are employed to improve the perfor- mance further. LEVER (Ni et al., 2023) improves the performance of code LLMs on language-to- code tasks by training separate verifiers to validate the programs generated by LLMs and their exe- cution results. Binder (Cheng et al., 2023) maps 2 deffunction(df): # step 1: standardize the data format df['year'] = df['year'].str.extract('(\ 4)‘) # step 2: set the data type for each column df['year'] = df['year'].astype(int) df['national_cup'] = df['national_cup'].astype(str) # step 3 (optional): extract the information df['years_to_win_after_1936'] = df['year'] -1936 returndf # call the function, don't change dfanymore df= function(df) yearnational cup 1936/1937champion 1931n/a ...... yeardivisionplayoffsnational cup 1936/37n/anot qualifychampion 19311no playoffn/a ............ yeardivisionplayoffsnational cup 19311no playoffn/a spring 19321no playoff1st round fall 19321no playoffn/a 1933/34n/ano playoff? ............ 1936/37n/anot qualifychampion ............ Question(Q): how long did it take for the new yorkamericansto win the national cup after 1936? Title: New York Americans (soccer) ⑤ Answer Generation year national cup years_to_w in_after_19 36 1936champion0 1931n/a-5 ......... step1: select the `years_to_win_after_1936` where `national_cup` = 'champion' SQL1: SELECT `years_to_win_after_1936` FROM test WHERE `year` > 1936 AND `national_cup` = 'champion' years_to_win_after_1936 17 The 'year' column will help us identify the year after 1936 when they won the National Cup, and the 'national_cup' column will indicate whether they were the champions in that specific year. The answer is year, national_cup. Therefore, the answer is 17 years. ④ SQL Generationwith CoTand Execution ③ Data Redefinition with Python Code and Execution ② Column Extraction ① Row Retrieval Execute Question(Q): how long did it ... after 1936? Question(Q):how long did it ... after 1936? Question(Q):how long did it ... after 1936? Question(Q):how long did it ... after 1936? Figure 2: The overview of ProgramTab for table-based reasoning. the task input to a program that allows generat- ing SQL or Python programs and extending their functions by calling LLMs in the program. Re- AcTable (Zhang et al., 2024b) breaks down the problem into multiple steps and uses LLMs to gen- erate code programs that are executed through ex- ternal tools for each step. Finally, it leverages ma- jority voting to improve overall accuracy. Wang et al. (2024) proposes a Chain-of-Table framework that designs a series of table operations and dynam- ically plans an operation chain based on the inputs. It’s difficult for LLMs to perform reasoning when confronted with large tables with multiple rows. Dater (Ye et al., 2023), TabSQLify (Nahid and Rafiei, 2024b) and H-STAR (Nikhil et al., 2024) decompose the original table into the sub-table by selecting the relevant rows and columns. After that, Dater and Alter (Zhang et al., 2024a) also propose the parsing-execution-filling and query augmenta- tion strategy respectively to decompose a complex question into simpler step-by-step sub-questions by generating an intermediate SQL.E 5 (Zhang et al., 2024c) presents an algorithm to condense large tables while maintaining useful information. Unlike above works, which typically extract in- formation directly using SQL queries to obtain answers—thus increasing the difficulty of SQL generation, we propose an innovative approach that leverages LLMs to generate code for data preprocessing. Besides, the most similar work is NormTab (Nahid and Rafiei, 2024a), which uti- lizes LLMs to regularize table content, making it conducive to SQL query generation. ProgramTab leverages LLMs to generate relevant processing code for table normalization, whereas NormTab has LLMs directly output the entire table in text form. In comparison, ProgramTab is more efficient, and through code execution, it reduces the likelihood of inconsistencies in the processing of data within the same column. Additionally, we propose an improvement for SQL genera- tion by decomposing the question into multiple sub-questions and providing the corresponding SQL for each sub-question, aiming to further enhance performance. 3 ProgramTab Reasoning As shown in Figure 2, ProgramTab consists of five procedures: 1) row retrieval, 2) column extrac- tion, 3) data definition with code, 4) SQL gener- ation and 5) answer generation. In this section, we describe the above procedures in detail. The original table is denoted as T . 3.1 Row Retrieval To alleviate the limitation of the input length of LLMs, we first execute row retrieval, extracting the most relevant rows to represent the entire table content. Specifically, for each row of data in the table, we concatenate the column name and value 3 ### Task description: Please select the relevant columns about the question from ### Table headers. We also provide a few rows about the value of different headers to assist to choose the columns. Please follow the format that describe the reasons of choosing the column firstly and give the conclusion as 'The answer is' finally. ###Example1: Question: which team won previous to crettyard? ### Table header: team | county | wins | years won ### A few rows: row1: Greystones | Wicklow | 1 | 2011 row2: Ballymore Eustace | Kildare | 1 | 2010 row3: Maynooth | Kildare | 1 | 2009 ### Answer: To find out which team won before crettyard, we need to look at the 'years won' column to determine the year crettyardwon and find the team that won the year before. Besides, the 'team' column is also crucial because it has the names of the teams that won in those years. The answer is team, years won. ###Example2: Question: did february2012 or july2006 have more total votes? ### Table header: polling_firm| month | link | favor | oppose ### A few rows: row1: utgers-eagleton| march 2014 | | 64 | 28 row2: quinnipiac| july2013 | | 60 | 31 row3: rutgers-eagleton| june2013 | | 59 | 30 ### Answer: To answer the question about whether February 2012 or July 2006 had more total votes, we need to look at the 'month' column to find the data for these two specific months. In addition, the 'favor' and 'oppose' columns are also important because they contain the number of votes. By adding these two columns together, we can get the total votes for each month. The answer is month, favor, oppose. ### Problem to be solved: Question: ### Table header: ### A few rows: ### Answer: Figure 3: Prompt for LLMs to extract columns. of the cells to form a text segment, and an embed- ding model is utilized to calculate the relevance score between the row data and the question. Ulti- mately, the top K most relevant rows are selected as instances in the prompt templates of the following steps, which effectively alleviates the whole table as the input context. The value of ‘K’ is determined based on the context length supported by the model being used. We set K to 10 for all the models. 3.2 Column Extraction To minimize the impact of irrelevant data, it is essential to extract the relevant columns and utilize them for LLMs to conduct reasoning (Zhang et al., 2024a). As shown in Figure 3, given the question, table header, and top K rows as input context, we prompt LLMs to follow the examples and extract the related columns with additional explanation. Based on the LLMs filter columns, we extract them from T and obtain T col . 3.3 Data Redefinition with Code Specifically, to maintain the flexibility of table data, the string type is adopted for the table data espe- cially collected from the web. Besides, the format of data is not always consistent which causes a great challenge for SQL generation. For example, as shown in Figure 2, the values of columnyearin Tare not rigorous, which conclude three different formats with string type: ‘1931’, ‘spring 1932’, ### Task description: Based on the questions and current data format with python, define a new function to supplement subsequent processing operations with python according to the following steps: -step 1 (optional): standardize the data format for each column. -step 2: set the data type for each column (the type must be one of int, float, string and datetime). -step 3 (optional): extract the useful information by adding new columns if needed. -Note: you just need to output the code, don't answer the questions and don't redefine the df. Here are some examples to help you understand this task. ###Example1: Question: how many people stayed at least 3 years in office? ### Current code of data format: data = 'took_office': ['march 4, 1803', 'march 4, 1809', 'march 4, 1815'], 'left_office': ['march 3, 1809', 'march 3, 1815', 'april18, 1816’] df= pd.DataFrame(data) ### Answer: def function(df): # step 1: standardize the data format: the question only need the years df['took_office'] = df['took_office'].map(lambda x: x.split(', ')[-1]) df['left_office'] = df['left_office'].map(lambda x: x.split(', ')[-1]) # step 2: set the data type: set the year to int type df['took_office'] = df['took_office'].astype(int) df['left_office'] = df['left_office'].astype(int) # step 3: extract the information df['stayed_years_in_office'] = df['left_office'] - df['took_office’] return df # call the function, don't change dfanymore df= function(df) ###Example2: Question: what's the number of parishes founded in the 1800s? ### Current code of data format: data = 'parish': ['stmary', 'the immaculate conception', 'stjamesthe less'], 'founded': ['1852', '1854', '1828’] df= pd.DataFrame(data) ### Answer: def function(df): # step 1: all data format are the same, there is no need to change # step 2: set the data type df['parish'] = df['parish'].astype(str) df['founded'] = df['founded'].astype(int) # step 3: there is no data need to extract return df # call the function, don't change dfanymore df= function(df) ### Problem to be solved: Figure 4: The prompt for LLMs to perform data redefi- nition with code. and ‘1933/34’. Therefore, it’s necessary to rede- fine data, including unifying the format, defining the data type, and extracting additional information (the detailed discussion about data redefinition is presented in Section 5.1). The related prompt is shown in Figure 4, given the current data format code, we acquire LLMs to generate Python code with the following steps. Firstly, if there is column data with inconsistent formats, standardize it to form a unified format. Besides, the data type for each column must be set to make it suitable for performing mathematical logical operations, such as defining the column to the integer type. We re- quire that the data type must be one of integer, float, string, and datetime types. Finally, additional infor- mation could be extracted by adding new columns. Among them, annotations are added for each step to benefit LLMs to follow the above steps more effectively. Besides, steps 1 and 3 are optional, depending on the specific cases. For instance, the formats of each column in Figure 4 example 2 are consistent, there is no need to extract extra infor- 4 ### Task description: You are a data scientist specializing in text-to-SQL tasks. Given the Question and Database schema, you should decompose the question into multiple steps and generate their SQL. Please add the `` for every column you select in the SQL. Note: Just output the step and sql. Here are some examples to help you understand this task. ### Example1: Database title: The table about 2000 Olympic Games. Database scheme: CREATE TABLE test( "race_name" TEXT; VALUES: [vueltaa guatemala, vueltaa colombia], "winner_country" TEXT; VALUES: [usa, aus]) ### Question: who won more races, the usaor australia? ### Sub-step and their SQL: step1: obtain the count of usawin. SQL1: SELECT COUNT(`race_name`) FROM test WHERE `winner_country` = 'usa’ step2: obtain the count of auswin. SQL2: SELECT COUNT(`race_name`) FROM test WHERE `winner_country` = 'aus’ step3: Compare the count of usaand aus, and select the most SQL3: SELECT `winner_country`, won_countFROM (SELECT `winner_country`, COUNT(`race_name`) AS won_countFROM test WHERE `winner_country` in ('usa', 'aus')) ORDER BY won_countDESC LIMIT 1 ### Example2: Database title: The table about members of Third Incarnation of Lachlan. Database scheme: CREATE TABLE test( "row_id" INTEGER; VALUES: [0, 1], "member" TEXT; VALUES: [john ryan, jamesmartin], "term" TEXT; VALUES: [1859-1864, 1864-1869]) ### Question:of the members of the third incarnation of the lachlan, who served the longest? ### Sub-step and their SQL: step1: obtain the all `member` and sorted them based on `term` with DESC SQL1: SELECT `member` FROM test ORDER BY `term` DESC step2: select the longest `member` by LIMIT 1 SQL2: SELECT `member` FROM test ORDER BY `term` DESC LIMIT 1 ### Problem to be solved: Figure 5: Prompt for LLMs to generate SQL with CoT. mation. As a result, steps 1 and 3 are unnecessary. 3.4 SQL Generation The table data after redefining is unified and meets the requirements for SQL execution. In this step, we make use of few-shot learning to prompt LLMs to perform SQL generation. Specifically, as pre- sented in Figure 5, the essential information is pro- vided, such as the database title, schema, and top K relevant rows. With these contexts, LLMs are prompted to decompose the question into multi- ple steps and generate their sub-SQL with the CoT method. We find that the CoT style is beneficial for LLMs to generate the final SQL queries, and the specific analysis is described in Section 5.1. 3.5 Answer Generation After executing the SQL query obtained from the previous step, the most relevant information is gained from the table. As presented in Figure 6, during this step, based on the results from executing the SQL query and the question, we utilize LLMs to reason with the additional explanation and fi- nally make a conclusion. Consequently, we can conveniently extract the results from the conclu- sions as final answers. This approach helps LLMs concentrate on the relevant parts to understand the context and answer the questions. ### Task description: Based on the table title, question and execution result of the sqlquery bellow, find the answer to the given question correctly. If there are multiple answers, please split them by ' | '. Note: Only choose the answers from SQL Answer. Table_title: piotrkędzia Question: in what city did piotr'slast 1st place finish occur? SQL: select `year`, `venus` from test where `position` = '1st' order by `year` DESC LIMIT 1 SQL Answer: Table Schema: year | venus Values: 2007 | bangkok, Thailand A: The SQL answer contain the year and venusabout piotr'slast 1st place, and the question ask about the city which means the venus, and the city where Piotr's last 1st place finish occurred is Bangkok, Thailand. Therefore, the answer is bangkok, thailand. Table_title: playa de orointernational airport Question: how many more passengers flew to losangelesthan to saskatoon from manzanillo airport in 2013? SQL: select `city`, `passengers` from test where `city` in ('united states, losangeles', 'canada, saskatoon’); SQL Answer: Table Schema: city | passengers Values: united states, losangeles| 14,749 canada, saskatoon | 2,282 A: The SQL answer contains the number of passengers who flew to los angelesand saskatoon from manzanillo airport 14,749, 2,282. So, the difference in the number of passengers between losangelesand saskatoon is 14,749 -2,282 = 12,467. Therefore, the answer is 12,467. ### Problem to be solved: Figure 6: Prompt for LLMs to generate final answers. 4 Experiments 4.1 Datasets We design relevant prompts and utilize the pow- erful in-context learning ability of LLMs to di- rectly reason on the test set.We evaluate the proposed ProgramTab on three public table reasoning benchmarks: TabFact (Chen et al., 2020), WikiTQ (Panupong and Percy, 2015) and HiTab (Cheng et al., 2022). Among them, Tab- Fact is a table-based binary fact verification bench- mark. Given a statement, we need to ascertain the truthfulness of it based on the table. We report the accuracy of the test set, which contains 2,024 statements and 298 tables. Besides, WikiTQ is one of the most commonly used and highly com- plex datasets, collected and annotated based on Wikipedia tables. The WikiTQ comprises 4,344 question-answer pairs in the test set. HiTab is the dataset that contains hierarchical tables with com- plex hierarchical indexing. 4.2 Baselines We divide the baselines into two categories: (1) approaches that spend additional computing re- sources to train proprietary models with custom training data, such as TaPas (Herzig et al., 2020), GraPPa (Yu et al., 2021), TAPEX (Liu et al., 2022), PASTA (Gu et al., 2022), TaCube (Zhou et al., 2022), OmniTab (Jiang et al., 2022), ITR (Lin et al., 2023) and CABINET (Patnaik et al., 2024). (2) without training, approaches that design few shot prompts and employ the in-context ability of 5 LLMs, such as TableCoT (Tai et al., 2023), Re- AcTable (Zhang et al., 2024b), Binder (Cheng et al., 2023), Dater (Ye et al., 2023), Chain-of- Table (Wang et al., 2024), Alter (Zhang et al., 2024a),E 5 (Zhang et al., 2024c), NormTab (Nahid and Rafiei, 2024a), TabSQLify (Nahid and Rafiei, 2024b) and H-STAR (Nikhil et al., 2024). 4.3 Implementation Details In our settings, we conduct experiments by uti- lizing closed-source LLMs (GPT-3.5-Turbo and GPT-4o-mini 1 ) and the open-source LLM Llama- 3.1-70B-Instruct 2 as the backbones. The prompt templates for each procedure are described in Sec- tion 3. Besides, the details of hyper-parameters are presented in Appendix A.2. Notably, syntax errors occasionally occurred during data redefinition and SQL generation, resulting in non-executable code. To address this issue, we adopted a retry mecha- nism. Specifically, when a runtime error occurred during both processes, we attempted to rerun the process, with a maximum of five attempts. If all five attempts failed, it was concluded that LLMs were unable to handle the given table, and no fur- ther steps were executed. About the evaluation metrics, we follow Nahid and Rafiei (2024b) to use the official denotation accuracy and employ the binary classification accuracy for WikiTQ and Tab- Fact datasets evaluation respectively. GTE-base (Li et al., 2023) is employed as the embedding model. 4.4 Results As presented in Table 1 and Table 2 (the additional results on HiTab in Appendix A.5.), (1) the previ- ous work, training with specific tasks perform well. Specifically, PASTA (Gu et al., 2022) achieves 90.8% accuracy on TabFact, while CABINET (Pat- naik et al., 2024) obtains 69.1% on WikiTQ. Us- ing GPT-4o-mini as the backbone, ProgramTab achieved performance comparable to PASTA on the TabFact dataset. Furthermore, on the WikiTQ dataset, ProgramTab outperformed CABINET re- gardless of the large model used as its backbone. Due to unnecessary additional fine-tuning, the gen- eralization of ProgramTab is better. (2) Compared to previous work without training, ProgramTab with different LLM backbones outperforms the other baselines on all evaluation benchmarks. In ad- dition, our framework with GPT-4o-mini achieves 1 https://openai.com/index/gpt-4o-mini-advancing-cost- efficient-intelligence/ 2 https://ai.meta.com/blog/meta-llama-3-1/ MethodsBackboneAccuracy Previous Work with Training TaPasBERT83.9 TapexBART86.7 PASTADeBERTaV390.8 Previous Work without Training E 5 GPT-488.7 ReAcTable GPT-3.5-Turbo 73.1 TableCoT73.1 Binder79.1 Dater78.0 Alter84.3 NormTab68.9 TabSQLify79.5 H-STAR85.0 ProgramTab (Ours)85.9 Binder Llama-3.1-70B-Instruct 78.1 Dater81.6 Chain-of-Table85.6 TabSQLify70.7 ProgramTab (Ours)86.8 Binder GPT-4o-mini 84.6 Dater80.9 Chain-of-Table84.2 TabSQLify78.7 H-STAR89.4 ProgramTab (Ours)89.6 Table 1: Accuracy of ProgramTab compared to the base- lines on TabFact test set. better performance compared toE 5 with GPT-4. (3) With stronger coding and reasoning abilities, ProgramTab with Llama-3.1-70B-Instruct and GPT- 4o-mini achieve better performance. 5 Analysis 5.1 Ablation Study Results To evaluate the effectiveness of each procedure in the ProgramTab framework, we pay attention to two important steps: data redefinition (DR) and SQL generation (SG). Specifically, we remove the DR procedure described in Section 3.3 and keep the other steps unchanged. The result in Table 3 shows that without the DR step to preprocess the tabular data, it will require SQL to preprocess the data and extract the relevant information, which increases the complexity of generating SQL for LLMs. Therefore, the performance significantly decreases on both datasets, especially on WikiTQ which is more complex. This conclusion is also ver- ified by Wang et al. (2024). Besides, we replace the procedure described in Section 3.4 with the SQL generation without CoT (denotes as w/o CoT SG). The special prompt is shown in Appendix A.1. Ta- ble 3 presents that the performance of ProgramTab w/o SG CoT drops when discarding question de- composition. It verifies that compared with direct SQL generation, decomposing the questions into multiple steps and generating their sub-SQL is ef- fective in reducing the difficulty of SQL generation. 6 MethodsBackboneAccuracy Previous Work with Training TaPasBERT48.8 GraPPaRoBERTa52.7 Tapex BART 57.5 TaCube60.8 OmniTab62.8 ITR63.4 CABINET69.1 Previous Work without Training TableCoT Codex 48.8 Binder61.9 ReAcTable65.8 Dater65.9 E 5 GPT-465.5 ReAcTable GPT-3.5-Turbo 52.5 TableCoT52.4 Binder55.4 Dater52.8 Alter67.4 TabSQLify64.7 NormTab61.2 H-STAR69.6 ProgramTab (Ours)70.3 Binder Llama3.1-70B-Instruct 50.5 Dater43.5 Chain-of-Table62.2 TabSQLify55.8 ProgramTab (Ours)75.5 Binder GPT-4o-mini 58.8 Dater58.3 Chain-of-Table55.6 TabSQLify57.0 H-STAR74.9 ProgramTab (Ours)76.0 Table 2: Performance of ProgramTab compared to the baselines on WikiTQ test set. 5.2Performance Analysis under Large Tables As described in Section 1, Chen (2023) and Ye et al. (2023) have presented that LLMs suffer from significant performance degeneration when deal- ing with large tables. To evaluate the effectiveness of ProgramTab, we extract the large tables from WikiTQ and TabFact datasets. Specifically, we de- fine the large tables for WikiTQ when the token counts are larger than 4000 because 4000 tokens are the maximum token limitation for GPT-3.5-Turbo. Besides, We follow Nahid and Rafiei (2024b) to choose 1200 tokens for TabFact because the ta- bles almost contain few data. We then compare ProgramTab with Binder, Dater, Chain-of-Table, TableCoT, and TabSQLify. As shown in Table 4, we observe that all strong baselines suffer from a significant decline in performance on two datasets. For example, Binder with Codex merely achieves 29.6% accuracy on the WikiTQ dataset and even can’t be applied when utilizing GPT-3.5-Turbo as the backbone. Besides, TabSQLify obtains subop- timal performance thanks to its effective extraction of columns and rows employing the text-to-SQL method. In contrast, ProgramTab significantly out- performs all baselines and even improves compared MethodsTabFactWikiTQ Binder79.155.4 Dater78.052.8 TabSQLify79.564.7 ProgramTab85.970.3 w/o DR81.6 (↓ 4.3)59.4 (↓ 10.9) w/o CoT SG84.1 (↓ 1.8)65.0 (↓ 5.3) Table 3: Ablation results of GPT-3.5-Turbo-based Pro- gramTab with and without data redefinition and CoT SQL generation. MethodsBackboneTabFactWikiTQ BinderCodex-29.6 Chain-of-TableGPT-3.5-Turbo-16k-0613-44.8 Binder GPT-3.5-Turbo -0.0 Dater-34.6 TableCoT55.535.1 TabSQLify72.8 52.3 ProgramTab86.668.0 Table 4: Performance of ProgramTab and strong base- lines on large tables from TabFact and WikiTQ. with performance on the full TabFact dataset. It could be clarified that the row retrieval and column extraction procedures in our framework are effec- tive in providing the relevant rows as the context, which is beneficial for SQL generation to extract the final information from the large tables. 5.3 Robustness Analysis Following Nahid and Rafiei (2024b), we verify the robustness of ProgramTab based on the different cutoff thresholds. Specifically, the cutoff thresholds are established to discard tabular tokens exceeding these limits. For example, if the original table has 800 tokens and the maximum threshold is set to 600, it means that 200 tokens of the original ta- ble are truncated, and the percentage is 200/800 = 25.0%. In our experiment, we set the cutoff thresh- old at 2000 and 600 for WikiTQ and TabFact re- spectively. Table 5 shows four categories based on the above thresholds and presents that ProgramTab with GPT-3.5-Turbo outperforms TabSQLify ex- cept on the 25%-50% cutoff on WikiTQ. The re- sults further demonstrate that the ProgramTab can extract the relevant information under limited token boundary conditions and is not sensitive to input length limitations for LLMs. 7 MethodsDatasets Cut-off(%) 0-10%10-25%25-50%50%+ TabSQLify WikiTQ 64.660.666.356.2 ProgramTab70.862.462.668.0 TabSQLify TabFact 79.180.870.072.8 ProgramTab89.086.577.586.4 Table 5: Performance of ProgramTab on the different cutoff thresholds categories. Methods# of samples / stepTotal # of samples BinderNeural SQL: 5050 Dater Decompose Table: 40 100 Generate Cloze: 20 Generate SQL: 20 Query: 20 Chain-of-Table Dynamic Plan≤ 5 ≤ 25 Generate Args≤ 19 Query: 1 TabSQLify Decompose Table: 1 2 Query: 1 ProgramTab Column Extraction: 1 4 Data Redefinition with Code: 1 Generate SQL: 1 Query: 1 Table 6: The number of samples generated by different methods adopting LLMs. 5.4 Efficiency Analysis Following Wang et al. (2024), we analyze the ef- ficiency of ProgramTab from two aspects. (1) We evaluate the number of samples generated by LLMs. For each reasoning step, compared to the approaches that apply the self-consistency (Binder and Dater) strategy to generate multiple samples or adopt the iterative sample creation process (Chain- of-Table), ProgramTab adopts a greedy search strat- egy to produce a single response. Specifically, Ta- ble 6 shows the number of samples generated by LLMs for a single question in different methods on the WikiTQ dataset. We can find that LLMs are required to generate multiple samples for Binder and Dater, while Chain-of-Table adopts a more efficient approach to reduce the number of sam- ples. TabSQLify achieves the minimum number of samples. Our approach adopts a greedy search strategy to obtain one response for each step, for a total of only four samples. (2) We also validated the end-to-end performance of ProgramTab on the entire test set. For each sample, ProgramTab re- quires an average time of 4 seconds to generate the final answer. In comparison, Dater requires generating 100 samples, with an average time of over 10 seconds. Clearly, our method has signifi- 16% 31% 22% 31% TabFact Missing Columns Error 12% 52% 32% 4% WikiTQ Missing Columns Error SQL Error Code Error Reasoning Error Figure 7: Statistic of different error types on TabFact and WikiTQ datasets. cantly lower latency. Consequently, ProgramTab efficiently reduces computation time and resource costs and performs better. 5.5 Error Analysis To systemically analyze the shortcomings of pro- gramTab with GPT-3.5-Turbo, we select two test sets (i.e., TabFact, and WikiTQ), and randomly choose 100 error samples from each dataset. Then, we manually examine these failures and they are classified into four error categories: 1) Missing Columns Error: LLMs don’t select the relevant columns. 2) SQL Error: the generated SQL queries incorrectly filter the relevant information or contain syntax rule errors. 3) Code Error: the generated Python codes fail to unify the format and type of data, or introduce irrelevant information. 4) Rea- soning Error: LLMs fail to generate the correct answers given the extracted relevant information. It is worth noting that we additionally verify the performance of row retrieval in Appendix A.4. As shown in Figure 7, we can observe that the missing column and reasoning errors respectively account for a small portion of TabFact and WikiTQ. The main source of errors focuses on the code and SQL errors, especially on the WikiTQ. We replaced GPT- 3.5-Turbo with GPT-4o-mini for code and SQL generation, and found that GPT-4o-mini effectively avoids the errors encountered with GPT-3.5-Tubo. The performance of these two LLMs in Table 1 and 2 can also be verified. Consequently, enhanc- ing the capacity of code generation is effective in improving the performance further. 6 Conclusion In this paper, we illustrate the limitations of cur- rent table-based reasoning with LLMs approaches, including suffering from significant performance degradation when faced with large tables, and the inconsistent table data structure increases the difficulty of SQL generation. Consequently, we propose the ProgramTab framework, which suffi- 8 ciently implements the strong in-context learning ability of LLMs to perform tabular data preprocess- ing with Python code and key information extrac- tion with SQL generation. It achieves the best per- formance compared with the baselines and is not limited by the input length of table data. Hoping this flexible table-based reasoning framework can shed new light on the understanding of prompting LLMs for table understanding. 7 Limitations In this section, we present several of the limitations of our approach - ProgramTab. Firstly, the data redefinition with code can preprocess the table data well, but more preprocessing for more complex table structures should be explored further. What’s more, how to perform row retrieval more efficiently from tables with large amounts of rows is another optimization direction. References Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopou- los, Oana Cocarascu, and Arpit Mittal. 2021. Fever- ous:Fact extraction and verification over un- structured and structured information.Preprint arXiv:2106.05707. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and et al. 2020. Language models are few- shot learners. InAdvancesinneuralinformation processingsystems, pages 33:1877–1901. Wenhu Chen. 2023. Large language models are few(1)- shot table reasoners. InFindingsoftheAssociation forComputationalLinguistics:EACL2023, pages 1120–1130. Association for Computational Linguis- tics. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022.Program of thoughts prompting: Disentangling computation from rea- soning for numerical reasoning tasks.Preprint arXiv:2211.12588. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020.Tabfact:A large-scale dataset for table-based fact verifica- tion.InInternationalConferenceonLearning Representations(ICLR). Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022.HiTab: A hierarchical table dataset for question answering and natural language generation. InProceedings ofthe60thAnnualMeetingoftheAssociation forComputationalLinguistics(Volume1:Long Papers), pages 1094–1110. Association for Compu- tational Linguistics. Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models in symbolic languages.InInternational ConferenceonLearningRepresentations(ICLR). Minseok Cho, Gyeongbok Lee, and Seung won Hwang. 2019. Explanatory and actionable debug- ging for machine learning: A tableqa demonstra- tion. InProceedingsofthe42ndInternationalACM SIGIRConferenceonResearchandDevelopmentin InformationRetrieval, page 1333–1336. ACM. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. InProceedingsofthe2019Conference oftheNorthAmericanChapteroftheAssociation forComputationalLinguistics:HumanLanguage Technologies,Volume1(LongandShortPapers), pages 4171–4186. Association for Computational Linguistics. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. InAdvancesinInternational ConferenceonLearningRepresentations. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. Pal: Program-aided language models. InInInternationalConferenceonMachine Learning, pages 10764–10799. Zihui Gu, Ju Fan, Nan Tang, Preslav Nakov, Xi- aoman Zhao, and Xiaoyong Du. 2022. PASTA: Table-operations aware fact verification via sentence- table cloze pre-training.InProceedingsofthe 2022ConferenceonEmpiricalMethodsinNatural LanguageProcessing, pages 4971–4983. Association for Computational Linguistics. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTav3: Improving deBERTa using ELECTRA- style pre-training with gradient-disentangled embed- ding sharing.PreprintarXiv:2111.09543. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisen- schlos. 2020.TaPas:Weakly supervised ta- ble parsing via pre-training.InProceedingsof the58thAnnualMeetingoftheAssociationfor ComputationalLinguistics, pages 4320–4333. Asso- ciation for Computational Linguistics. Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego De, Las Casas, Lisa Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, 9 Katie Millican, GeorgeVanDen Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.PreprintarXiv:2203.15556. Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neu- big, and Weizhu Chen. 2022. OmniTab: Pretraining with natural and synthetic data for few-shot table- based question answering. InProceedingsofthe 2022ConferenceoftheNorthAmericanChapter oftheAssociationforComputationalLinguistics: HumanLanguageTechnologies, pages 932–942. As- sociation for Computational Linguistics. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. InProceedingsofthe58thAnnual MeetingoftheAssociationforComputational Linguistics, pages 7871–7880. Association for Com- putational Linguistics. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning.PreprintarXiv:2308.03281. Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adria de Gispert, and Gonzalo Iglesias. 2023. An inner table retriever for robust table question answering. InProceedingsofthe61stAnnualMeetingofthe AssociationforComputationalLinguistics(Volume 1:LongPapers), pages 9909–9926. Association for Computational Linguistics. Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. Tapex: Table pre-training via learning a neural sql ex- ecutor. InInInternationalConferenceonLearning Representations. Md Mahadi Hasan Nahid and Davood Rafiei. 2024a. NormTab: Improving symbolic reasoning in LLMs through tabular data normalization.InFindings oftheAssociationforComputationalLinguistics: EMNLP2024, pages 3569–3585. Association for Computational Linguistics. Md Mahadi Hasan Nahid and Davood Rafiei. 2024b. TabSQLify: Enhancing reasoning capabilities of LLMs through table decomposition. In2024Annual ConferenceoftheNorthAmericanChapterofthe AssociationforComputationalLinguistics. Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen tau Yih, Sida I Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. InProceedingsofthe 40thInternationalConferenceonMachineLearning (ICML’23). Abhyankar Nikhil, Gupta Vivek, Roth Dan, and Reddy Chandan K. 2024. H-star: Llm-driven hy- brid sql-text adaptive reasoning on tables.Preprint arXiv:2407.05952. OpenAI. 2022. Gpt-3.5-turbo.TechnicalReport. OpenAI. 2023. Gpt-4.TechnicalReport. Pasupat Panupong and Liang Percy. 2015.Com- positional semantic parsing on semi-structured ta- bles. InProceedingsofthe53rdAnnualMeetingof theAssociationforComputationalLinguisticsand the7thInternationalJointConferenceonNatural LanguageProcessing(Volume1:LongPapers), pages 1470–1480. Association for Computational Linguistics. Sohan Patnaik, Heril Changwal, Milan Aggarwal, Sumit Bhatia, Yaman Kumar, and Balaji Krishna- murthy. 2024. Cabinet: Content relevance-based noise reduction for table question answering. In TheTwelfthInternationalConferenceonLearning Representations. Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022.Evaluating the text-to-sql ca- pabilities of large language models.Preprint arXiv:2204.00498. Chang-Yu Tai, Ziru Chen, Tianshu Zhang, Xiang Deng, and Huan Sun. 2023. Exploring chain of thought style prompting for text-to-SQL. InProceedings ofthe2023ConferenceonEmpiricalMethodsin NaturalLanguageProcessing, pages 5376–5393. As- sociation for Computational Linguistics. Touvron, Hugo, Lavril, Thibaut, Izacard, Gautier, Mar- tinet, Xavier, Lachaux, Marie-Anne, Lacroix, Tim- oth’e, Rozi‘ere, Baptiste, Goyal, Naman, Hambro, Eric, Azhar, Faisal, Rodriguez, Aurelien, Joulin, Armand, Grave, Edouard, Lample, and Guillaume. 2023. Llama: Open and efficient foundation lan- guage models.PreprintarXiv:2302.13971. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Mar- tin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. Chain-of-table: Evolv- ing tables in the reasoning chain for table under- standing. InInternationalConferenceonLearning Representations(ICLR). Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompt- ing elicits reasoning in large language models. In Proceedingsofthe36thInternationalConference onNeuralInformationProcessingSystems, pages 24824–24837. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. InInternationalConferenceonLearning Representations(ICLR). 10 Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023.Large lan- guage models are versatile decomposers: Decom- pose evidence and questions for table-based reason- ing. InProceedingsofthe46thInternationalACM SIGIRConferenceonResearchandDevelopmentin InformationRetrieval, pages 174–184. Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, richard socher, and Caiming Xiong. 2021. Grapa: Grammar-augmented pre-training for table seman- tic parsing. InInternationalConferenceonLearning Representations. Han Zhang, Yuheng Ma, and Hanfang Yang. 2024a. Alter: Augmentation for large-table-based reasoning. PreprintarXiv:2407.03061. Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020. Table fact verification with structure-aware trans- former. InProceedingsofthe2020Conferenceon EmpiricalMethodsinNaturalLanguageProcessing (EMNLP), pages 1624–1629. Association for Com- putational Linguistics. Yunjia Zhang, Jordan Henkel, Avrilia Floratou, and Joyce Cahoon. 2024b. Reactable: Enhancing react for table question answering. InProceedingsofthe VLDBEndowment17(8), pages 1981–1994. Zhehao Zhang, Yan Gao, and Jian-Guang Lou. 2024c. e 5 : Zero-shot hierarchical table analysis using aug- mented LLMs via explain, extract, execute, ex- hibit and extrapolate. InProceedingsofthe2024 ConferenceoftheNorthAmericanChapterofthe AssociationforComputationalLinguistics:Human LanguageTechnologies(Volume1:LongPapers), pages 1244–1258. Association for Computational Linguistics. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023.Automatic chain of thought prompting in large language models.In AdvancesinInternationalConferenceonLearning Representations. Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng, Fan Cheng, Shi Han, and Dongmei Zhang. 2022. TaCube: Pre-computing data cubes for answering numerical-reasoning questions over tabular data. In Proceedingsofthe2022ConferenceonEmpirical MethodsinNaturalLanguageProcessing, pages 2278–2291. Association for Computational Linguis- tics. 11 A Appendix A.1 Prompts for SQL Generation without CoT ### Task description:You are a data scientist specializing in text-to-SQL tasks. You should write a valid SQLite to solve the following question based on the database scheme and hint. Please add the `` for every column you select in the SQL. Note: Just output the sql. Here are some examples to help you understand this task. ### Example1: Database title: The table about 2000 Olympic Games. Database scheme: test( \"race_name\" TEXT; VALUES: [vueltaa guatemala, vueltaa colombia], \"winner_country\" TEXT; VALUES: [usa, aus]) ### Question: more races, the usaor australia? ### SQL: SELECT `winner_country`, won_countFROM (SELECT `winner_country`, COUNT(`race_name`) AS won_countFROM test WHERE `winner_country` in ('usa', 'aus')) ORDER BY won_count DESC LIMIT 1 ### Example2: Database title: The table about members of Third Incarnation of Lachlan. Database scheme: test( \"row_id\" INTEGER; VALUES: [0, 1], \"member\" TEXT; VALUES: [john ryan, james martin], \"term\" TEXT; VALUES: [1859-1864, 1864-1869]) ### Question: members of the third incarnation of the lachlan, who served the longest? ### SQL: SELECT `member` FROM test ORDER BY `term` DESC LIMIT 1 ### Example3: Database title: The table about different season of giant slalom and super g. Database scheme: test( \"season\" TEXT; VALUES: [1986, 1987], \"slalom\" TEXT; VALUES: [39, 24], \"giant_slalom\" TEXT; VALUES: [23, 9], \"super_g\" TEXT; VALUES: [19, 18]) ### Question: g had a slalom of less than 5 when the giant slalom was 1? ### SQL: SELECT `super_g` FROM test WHERE `slalom` < 5 AND `giant_slalom` = 1 ### Problem to be solved: Figure 8: Prompt for LLMs to generate SQL w/o CoT. A.2 LLM Hyper-parameters For all the procedures described in Section 3, we set the same hyper-parameters for LLMs. Specifically, the temperature is set to 0.6 while both top_p and the sample number are 1. A.3 Table Size Reduction We analyze the efficiency of ProgramTab in filter- ing irrelevant information and extracting the key tabular data from tables. To accomplish this, we count the average number of table cells that feed LLMs to generate the final answers. As presented in Figure 9, the average number of full table cells (original) is 183 and 101 respectively. There is a significant reduction after employing the Tab- SQLify approach. Our framework ProgramTab employs SQL generation procedure to effectively filter much irrelevant information and extract the most related data. It respectively reduces the aver- age number of table cells to 4 and 15 for WikiTQ 183 101 32 45 4 15 0 20 40 60 80 100 120 140 160 180 200 WikiTQTabFact OriginalTabSQLifyProgramTab Average Number of Table Cells Figure 9: Comparison of the average number of table cells on two datasets. MethodsBackboneHiTab ReAct (Yao et al., 2023)GPT-481.87 E 5 (Zhang et al., 2024c)GPT-485.08 ProgramTabGPT-4o-mini83.57 Table 7: Performance of ProgramTab on HiTab dataset. and TabFact datasets. These results also verify that ProgramTab can perform critical information ex- traction from amounts of tabular cells, and validly deal with large tables. A.4 Performance of Row Retrieval We evaluate the performance of row retrieval stage in ProgramTab. Specifically, the average number of rows per table is 26 in the WikiTQ dataset. In the above experiments, we set K to 10, which means that on average, we can filter out 16 rows per table, resulting in a filtering ratio of 61.5%. Addition- ally, among the top 10 rows retrieved, 86% of the samples contained the final answer. A.5 Experiments on HiTab To present the effectiveness of ProgramTab when applied to more complex tabular structures, we supplemented ProgramTab’s experiments on the HiTab dataset, which contains hierarchical tables. To achieve this, we first reconstructed the hierar- chical tables by merging certain column header in- formation using a ":" delimiter, making them more suitable for processing by ProgramTab. The final experimental results are as follows: ProgramTab with GPT-4o-mini demonstrates promising perfor- mance, while adopting theE 5 method to process hierarchical tables yields even better performance. 12