Paper deep dive
SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows
Jian Zhu, Yuzheng Zhang, Zeyao Ma, Bohan Zhang, Armin Schoepf, Daniel Woloch, Peter Yiliu Wang, Guangyu Robert Yang, Samuel Jacob, Siddharth Nagisetty, Abhiram Chundru, Jean Lin, Spencer Mateega, Jing Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 7/5/2026, 3:11:23 AM
Summary
SpreadsheetBench 2 is a new workflow-level benchmark designed to evaluate spreadsheet agents on end-to-end business tasks including generation (financial modeling/templates), debugging, and visualization. Unlike previous benchmarks focusing on isolated cell edits, it uses authentic business data (financial reports, corporate filings) and complex multi-sheet workbooks with cross-sheet dependencies. The benchmark contains 321 tasks and reveals that even frontier LLMs struggle with real-world spreadsheet workflows, with the best model (Claude Opus 4.6) achieving only 34.89% overall accuracy.
Entities (10)
Relation Signals (6)
Generation â containssubtask â Financial Modeling
confidence 100% ¡ Generation (Financial Modeling and Template)
Claude Opus 4.6 â evaluatedon â SpreadsheetBench 2
confidence 100% ¡ We evaluate eight frontier large language models... the best-performing model, Claude Opus 4.6...
SpreadsheetBench 2 â includestaskcategory â Generation
confidence 100% ¡ covers three task categories: generation, debugging, and visualization.
SpreadsheetBench 2 â includestaskcategory â Debugging
confidence 100% ¡ covers three task categories: generation, debugging, and visualization.
SpreadsheetBench 2 â includestaskcategory â Visualization
confidence 100% ¡ covers three task categories: generation, debugging, and visualization.
SWE-agent â usedasbasefor â Agent Scaffold
confidence 100% ¡ Our agent scaffold is built on SWE-agent
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings. We introduce \textsc{SpreadsheetBench 2}, a workflow-level benchmark for spreadsheet agents that covers three task categories: generation, debugging, and visualization. The benchmark is constructed from authentic business data, including financial reports and corporate filings, and is annotated and validated by domain experts. The benchmark contains 321 tasks; each instance averages 11.8 worksheets and requires 593.5 cell modifications, reflecting large multi-sheet workbooks with cross-sheet dependencies. We evaluate eight frontier large language models under a unified multi-turn agent scaffold, and additionally include several LLM-based spreadsheet products as complementary baselines. Results show that current systems remain far from reliable on real-world workflows: the best model achieves 34.89\% overall task accuracy, and debugging accuracy is as low as 12.00\%. Trajectory analysis and a failure taxonomy further indicate that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks. Together, these findings position \textsc{SpreadsheetBench 2} as a challenging testbed for advancing reliable spreadsheet automation. Project page: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2606.29955v1
- Canonical: https://arxiv.org/abs/2606.29955v1
Trouble viewing inline? Open PDF directly â
Full Text
82,908 characters extracted from source content.
Expand or collapse full text
SPREADSHEETBENCH 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows Jian Zhu 1â Yuzheng Zhang 1â Zeyao Ma 1 Bohan Zhang 1 Armin Schoepf 2 Daniel Woloch 2 Peter Yiliu Wang 3 Guangyu Robert Yang 3 Samuel Jacob 4 Siddharth Nagisetty 4 Abhiram Chundru 4 Jean Lin 4 Spencer Mateega 4 Jing Zhang 1â 1 School of Information, Renmin University of China 2 Aptura AI 3 Shortcut AI 4 AfterQuery Abstract Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate iso- lated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings. We introduce SPREADSHEETBENCH 2, a workflow-level benchmark for spreadsheet agents that covers three task categories: generation, debugging, and visualization. The bench- mark is constructed from authentic business data, including financial reports and corporate filings, and is annotated and validated by domain experts. The benchmark contains 321 tasks; each instance averages 11.8 worksheets and requires 593.5 cell modifications, reflecting large multi-sheet workbooks with cross-sheet dependen- cies. We evaluate eight frontier large language models under a unified multi-turn agent scaffold, and additionally include several LLM-based spreadsheet products as complementary baselines. Results show that current systems remain far from reliable on real-world workflows: the best model achieves 34.89% overall task accu- racy, and debugging accuracy is as low as 12.00%. Trajectory analysis and a failure taxonomy further indicate that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks. Together, these findings position SPREADSHEETBENCH 2 as a challenging testbed for advancing reliable spread- sheet automation. Project page: https://spreadsheetbench.github.io/ 1 Introduction Spreadsheets serve as a core infrastructure for structured data processing in modern organizations, supporting tasks such as financial modeling, reporting, and decision-making [1,2,3,4,5,6,7]. Recent advances in large language models (LLMs) have enabled the development of spreadsheet agents capable of generating formulas, manipulating data, and automating a wide range of spreadsheet operations [8,9,10,11,12,13]. However, despite strong performance on existing benchmarks, these agents remain far from reliable in real-world spreadsheet scenarios. A key limitation lies in current evaluation benchmarks, which primarily assess spreadsheet agents at the level of isolated operations rather than complete workflows [8,14,15]. Existing work often focuses on small or simplified workbooks with limited cross-sheet dependencies [16,17,18,19], emphasizing local cell-level manipulations instead of end-to-end task completion. In addition, many datasets are derived from synthetic sources or community forums, resulting in tasks that are less representative of real-world, professional business spreadsheet workflows [15, 20, 21, 22, 23]. â Equal contribution. â Corresponding author. zhang-jing@ruc.edu.cn Preprint. arXiv:2606.29955v1 [cs.SE] 29 Jun 2026 Debugging â˘Audit and repair the spreadsheet. â˘Identify and fix formula errors (#REF!, #VALUE!). â˘Correct cascading reference issues. Instruction (excerpt) Summary Equity Returns Margin Analysis Corrupted Spreadsheet Coorrected Spreadsheet Financial Modeling / Template â˘Complete the financial model based on the provided assumptions. â˘Fill in the missing quarterly figures. â˘Calculate D/E Ratio for Q1âQ2. Instruction (excerpt) Cost Drivers Ratios Analysis Cash Flow Incomplete Financial Model Completed Financial Model Data Visualization â˘Create a scatter chart from my blood sugar data.. â˘Three categories: Low (<7), Normal (7-12), and High (>12). â˘Add data labels showing the value. Instruction (excerpt) Summary Dashboard Equity Returns Visualization Chart Figure 1: SPREADSHEETBENCH 2 consists of three representative task categories: Debugging, Generation (Financial Modeling and Template), and Data Visualization. Debugging tasks focus on identifying and repairing errors; Generation tasks (Financial Modeling and Template) involve completing or constructing spreadsheets; Visualization tasks require producing analysis-ready charts. Detailed examples of each category are provided in Appendix E. In practice, spreadsheet usage follows a structured workflow. Tasks commonly involve constructing or completing spreadsheet artifacts, identifying and correcting errors, and presenting results through visualization and reporting. These stages introduce challenges such as cross-sheet reasoning, long- range dependencies, and multi-step coordination, which are largely absent from existing benchmarks. To better reflect these characteristics, we introduce SPREADSHEETBENCH 2, a challenging benchmark for evaluating spreadsheet agents on workflow-level tasks grounded in real-world, professional business scenarios. As illustrated in Figure 1, the benchmark organizes tasks into three representative categories: generation (financial modeling and template completion), debugging, and visualization, each corresponding to a key stage in spreadsheet workflows. SPREADSHEETBENCH 2 is constructed from authentic business data sources, including financial reports and corporate filings, and paired with expert-annotated instructions, requiring over 1,500 hours of expert effort (Appendix B.2). The benchmark consists of 321 tasks, each defined by a natural language instruction and an input spreadsheet representing an in-progress business artifact. These tasks involve complex multi-sheet structures, rich cross-sheet dependencies, and substantial modification requirements, making them significantly more challenging than prior datasets. We evaluate eight state-of-the-art large language models under a unified multi-turn agent scaffold, where models interact with spreadsheets through a command-line interface. In addition, we include several LLM-based spreadsheet products (e.g., Claude for Excel, ChatGPT for Excel) as complemen- tary baselines. Despite strong performance on existing benchmarks, current LLMs and spreadsheet products achieve limited success on SPREADSHEETBENCH 2, with overall accuracy remaining below 35% and substantially lower performance on debugging tasks. These results highlight a significant gap between current model capabilities and the requirements of real-world spreadsheet workflows. Further analysis reveals that failures are primarily driven by insufficient inspection and incorrect selection of target cells for modification. Moreover, to isolate the impact of scaffold design, we fix GLM-5 and find that our SWE-agent-based scaffold outperforms three coding agent scaffolds on 50 samples. While modification scores are comparable across scaffolds, task-level accuracy is substantially lower for general-purpose coding scaffolds, indicating weaker end-to-end correctness on spreadsheet workflows. Our contributions are summarized as follows: (1) We introduce SPREADSHEETBENCH 2, a workflow- level benchmark grounded in real-world business settings, featuring expert-annotated tasks over 2 complex multi-sheet spreadsheets with cross-sheet dependencies, thereby shifting evaluation from isolated operations to end-to-end task completion. (2) We provide a systematic evaluation of state-of- the-art large language models under a unified multi-turn agent scaffold, together with complementary evaluation of LLM-based spreadsheet products, and we further benchmark different agent scaffolds to quantify how scaffold design affects end-to-end performance, revealing substantial performance gaps and key failure modes in real-world spreadsheet automation. 2SPREADSHEETBENCH 2 SPREADSHEETBENCH 2 is designed to evaluate spreadsheet agents on realistic, workflow-level tasks in business settings. To achieve this goal, the benchmark is constructed around three key aspects: (1) task formulation based on real-world spreadsheet workflows, (2) data construction grounded in authentic business data, and (3) evaluation protocols that capture both cell-level correctness and task-level outcomes. In this section, we describe task categories, dataset construction, and evaluation metrics. 2.1 Task Categories Real-world spreadsheet usage follows a structured workflow rather than isolated operations. In business settings, tasks typically involve three stages: constructing spreadsheet artifacts, validating and debugging logical dependencies, and communicating insights through visualization and reporting. These stages define the lifecycle of spreadsheet usage in practice. Motivated by this workflow perspective, SPREADSHEETBENCH 2 organizes tasks into three categories: Generation, Debugging, and Visualization. Generation (Financial Modeling and Template). Generation tasks evaluate the ability to construct or complete spreadsheet artifacts from partially specified inputs or high-level objectives. These tasks involve cross-sheet reasoning, formula propagation, and the construction of structured spreadsheet. They correspond to the construction phase of spreadsheet workflows, where users design and populate structured workbooks from scratch or incomplete specifications. We include two levels of difficulty: Template tasks involve simpler, typically single-structure completion (avg. 1.2 sheets), while financial modeling tasks represent more complex scenarios with multiple interdependent components across sheets (avg. 15.3 sheets), requiring coordinated reasoning over large spreadsheet structures. Debugging. Debugging tasks evaluate the ability to identify and repair errors in existing spreadsheets. In real-world settings, the causes and locations of errors are often unknown. Therefore, instructions do not explicitly specify error types or target cells, requiring models to autonomously locate and fix issues. These tasks cover ten error types, including double counting (values are unintentionally aggregated multiple times, leading to systematic overestimation), incorrect references (misconfigured cell links or lookup ranges that break dependencies), formula errors (syntactic or logical mistakes that produce invalid or unintended outputs), and others (Appendix B.4). They correspond to the validation and correction phase of spreadsheet workflows and are particularly challenging, as they require holistic reasoning over complex, interdependent spreadsheet structures without explicit information about error locations or types. Visualization. Visualization tasks evaluate the ability to transform spreadsheet data into analysis- ready visual artifacts, such as charts and pivot-based summaries. In business contexts, spreadsheets function not only as computational tools but also as communication instruments for analysis and decision-making. Unlike standard visualization benchmarks, the input data in these tasks is often irregularly structured, distributed across multiple sheets, or embedded within semi-structured tables, requiring models to correctly interpret and extract relevant information. The target visualizations can involve complex structures, such as multiple data series, hierarchical axes, and customized formatting. These tasks correspond to the presentation stage of spreadsheet workflows and require both accurate data selection and precise visual construction within spreadsheet-native formats. 2.2 Benchmark Construction As shown in Figure 2, SPREADSHEETBENCH 2 is constructed through a three-stage pipeline: data collection, task construction, and data validation. 3 Data Collection Annual Reports, Financial Statements, Investment, etc. Collect Real-World Business Data Transform into Gold- Standard Spreadsheets Financial Experts Annotate and Build Gold-Standard Spreadsheet Solutions Task Construction Financial Modeling/Template Build valuation models, forecasts, financial analyses using spreadsheet formulas. Debugging Please audit and fix this file thoroughly,identify and correct errors in formulas,references, etc. Visualization Create charts and visual summaries within spreadsheets to clearly present key patterns and trends in the data. Expert Annotation Data Validation Independent Review by Two Experts Expert A Compare with Ground Truth Generate Solution Expert B Compare with Ground Truth Generate Solution Compare Outputs Between Experts Agreement & Consistency Check SPREADSHEETBENCH 2 Figure 2: The benchmark construction pipeline of SPREADSHEETBENCH 2. Data Collection. We construct SPREADSHEETBENCH 2 using data from authentic business sources, including publicly available financial reports and corporate filings from the NYU Stern (Damodaran) dataset, Screener.in, Bloomberg, Bseindia (More details in Appendix B.1). Financial experts curate these materials and build complete, internally consistent spreadsheets as gold-standard solutions, with correct formulas, cross-sheet references, and domain-appropriate logic. To capture the diversity of real-world spreadsheet workflows, the dataset covers a wide range of financial and accounting scenarios, including accounting tasks (e.g., consolidation, deferred tax, equity investments) and finance-related analyses such as discounted cash flow (DCF), leveraged buyout (LBO), mergers and acquisitions (M&A), and comparable company analysis. Task Construction. Based on the gold-standard spreadsheets, each task is created by transforming a complete workbook into a partially specified input spreadsheet paired with a natural-language instruction. Depending on the task category, this transformation involves selectively removing target regions, injecting controlled errors, or specifying visualization objectives, such that the resulting task requires the model to recover or produce the missing content. All tasks are manually designed and annotated by financial experts to reflect realistic business spreadsheet workflows. The construction process enforces several quality constraints: each task must admit a unique, deterministic solution given only the input spreadsheet and instruction; the instruction must be self-contained without requiring external knowledge beyond standard financial domain conventions; and the task must involve multi-step reasoning over complex spreadsheet structures rather than isolated cell-level edits. As a result, solving a single task typically requires coordinated operations across multiple sheets, including formula propagation, reference resolution, structural inference, and consistency verification, mirroring the cognitive demands of real-world spreadsheet work in professional settings. Data Validation. To ensure the overall quality of the benchmark, each task is independently reviewed by two experts who were not involved in its construction. Reviewers are provided only with the input spreadsheet and the natural-language instruction, and are asked to solve the task independently. Their solutions are then compared against the original ground-truth spreadsheet. If discrepancies arise, the task is iteratively revised through expert discussion and correction until consistent solutions are obtained. This cross-validation process ensures that task instructions are unambiguous, all required operations are well-defined, and the final solution is uniquely determined by the given spreadsheet and instruction. 2.3 Benchmark Statistics Data Statistics. SPREADSHEETBENCH 2 comprises 321 tasks across three categories: generation (financial modeling and template), debugging, and visualization. Table 1 summarizes the key statistics. The benchmark comprises an average of 11.8 worksheets and 593.5 modified cells per task, with Financial Modeling and Debugging emerging as the most challenging categories, averaging 1,164.5 and 656.6 modified cells per task, respectively. Solving a single task thus requires reasoning over 4 Table 1: Overview statistics of SPREADSHEETBENCH 2. Avg. Sheets and Max. Sheets correspond to the mean and maximum number of worksheets per task file, respectively. Avg. Words indicates the length of the input instructions, while Modif. Cells denotes the average number of cells requiring modification. Category#TasksAvg. SheetsMax. SheetsAvg. WordsModif. Cells Financial Modeling10015.343705.91164.5 Debugging10021.199387.0656.6 Template971.23139.965.7 Visualization241.78617.788.3 Overall32111.899429.0593.5 Table 2: Comparison of SPREADSHEETBENCH 2 with existing benchmarks, emphasizing real-world tasks, multi-sheet and cross-sheet reasoning, and high domain expertise. AspectSheetCopilot [8] InstructExcel [14] SpreadsheetBench [15]Ours RealismSyntheticSyntheticSemi-realReal-world + Expert-curated Total Tasks2214,850912321 Avg. Instr. Words27.99.885.7429.0 Multi-Sheet TasksRareLimitedLimitedExtensive Domain ExpertiseLowLowLowHigh Avg. Sheets/File âź1âź2âź1.411.8 Error Taxonomyââ Chart Tasksââ Real Workflowââ cross-worksheet dependencies and coordinating hundreds of cell-level modifications, a level of complexity characteristic of real-world professional spreadsheet use. Comparison with existing benchmarks. Table 2 compares SPREADSHEETBENCH 2 with repre- sentative spreadsheet benchmarks across multiple dimensions. First, SPREADSHEETBENCH 2 is grounded in real-world business data curated by domain experts, whereas SheetCopilot [8] and In- structExcel [14] rely on synthetic instructions and SpreadsheetBench [15] draws from semi-real forum posts. Second, our tasks exhibit substantially higher structural complexity: an average of 11.8 sheets per file compared to roughly 1â2 in prior work, with instructions averaging 429.0 words that specify multi-step business objectives rather than single atomic operations. Third, SPREADSHEETBENCH 2 introduces task dimensions absent from existing benchmarks, including a systematic error taxonomy for debugging, native spreadsheet chart generation for visualization, and end-to-end workflow-level evaluation that requires sustained cross-sheet reasoning and domain-specific knowledge. These properties make SPREADSHEETBENCH 2 a more realistic and challenging testbed for evaluating agent capabilities in professional spreadsheet environments. 2.4 Evaluation Metrics We adopt two evaluation protocols tailored to different output types. For Financial Modeling, Template, and Debugging tasks, we compare the agentâs output against the golden spreadsheet. For Visualization tasks, whose outputs are charts, we employ a VLM-based evaluation. Spreadsheet Execution Metrics. By comparing the input spreadsheet with the golden file, we identify the set of target cellsâthose whose values or formulas must be changed to complete the task. We report two complementary metrics that operate at different granularities. Modification is a cell-level metric: for each task, we compute the fraction of target cells whose computed values match the golden file; we then report the average across all tasks in a category. It captures how precisely the agent performs the required edits, independent of side effects elsewhere. Accuracy is a task-level metric: a task is scored 1 if every cell in the output spreadsheet matches the golden file (covering both target and non-target cells), and 0 otherwise; we report the fraction of tasks that achieve this full match. VLM-as-a-Judge. Rule-based verification is impractical for visualization tasks due to the diversity of chart types, layouts, and formatting. Instead, we design fine-grained rubrics for each task along 5 Table 3: Performance of open-source and closed-source models across four spreadsheet task cate- gories. Modif. reports the average fraction of target cells whose computed values match the golden file. Acc. reports the fraction of tasks where every cell in the output matches the golden file. For Visualization tasks, Acc. instead reports the average rubric pass rate (fraction of expert-designed rubric satisfied per task). The Overall column aggregates across all applicable tasks per metric. Model TemplateFinancial ModelingDebuggingVisualizationOverall Modif.Acc.Modif.Acc.Modif.Acc.Acc.Modif.Acc. Closed-source Models Claude Opus 4.6 91.9752.5889.6934.0050.3812.0062.5077.2034.89 GPT-5.2 83.8435.0587.0733.0039.078.0045.8369.8526.79 Gemini 3.1 Pro 86.0628.8785.2631.0041.867.0041.6770.9123.68 Open-source Models GLM-5 66.6017.5378.4422.0029.067.0037.5057.9517.14 Deepseek-V3.275.1925.7759.787.0028.7110.0033.3354.3515.58 Kimi K2.5 67.0818.5664.8615.0019.484.0041.6750.3114.64 Qwen3.5-397B-A17B 65.6217.5369.5910.0024.063.0025.0052.9611.22 MiniMax M2.550.147.2262.058.0016.764.0016.6742.917.17 two dimensions: Data Correctness (e.g., whether plotted values match the source data) and Format Compliance (e.g., chart type, axis labels, color schemes). All rubrics are constructed and validated by domain experts (Appendix B.3). We use GLM-4.6V [24] as the judge to produce a binary PASS/FAIL signal per criterion, and report the rubric pass rate as the task score. 3 Experiments 3.1 Experimental Settings LLMs. We evaluate eight large language models via API-based inference, including five open- source models (GLM-5.0 [25], MiniMax M2.5 3 , Kimi K2.5 [26], Qwen3.5-397B-A17B [27], and DeepSeek-V3.2 [28]) and three closed-source models (GPT-5.2 4 , Gemini 3.1 Pro 5 , and Claude Opus 4.6 6 ). All models are evaluated in thinking mode, with GPT-5.2 and Gemini 3.1 Pro configured to use high reasoning effort. In addition, we include several LLM-based spreadsheet tools (Kimi Sheet, GLM in Excel, Claude for Excel, and ChatGPT for Excel) for complementary comparison; as these products do not expose a unified API interface compatible with our evaluation pipeline, we evaluate them through manual execution on a subset of 30 representative examples. Agent Scaffold. Our agent scaffold is built on SWE-agent [29] and retains its iterative observeâ reasonâact workflow [30]. In the main setting, the agent interacts with spreadsheets through three tools:bash,view_xlsx, andsubmit. Thebashtool executes local commands for programmatic workbook operations,view_xlsxprovides read-only inspection of workbook structure and sheet contents (including formulas and evaluated values), andsubmitfinalizes the episode for scoring. We set the maximum number of interaction turns to 50 and keep the scaffold fixed across models by using the same prompt template and tool specifications (additional implementation details are provided in Appendix C). We further compare GLM-5 across different agent scaffolds, including our scaffold, Claude Code, Kilo Code, and Cline, to study the impact of scaffold design. 3 https://w.minimax.io/news/minimax-m25 4 https://openai.com/index/introducing-gpt-5-2/ 5 https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ 6 https://w.anthropic.com/news/claude-opus-4-6 6 Claude Opus 4.6 Gemini 3.1 Pro Claude for Excel GPT-5.2 Kimi K2.5 GLM-5 ChatGPT for Excel GLM in Excel Kimi Sheet Qwen3.5-397B-A17B MiniMax M2.5 Deepseek V3.2 0 20 40 60 Score (%) 66.1 28.8 55.1 19.8 54.2 15.4 54.4 14.3 30.9 13.2 45.4 11.1 63.0 9.8 45.2 9.8 39.8 4.3 40.0 4.3 30.1 3.2 39.5 2.1 Overall Modif. Overall Acc. Figure 3: Performance on a 30-example representative subset covering Financial Modeling, De- bugging, and Visualization tasks. We compare different models and four LLM-based spreadsheet products. The results show that LLM-based spreadsheet products do not outperform models under our agent scaffold, with Claude for Excel achieving the best performance among the spreadsheet products. 3.2 Main Results SPREADSHEETBENCH 2 remains challenging even for frontier models. As shown in Table 3, the best-performing model, Claude Opus 4.6, achieves only 34.89% overall accuracy, and six of the eight evaluated models fall below 25%. Closed-source models hold a clear advantage over their open- source counterparts: the top three closed-source models reach 23.68%â34.89% accuracy, substantially outperforming all open-source models (7.17%â17.14%). The gap is equally pronounced within individual categories: Claude Opus 4.6 achieves 89.69% Modification but only 34.00% Accuracy on Financial Modeling, and 50.38% Modification but only 12.00% Accuracy on Debugging. This suggests that while current LLMs can perform individual cell-level edits with moderate success, they struggle to maintain cross-cell consistency and end-to-end correctness across multi-step workflowsâ indicating that the bottleneck lies not in isolated operations but in coordinating coherent modifications over complex, interdependent spreadsheet structures. Performance varies significantly across task categories. Table 3 presents the accuracy scores of various models across different task categories. The results reveal a pronounced performance disparity among task types: visualization tasks exhibit the lowest difficulty, with Claude Opus 4.6 achieving the highest accuracy of 62.5%; generation tasks, specifically template and financial modeling, follow with moderate performance; whereas debugging tasks pose the greatest challenge, with even the best-performing model, Claude Opus 4.6, attaining only 12% accuracy. These findings indicate that while current large language models have developed strong capabilities in spreadsheet visualization and foundational structure generation, they still face significant performance bottlenecks in real-world business scenarios that demand complex logical reasoning, error diagnosis, and automated correction. LLM-based spreadsheet products do not show clear performance advantages. As shown in Figure 3, we compare four LLM-based spreadsheet productsâKimi Sheet, GLM in Excel, Claude for Excel, and ChatGPT for Excelâagainst foundation models on a challenging subset of 30 examples spanning Financial Modeling, Debugging, and Visualization (10 each). For a fair comparison, each product is tested by human operators given only the task instruction and input file. The results show that none of the spreadsheet products surpass the foundation models, with Claude for Excel achieving the highest accuracy among them at only 15.4%, suggesting that the workflow-level complexity of SPREADSHEETBENCH 2 poses significant challenges for current LLM-based spreadsheet products. 3.3 Analysis and Discussions Ten Error Types of Debugging Tasks. In Figure 4, we compare model performance across ten debug- ging error subcategories (see Table 6). Difficulty varies: Incorrect Index Match and Incorrect Sign are relatively tractable, while Errors is the most challenging, with even the best model barely exceeding 20%. This category involves diagnosing and fixing formulas that produce explicit spreadsheet error values (e.g., #REF!, #NUM!), which often stem from complex cross-sheet dependencies or invalid references, making them particularly difficult to localize and resolve. Models exhibit complementary strengths, with Claude Opus 4.6 performing best on Incorrect Sign and GPT-5.2 leading on Double 7 Double Counting Embedded Hardcodes ErrorsInconsistent Color Incorrect Average Cross-Sheet Refs Incorrect Index Match Incorrect Sign Rel vs Abs Refs Unit Mismatch Claude Opus 4.6 GPT-5.2 Gemini 3.1 Pro Deepseek V3.2 GLM-5 Qwen3.5-397B-A17B Kimi K2.5 MiniMax M2.5 40.339.518.448.527.343.757.686.656.545.0 56.631.322.361.937.744.148.320.332.431.9 37.624.011.039.417.930.644.939.922.720.9 28.831.715.930.021.18.253.442.130.922.1 16.334.416.120.021.525.849.936.029.332.5 32.612.78.78.333.132.933.335.812.630.6 20.216.512.410.011.522.838.517.829.611.6 15.712.215.95.710.410.939.414.730.810.3 Figure 4: Modification scores across 10 error subcategories (Appendix B.4) in Debugging tasks. Counting and Inconsistent Color, and no single model dominates across all subcategories. Notably, performance drops markedly on Errors, highlighting the persistent difficulty of diagnosing and fixing formula-level failures in spreadsheets. Interaction Turns and Effective Step Ratio. We compare Claude Opus 4.6 and MiniMax M2.5 in Figure 6(a). Claude Opus 4.6 consistently achieves higher modification scores and accuracy across all task categories than MiniMax M2.5. It also requires fewer average interaction steps across four task types, suggesting that increased interaction-step count does not necessarily improve performance in spreadsheet-based tasks. To explain this efficiency gap, we define an effective step as one executed without errors (e.g., tool-call failures, code execution errors, or bash failures). Figure 6(b) reveals that Claude Opus 4.6 maintains a substantially higher effective-step ratio, confirming that its advantage stems from more reliable per-step execution rather than a larger interaction budget: each step more consistently translates into meaningful task progress. Failure Taxonomy. To elucidate the underlying causes of agent failures, we conducted a systematic analysis of failure trajecto- ries, categorizing errors into six distinct types, as summarized in Table 7, where the detailed definitions of each type are pro- vided. Leveraging Claude Code for automated trajectory analysis of Claude Opus 4.6, we derived the error distribution illustrated in Figure 5. Our results indicate that the predominant error cat- egories are Insufficient Inspection and Wrong Target Selection. This suggests that agents struggle to adequately comprehend the intricate interdependencies within spreadsheetsâsuch as formu- laic, numerical, and domain-specific logical relationshipsâand fail to accurately pinpoint the target cells requiring modification. Notably, these challenges reflect common pain points experienced by humans in real-world spreadsheet workflows. Figure 5: Failure Taxonomy dis- tribution for Claude Opus 4.6 tra- jectories of unresolved tasks. Different Agent Scaffold. Using GLM-5 as the fixed backbone, we compare our SWE-agent-based scaffold against three coding agent scaffolds on a subset of 50 samples (Table 4). Our scaffold achieves the highest modification score (46.32%) and accuracy (15.45%), with Claude Code as the closest competitor (43.47% / 14.20%). While modification scores remain within a narrow 4- point band, accuracy diverges sharply: Cline and Kilo Code attain only 8.66%, suggesting that general-purpose coding scaffolds can produce plausible edits but struggle with end-to-end correctness on spreadsheet tasks. Table 4: Overall performance comparison across different agent scaffolds. Agent ScaffoldModif.Acc. Our Scaffold46.3215.45 Claude Code43.4714.20 Cline41.988.66 Kilo43.128.66 Tool Usage Patterns. To understand the behavioral patterns of agents in spreadsheet tasks, we categorize tool usage trajectories into three functional phases (Figure 6(c)): Inspect (reading contents and examining formulas), Implement (writing cells and executing code), and Verify (re-reading modified cells to confirm correctness). Claude Opus 4.6 exhibits a higher inspection proportion than 8 MiniMax M2.5Claude Opus 4.6 0 10 20 30 Average steps 12.5 8.6 29.8 21.7 33.7 24.7 30.0 12.2 Temp. Fin. Dbg. Viz. MiniMax M2.5Claude Opus 4.6 (a) MiniMax M2.5Claude Opus 4.6 0.4 0.5 0.6 0.7 0.8 Effective step ratio 0.58 0.72 0.46 0.53 0.45 0.59 0.51 0.75 Temp. Fin. Dbg. Viz. MiniMax M2.5Claude Opus 4.6 (b) Temp.Fin.Dbg.Viz. 0 25 50 75 100 Percent (%) Claude Opus 4.6MiniMax M2.5Claude Opus 4.6MiniMax M2.5 InspectImplementVerify (c) Figure 6: Overview of trajectory-level behavior across four task domains. (a) Average number of steps per task for each model. (b) Effective-step ratio, measuring the fraction of steps that directly contribute to task progress. (c) Process distribution, showing the proportion of inspect, implement, and verify phases within each domain. MiniMax M2.5 in Template, Visualization, and Financial Modeling tasks, but this trend reverses in Debugging tasks, where both models allocate over 60% of their trajectory to inspection. We attribute this to the absence of explicit step-by-step instructions in Debugging tasks, which necessitates extensive sheet exploration and imposes significant demands on the agentâs ability to localize information within complex spreadsheet environments. Case Study. We highlight a representative Financial Modeling failure (details in Appendix D.4, Figure 15). The task requires filling six groups of empty cells across a multi-sheet banking model. Claude Opus 4.6 derived formulas from general financial knowledge (e.g., âBook Value per Share = Total Equity / Shares Outstandingâ) rather than reverse-engineering the conventions already established in the workbook; the formulas are textbook-correct but the cell references did not match the actual row layout, yielding only 4.1% modification accuracy. This reveals that current agents can retrieve relevant domain knowledge but lack the ability to ground it in a specific workbookâs dependency structure, a capability gap that future spreadsheet agents must address. Five additional cases spanning Debugging and Visualization are provided in Appendix D.4. 4 Related Work Spreadsheet and Table Benchmarks. A substantial body of work has developed benchmarks for table understanding, spanning question answering over structured tables, fact verification against tabular evidence, and numerical reasoning over hybrid tabular-textual data [31,32,33,34,35,36,37, 20]. On the spreadsheet side, prior efforts address two complementary directions: table encoding and representation learning for spreadsheet structures [38,39], and benchmarking spreadsheet manipulation from natural language instructions, including formula generation, cell editing, and task- level completion [8,14,40,15]. However, these benchmarks predominantly target isolated atomic operations, such as generating a single formula or editing specific cells, rather than the multi-step, cross-sheet workflows that characterize real business spreadsheet usage. SPREADSHEETBENCH 2 addresses this gap with end-to-end workflow-level tasks spanning generation, debugging, and visualization, averaging 11.8 sheets and 593.5 modified cells per instance. LLM-based Spreadsheet Agents. Recent advances in LLM-based agents have demonstrated the potential of autonomous tool use for complex tasks, with agent frameworks and benchmarks emerging across coding, web navigation, and desktop environments [30,41,42,43,44,45]. In the spreadsheet domain, several agent systems have been proposed to automate spreadsheet operations, adopting diverse strategies such as translating natural language into API calls, multi-module reasoning pipelines, neuro-symbolic computation, and multi-agent collaboration [8,9,10,11]. Despite this progress, existing spreadsheet agents are evaluated only on isolated operations, leaving a gap in assessing performance on realistic business workflows that require sustained cross-sheet reasoning and coordinated tool use. SPREADSHEETBENCH 2 provides a workflow-level evaluation testbed that requires agents to perform multi-step, cross-sheet reasoning and produce structurally coherent spreadsheet artifacts, enabling systematic comparison of agent capabilities on realistic business tasks. 9 5 Conclusion We present SPREADSHEETBENCH 2, a benchmark of 321 expert-annotated tasks evaluating spread- sheet agents on workflow-level generation, debugging, and visualization over multi-sheet workbooks with rich cross-sheet dependencies. The best model achieves only 34.89% accuracy (debugging: 12%), and stronger models benefit from more reliable per-step execution rather than larger interaction budgets, highlighting structured reasoning over interdependent workbooks as a key bottleneck. References [1]Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, and Xiaoyong Du. Large language model for table processing: A survey. Frontiers of Computer Science, 19(2):192350, 2025. [2]Mingyue Cheng, Qi Liu, Qingyang Mao, Yitong Zhou, Yupeng Li, Jiahao Wang, Jiaying Lin, Jiawei Cao, and Enhong Chen. A survey on table mining with large language models: Challenges, advancements and prospects. 2025. [3]Karl W Broman and Kara H Woo. Data organization in spreadsheets. The American Statistician, 72(1):2â10, 2018. [4] David McD Taylor, Peter W Hodkinson, Abdus Salam Khan, and Erin L Simon. Research skills and the data spreadsheet: A research primer for low-and middle-income countries. African Journal of Emergency Medicine, 10:S140âS144, 2020. [5]Abdullah Al Maruf, Rajesh Paul, Mohammad Hasan Imam, and Zahir Babar. A systematic review of the role of sql and excel in data-driven business decision-making for aspiring analysts. American Journal of Scholarly Research and Innovation, 1(01):249â269, 2022. [6]TG Grossman, Vijay Mehrotra, and ĂzgĂźr ĂzlĂźk. Spreadsheet information systems are essential to business. University of San Francisco working paper, 2005. [7]David A Bradbard, Charles Alvis, and Richard Morris. Spreadsheet usage by management accountants: An exploratory study. Journal of Accounting Education, 32(4):24â30, 2014. [8]Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhao-Xiang Zhang. Sheetcopilot: Bringing software productivity to the next level through large language models. Advances in Neural Information Processing Systems, 36:4952â4984, 2023. [9] Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, Jianye Hao, Hangyu Mao, and Fuzheng Zhang. Sheetagent: towards a generalist agent for spreadsheet reasoning and manipulation via large language models. In Proceedings of the ACM on Web Conference 2025, pages 158â177, 2025. [10] Ziwei Wang, Jiayuan Su, Mengyu Zhou, Huaxing Zeng, Mengni Jia, Xiao Lv, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang. Sheetbrain: A neuro-symbolic agent for accurate reasoning over complex and large spreadsheets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33800â33808, 2026. [11]Ruiyan Zhu, Xi Cheng, Ke Liu, Brian Zhu, Daniel Jin, Neeraj Parihar, Zhoutian Xu, and Oliver Gao. Sheetmind: An end-to-end llm-powered multi-agent framework for spreadsheet automation. arXiv preprint arXiv:2506.12339, 2025. [12]Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fain- man, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table fine-tuned gpt for diverse table tasks. Proceedings of the ACM on Management of Data, 2(3):1â28, 2024. [13]Harshit Joshi, Abishai Ebenezer, JosĂŠ Cambronero Sanchez, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radi Ë cek, and Gust Verbruggen. Flame: A small language model for spreadsheet formulas. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12995â13003, 2024. 10 [14]Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, and Elnaz Nouri. Instructexcel: A benchmark for natural language instruction in excel. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4026â4043, 2023. [15] Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. Spreadsheetbench: Towards challenging real world spreadsheet manipulation. Advances in Neural Information Processing Systems, 37:94871â94908, 2024. [16] Simon Thorne. Large language models for spreadsheets: Benchmarking progress and evaluating performance with flare. arXiv preprint arXiv:2506.17330, 2025. [17] Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821â1831, 2017. [18]Jonathan Herzig, Pawel Krzysztof Nowak, Thomas MĂźller, Francesco Piccinno, and Julian Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4320â4333, 2020. [19]Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. Tapex: Table pre-training via learning a neural sql executor. arXiv preprint arXiv:2107.07653, 2021. [20]Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xeron Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25497â25506, 2025. [21]Zheng Li, Yang Du, Mao Zheng, and Mingyang Song. Mimotable: A multi-scale spreadsheet benchmark with meta operations for table reasoning. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2548â2560, 2025. [22]Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, and HV Jagadish. Mmtu: A massive multi-task table understanding and reasoning benchmark. arXiv preprint arXiv:2506.05587, 2025. [23]Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330â42357, 2023. [24]Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025. [25] Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. [26] Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. [27]Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. [28]Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. 11 [29]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528â50652, 2024. [30] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. [31] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470â1480, 2015. [32]Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164, 2019. [33]Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026â1036, 2020. [34]Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3277â3287, 2021. [35]Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697â3711, 2021. [36]Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 6279â6292, 2022. [37]Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6588â6600, 2022. [38]Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 8413â8426, 2020. [39]Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Shiyu Xia, Mengyu Zhou, Yun Lin, JosĂŠ Cambronero, Yeye He, Shi Han, et al. Spreadsheetllm: Encoding spreadsheets for large language models. arXiv preprint arXiv:2407.09025, 2024. [40]Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui, and Haidong Zhang. Nl2formula: Generating spreadsheet formulas from natural language queries. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2377â2388, 2024. [41]Timo Schick, Jane Dwivedi-Yu, Roberto DessĂŹ, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539â 68551, 2023. [42]Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024. 12 [43]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. [44]Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. [45]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023. 13 A Broader Discussion A.1 Limitations Although SPREADSHEETBENCH 2 advances spreadsheet agent evaluation, several limitations remain. (1) The benchmark focuses on financial and business spreadsheet workflows, and the findings may not generalize to other spreadsheet-intensive domains such as scientific computing or engineering. (2) Due to the high cost of API-based LLM inference, we do not report error bars or confidence intervals for the experimental results. (3) For visualization tasks, the VLM-as-a-judge framework may introduce evaluation noise despite expert-designed rubrics; for cell-based tasks, exact-match evaluation may undercount semantically equivalent but syntactically different solutions. A.2 Ethical Considerations In this section, we discuss the ethical considerations regarding our data construction process. (1) Data Risk Control: During the benchmark construction process, we review all spreadsheet files and task instructions to ensure they do not contain content that is inappropriate for a general audience, such as violence, discrimination, or sensitive subjects. In addition, two authors conduct an independent verification to confirm that the benchmark does not contain any personally identifiable information. All financial data is derived from publicly available corporate filings and financial reports. (2) Copyright: Our benchmark is constructed from publicly available financial reports and corporate filings. To prevent copyright infringement, we do not directly redistribute any raw source documents. Instead, the released benchmark consists of expert-constructed spreadsheet tasks derived from these public materials, with numerical values and structural layouts modified where necessary. Considering potential licensing and copyright risks, we release the benchmark under a C BY-SA 4.0 license and refrain from secondary distribution of any raw source data. A.3 Broader Impacts On the positive side, advancing spreadsheet agent capabilities can reduce manual effort in repetitive business workflows, democratize access to financial modeling, and help detect costly spreadsheet errors that are a well-documented source of financial losses. On the negative side, over-reliance on AI-based spreadsheet agents without adequate human oversight could lead to undetected errors in financial models, and automation of financial tasks may raise concerns about workforce displacement. Our benchmark explicitly quantifies the gap between current model capabilities and real-world requirements, with the best model achieving only 34.89% overall accuracy, discouraging premature deployment and encouraging further research on reliability. A.4 License The SPREADSHEETBENCH 2 dataset is released under the C BY-SA 4.0 license, which permits sharing and adaptation, including commercial use, with appropriate credit and under the same license. The evaluation code is released under the MIT License. All third-party assets used in this work are properly cited and their respective licenses are respected. B Details of Data B.1 Data Sources The financial data underlying SPREADSHEETBENCH 2 is drawn from a diverse set of publicly avail- able sources. These include primary regulatory filings from Bseindia 7 , Bloomberg 8 , Screener.in 9 , and the NYU Stern (Damodaran) dataset 10 . For sources with redistribution restrictions, we release only derived task metadata and evaluation artifacts when permitted, while separately documenting access requirements and licensing constraints. Together, these sources provide broad coverage of 7 https://w.bseindia.com/corporates/ann 8 https://w.bloomberg.com/ 9 https://w.screener.in/ 10 https://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html 14 real-world business data across multiple geographies and industry sectors, ensuring the diversity and authenticity of the benchmark tasks. All source materials are publicly available at the time of collection, and no proprietary or restricted-access data is used in the construction of the benchmark. B.2 Annotation Effort The construction process required substantial expert time. Across the four task categories, the total annotation effort exceeded 1,500 hours: Financial Modeling tasks required around 400 hours, Debugging tasks around 500 hours, Visualization tasks around 380 hours, and Template tasks around 200 hours. These figures do not include additional quality assurance rounds, such as having selected tasks performed by human solvers to verify solvability, conducted both internally and by our data provider partners. B.3 Visualization Task Rubric Examples We evaluate visualization tasks using a VLM-as-a-judge approach with hand-crafted rubrics that assess two complementary aspects: Format Compliance (chart type, axis configuration, color scheme) and Data Correctness (value ranges, data labels). Table 5 shows the complete rubric for a football field valuation chart (Task 129), illustrating the granularity of our per-criterion evaluation. Table 5: Rubric example for Task 129: Football Field Valuation Chart. Each criterion is independently verified by the VLM judge. #CriterionCategory 1The chart is a horizontal bar chart (football field valuation chart) showing valuation ranges Format Compliance 2The chart title reads âProject Mars â Standalone Valuation SummaryâFormat Compliance 3 The methodologies appear top to bottom in this order: DCF (Based on Management Forecast), Precedent Transactions, Mgmt Projections 2026E, Mgmt Projections 2027E, Analyst Target Price Range, 52-Week Trading Range Format Compliance 4The X-axis starts at $15.00Format Compliance 5The X-axis ends at $65.00Format Compliance 6The X-axis displays gridlines at $5.00 intervalsFormat Compliance 7The Analyst Target Price Range bar is displayed in light blueFormat Compliance 8The 52-Week Trading Range bar is displayed in light blueFormat Compliance 9The DCF (Based on Management Forecast) bar is displayed in dark blueFormat Compliance 10The Precedent Transactions bar is displayed in dark blueFormat Compliance 11The Mgmt Projections 2026E bar is displayed in dark blueFormat Compliance 12The Mgmt Projections 2027E bar is displayed in dark blueFormat Compliance 13The DCF bar starts at 41.89 and ends at 51.56 on the X-axisData Correctness 14The data labels on the DCF bar read â41.89â and â51.56âData Correctness 15The Precedent Transactions bar starts at 38.53 and ends at 46.93Data Correctness 16The data labels on the Precedent Transactions bar read â38.53â and â46.93â Data Correctness 17The Mgmt Projections 2026E bar starts at 20.9 and ends at 37.99Data Correctness 18 The data labels on the Mgmt Projections 2026E bar read â20.9â and â37.99â Data Correctness 19The Mgmt Projections 2027E bar starts at 36.25 and ends at 58.69Data Correctness 20The data labels on the Mgmt Projections 2027E bar read â36.25â and â58.69â Data Correctness 21The Analyst Target Price Range bar starts at 33.72 and ends at 46.48Data Correctness 22 The data labels on the Analyst Target Price Range bar read â33.72â and â46.48â Data Correctness 23The 52-Week Trading Range bar starts at 38.65 and ends at 44.4Data Correctness 24 The data labels on the 52-Week Trading Range bar read â38.65â and â44.4â Data Correctness 15 B.4 Debugging Task Types Table 6 presents the taxonomy of spreadsheet error types used in the Debugging task. These ten categories were identified through iterative annotation of real-world spreadsheet errors encountered in financial modeling and accounting practice. The taxonomy spans a range of failure modes, from low-level syntactic issues such as formula errors and incorrect sign assignments, to higher-level structural problems including cross-sheet reference failures and unit mismatches. Each debugging instance contains one or more injected errors drawn from this taxonomy, and the agent is required to localize and correct all errors to restore the spreadsheet to its intended state. Table 6: Taxonomy of spreadsheet error types in the Debugging task. Error TypeDefinition Double CountingA value is aggregated multiple times within a computation, resulting in systematic overestimation. Embedded HardcodesConstant numeric values are hardcoded into formulas instead of being refer- enced from input cells, limiting correctness and adaptability. ErrorsFormulas that result in error messages such as #REF!, #NUM!, or similar, indicating invalid references or computations leading to unintended outputs. Inconsistent ColorFormatting rules are inconsistently applied across semantically equivalent cells, reducing interpretability. Incorrect AverageAverages are computed over incorrect ranges, leading to biased or misrepre- sented summary statistics. Cross-Sheet ReferencesInter-sheet references are missing, broken, or incorrectly specified, causing dependency failures in multi-sheet models. Incorrect index-matchLookup functions are misconfigured due to incorrect range alignment, match settings, or indexing logic. Incorrect SignNumerical signs are incorrectly assigned, reversing the intended meaning of computed values. Relative vs. Absolute ReferencesImproper use of relative and absolute cell references causes formulas to break when copied or propagated. Unit MismatchInconsistent or incompatible units are combined within computations, result- ing in scale or magnitude errors. C Details of Experiments C.1 Environment. Each task instance is executed inside a sandboxed Docker container built on Python 3.11 (Debian Bullseye). The container is pre-installed with common data-science libraries (NumPy, Pandas, Matplotlib, openpyxl) and LibreOffice with its Python UNO bridge, enabling both programmatic and application-level spreadsheet manipulation. No network access is provided during execution, ensuring that the agent must rely solely on its own reasoning and the provided tools. C.2 Tools. We conduct our experiments within the SWE-Agent framework. To rigorously evaluate the intrinsic capabilities of the model, we restrict the available toolset to three fundamental utilities: bash, view_xlsx, and submit. â˘bash: Executes arbitrary shell commands, providing the agent with general-purpose programmatic access to the operating system. Common usage includes file operations (ls,cat,cp,mv), running Python scripts viapython3for data manipulation and formula computation, and invoking standard Unix utilities for text processing (grep, awk, sed). ⢠view_xlsx: A read-only utility for inspecting.xlsxspreadsheet files. The agent must specify one of two modes:list, which enumerates all sheet names in the workbook, orcontent, which 16 displays the contents of a specific sheet. Thecontentmode optionally accepts a row range to limit the output (e.g., rows 1â50), and exposes both the original formulas and their evaluated values. â˘submit: Finalizes the agentâs solution and submits the modified spreadsheet for evaluation. The agent is expected to invoke this tool only after all edits have been completed and verified. C.3 Prompt. The agent receives a two-part prompt at each episode. The system prompt (Figure 7) establishes the agentâs role and interaction protocol, while the instance prompt supplies the task-specific context, including available tools, a recommended workflow, and the concrete instructions with input/output file paths. We use two instance prompt variants: one for Template, Financial Modeling, and Debug- ging tasks (Figure 8), and a separate one for Visualization tasks (Figure 9) that additionally specifies the visualization-oriented objective and restricts the library set. Template variables (shown in angle brackets) are populated per task instance. System Prompt You are a helpful Spreadsheet Automation Engineer that interacts with a computer shell to solve data tasks. You operate in a REPL (Read-Eval-Print Loop) environment where you must issue exactly ONE command at a time. Figure 7: System prompt used for all task instances. Instance Prompt Important â˘When completing spreadsheet tasks, strictly avoid altering any cells that already contain values unless explicitly instructed. Modify only the cells that are required for the task. â˘You need to complete the instructions and ensure that the original formatting is preserved as much as possible. Tools You are provided with three tools:bash,view_xlsxandsubmit. You must use these tools to complete the target task. You can only call ONE tool at a time per response. Environment You can complete the corresponding spreadsheet tasks using the following Python environment: Pandas, openpyxl, NumPy, and LibreOffice. Recommended Workflow 1. Inspect: Use view_xlsx to understand the structure and contents of the spreadsheet. 2. Plan: Decide how to manipulate the data using openpyxl or LibreOffice. 3. Implement: Create a Python script that performs the edit. 4. Execute: Run the script via python3. 5.Verify: Check the output file exists and ensure correctness. Do not combine verify and submit commands. 6. Submit: When verification is successful, run submit. Task Instructions You need to process a spreadsheet file based on specific instructions. Instruction: â¨instructionâŠInput File: â¨spreadsheet_pathâŠOutput Path: â¨output_path⊠Figure 8: Instance prompt template for Template, Financial Modeling, and Debugging tasks. Instance Prompt (Visualization) Important â˘When completing spreadsheet tasks, strictly avoid altering any cells that already contain values unless explicitly instructed. Modify only the cells that are required for the task. â˘You need to complete the instructions and ensure that the original formatting is preserved as much as possible. Tools You are provided with three tools:bash,view_xlsxandsubmit. You must use these tools to complete the target task. 17 Environment You can complete spreadsheet visualization tasks using the following Python environment: openpyxl and xlsxwriter. Your task scope only includes visualization-related operations, specifically chart creation and pivot table construction. Task Objective Your goal is to create clear, professional, and interpretable visualizations from the given spreadsheet. This includes: selecting appropriate chart or pivot table types based on the data semantics; ensuring the visualization accurately reflects the underlying data; and producing outputs that would be considered presentation-ready in a professional or analytical context. Recommended Workflow 1. Inspect: Use view_xlsx to understand the structure and contents of the spreadsheet. 2.Plan: Determine whether the task requires a chart or a pivot table, choose the visualization type, map data fields to visual elements, and consider design choices that improve clarity. 3. Implement: Create a Python script that performs the visualization. 4. Execute: Run the script via python3. 5. Verify: Check the output file exists and ensure correctness. Do not combine verify and submit commands. 6. Submit: When verification is successful, run submit. Task Instructions You need to process a spreadsheet file based on specific instructions. Instruction: â¨instructionâŠInput File: â¨spreadsheet_pathâŠOutput Path: â¨output_path⊠Figure 9: Instance prompt template for Visualization tasks. Encountered error Fixed 1 error Passed end-to-end 0 20 40 60 80 100 Percentage of error-hit tasks (%) 38% 72% 31% 7% Error rate: Opus 16.5%, MiniMax 61.9% 100% Template Encountered error Fixed 1 error Passed end-to-end 50% 47% 45% 9% Error rate: Opus 67.4%, MiniMax 88.0% 100% Financial Model Encountered error Fixed 1 error Passed end-to-end 37% 34% 6% 2% Error rate: Opus 62.0%, MiniMax 80.0% 100% Debugging Claude Opus 4.6MiniMax M2.5 Figure 10: Error-recovery analysis on Template, Financial Model, and Debugging tasks. For each task where the agent encounters at least one error, we track three stages: whether an error occurred (baseline), whether the agent fixed at least one error (per-error fix rate), and whether the task still passed end-to-end evaluation (end-to-end pass rate). MiniMax can fix individual errors but rarely produces a correct final output for error-affected tasks. D Additional Experimental Analysis D.1 Error-Recovery Analysis When an agent makes a mistake during execution (e.g., a runtime exception or a wrong intermediate result), can it fix the mistake and still finish the task correctly? Figure 10 answers this question by tracking three stages for every task that contains at least one error. The x-axis shows: 1. Encountered error: the starting point (100%), representing all tasks where the agent hit at least one error. 2. FixedâĽ1 error: of those tasks, how many did the agent manage to fix at least one error in. 3. Passed end-to-end: of those tasks, how many still produced a fully correct final output. 18 The key finding is the drop from stage 2 to stage 3. MiniMax M2.5 can fix individual errors reasonably often (72% on Template, 47% on Financial Model, 34% on Debugging), but its final pass rate collapses (to 7%, 9%, and 3%). In other words, MiniMax patches the immediate problem, but the rest of its solution is already too far off track to recover. Opus 4.6 shows a much smaller dropâfor instance, 50%â45% on Financial Modelâmeaning that when Opus fixes an error, it usually still gets the final answer right. Each panel also notes the error rate: the share of all tasks in that category where at least one error occurred. MiniMax hits errors in 62â88% of tasks, compared to 16â67% for Opus, which means MiniMax not only recovers worse but also needs to recover far more often. D.2 Fine-Grained Evaluation of Visualization Tasks As described in Section 2.4, Visualization tasks are scored using a vision-language model (VLM) against a hand-crafted rubric. Each rubric criterion falls into one of two dimensions: ⢠Data Correctness: whether the generated chart accurately reflects the underlying spreadsheet data (e.g., correct data series, values, and axis ranges). â˘Format Compliance: whether the chart meets formatting specifications such as chart type, axis labels, legend placement, color assignments, and layout. Figure 11 reports per-model scores on these two dimensions. A consistent pattern emerges: every model scores higher on Format Compliance than on Data Correctness. This means current models are relatively good at following explicit style instructions (e.g., âuse a bar chart with blue barsâ) but weaker at understanding which data to extract from complex spreadsheets and how to map it onto the chart. The gap between the two dimensions is smallest for Claude Opus 4.6 (82.0 vs. 72.7) and largest for MiniMax M2.5 (64.7 vs. 31.4), showing that stronger models pull ahead primarily by improving data handling, not formatting. Claude Opus 4.6 achieves the highest Overall score (76.2), followed by GPT-5.2 (70.4) and Gem- ini 3.1 Pro (69.0). Notably, Gemini 3.1 Pro has the highest Format Compliance (83.3) of any model, but its lower Data Correctness (60.7) limits its Overall ranking. At the other end, MiniMax M2.5 trails on both dimensions, with a Data Correctness of only 31.4âless than half that of the top modelâconfirming that extracting and correctly plotting data remains the primary bottleneck for weaker models. D.3 Failure Taxonomy To systematically understand why agents fail, we manually inspected a sample of unsuccessful trajectories from Claude Opus 4.6 and developed a taxonomy of six recurring failure modes, sum- marized in Table 7. The categories are designed to be mutually exclusive at the primary-cause level: each failed trajectory is assigned to the single category that best describes its root cause. Task Misunderstanding and Insufficient Inspection capture failures in the comprehension stage, where the agent either misinterprets the goal or skips critical context in the spreadsheet. Wrong Target Selection and Format/Output Error capture execution-stage failures, where the agent understands the task but applies changes to incorrect locations or produces malformed output. Turn Limit Exceeded reflects planning inefficiency, and Other subsumes rare or ambiguous cases that do not fit the preceding categories. As discussed in Section 3, the dominant failure modes are Insufficient Inspection and Wrong Target Selection, indicating that accurately locating and understanding target cells within complex spreadsheet structures remains a core challenge. D.4 Failure Cases To complement the quantitative failure taxonomy in Table 7, we present six representative failure cases drawn from Claude Opus 4.6âs failed trajectoriesâtwo per task category (Debugging, Financial Modeling, Visualization). These cases illustrate common reasoning breakdowns that lead to partial or complete task failure. 19 020406080 Score (%) MiniMax M2.5 Qwen3.5-397B-A17B Deepseek V3.2 Kimi K2.5 GLM-5 Gemini 3.1 Pro GPT-5.2 Claude Opus 4.6 31.4 47.2 51.4 50.4 55.5 60.7 65.6 72.7 64.7 67.9 69.8 76.2 80.9 83.3 78.6 82.0 44.0 55.0 58.2 60.1 65.1 69.0 70.4 76.2 Data Correctness Format Compliance Overall Figure 11: Rubric-based evaluation scores for Visualization tasks across two dimensions: Data Correctness (whether the chart accurately reflects the spreadsheet data) and Format Compliance (adherence to chart type, axis labels, colors, and layout specifications). The Overall score is a weighted combination of both. Models are sorted by Overall score. Table 7: Descriptions of failure mode categories. CategoryDefinition Task MisunderstandingThe agent misinterprets the userâs instructions and pursues an entirely incor- rect objective. Insufficient InspectionThe agent fails to sufficiently explore the spreadsheet prior to making edits, thereby missing critical context. Wrong Target SelectionThe agent correctly identifies the required action but applies it to the incorrect cells or worksheets. Turn Limit ExceededThe agent fails to complete the task before exceeding the maximum permitted number of dialogue turns. Format/Output ErrorThe agent derives the correct values but applies an incorrect format, or otherwise fails to generate a valid output file. OtherThere was some other problem that prevented the agent from resolving this issue. D.4.1 Debugging Cases Case 1. As shown in Figure 12, the agent correctly identified all three double-counting bugs through cross-sheet formula tracing, yet only fixed two of them. The unfixed bugâtwo balance sheet line items referencing identical source cellsâwas structurally unambiguous, but the agent grouped it with genuinely ambiguous patterns and chose not to act on it. This shows that correct diagnosis does not guarantee complete repair: the agent applied an overly conservative filter that treated a clear structural error as uncertain, leaving a known bug unresolved at submission time. Case 2. As shown in Figure 13, the agent correctly recognized that the task objective was to fix color-coding inconsistencies, but gradually drifted into a general spreadsheet audit. It applied formula corrections that were never requested while skipping the required color-only fixes. Worse, for cell C90, the agent changed the cell value fromâ0.3to0.3instead of merely recoloring the fontâconfusing a formatting inconsistency with a data error and introducing a destructive modification. This case illustrates a failure to stay within the taskâs specified scope: the agent expanded its edits beyond what the instruction asked for, and the out-of-scope changes were not only unnecessary but harmful. 20 Case 1: Double Counting (Debugging) Task Description Audit and fix a 13-sheet LBO/DCF model for Monro, Inc. (NASDAQ: MNRO). The task requires finding and fixing double-counting bugsâcomponents counted multiple times in aggregate formulas across interconnected sheets (Control, Summary, Assumptions, Income Statement, Balance Sheet, Cash Flow Statement, etc.). Agent Reasoning Over 17 inspection steps, the agent correctly identified three bugs: Bug A (Balance Sheet Row 46): Total Liabilities double-counts Current Portion of LT Debt. Bug B (Cash Flow Row 39): Net Cash Flow double-counts Net Income. Bug C (Balance Sheet Rows 31 & 39): Both âCurr. Port. Of Cap. Leasesâ and âCapital Leasesâ reference identical cells from the P&E sheet, causing the same balance to appear in both current and long-term liabilities. Key Decision Point (Step 18: fix script creation) The agent fixed Bugs A and B correctly but explicitly deprioritized Bug C: âIâl focus on fixing the clear double-counting issues first and leave these averaging patterns as-is unless theyâre definitively wrong.â This conflated an unambiguous bugâtwo distinct balance sheet line items pulling from the exact same cellâwith genuinely ambiguous patterns (averaging inconsistencies, rolling windows). Outcome Modification accuracy:0.8974 Unfixed Bug C propagation:Cell H48 â gold: 1094.14, agent: 1259.34 Failure mode:Insufficient Inspection Figure 12: Insufficient inspection case of Debugging task. Case 2: Inconsistent Color Coding (Debugging) Task Description Fix font color inconsistencies in a 13-sheet Atlassian DCF model. Convention: green = cross-sheet links, blue = hardcoded inputs, black = formulas/calculations. Identify cells where font color does not match content type and correct the color coding. Agent Reasoning The agent spent 17 steps inspecting sheets and 4 steps analyzing colors via Python scripts. It correctly noted the color convention but then overrode its own understanding: âI should focus on fixing those issues. . . Iâl be conservative and address only the clear violations.â The agent applied 13 fixesâmostly formula/logic corrections that were not requiredâwhile systemati- cally skipping the actual color-only fixes. Key Decision Point (Steps 18â20: analysis phase) The cells the agent decided were ânot clear violationsâ (D41/D47/D53, C62:H62, C90:H90) are precisely the cells the gold answer requires. Worse, for cell C90, the agent changed the value fromâ0.3to0.3 instead of merely recoloring the font to blueâa destructive error. Outcome Modification accuracy:0.0 Destructive change:C90 value altered (â0.3â 0.3) instead of font color change Failure mode:Task Misunderstanding Figure 13: Task misunderstanding case of Debugging task. D.4.2 Financial Modeling Cases Case 3. As shown in Figure 14, the agent constructed the Debt/Equity ratio formula by copying the numerator pattern from an adjacent Debt/Total Capitalization formula, without checking whether 21 the two ratios use the same definition of âDebt.â The adjacent formula sums both long-term and short-term borrowings, but Debt/Equity conventionally uses only long-term borrowings. The agent provided detailed reasoning for more complex subtasks (e.g., WACC calculation) but treated this formula as a trivial copy operation with no explicit reasoning trace. This shows that the agent relied on surface-level pattern matching from a neighboring cell rather than verifying the formula against the financial definitions used elsewhere in the workbook. Case 4. As shown in Figure 15, the agent generated all formulas from abstract financial knowledge (e.g., âBook Value per Share = Total Equity / Shares Outstandingâ) without verifying whether the referenced rows and columns match the workbookâs actual layout. As a result, nearly every formula was conceptually reasonable but structurally wrong for this particular banking model, yielding only âź4% cell accuracy. Additionally, by filling previously-empty cells that other formulas depended on being zero, the agent broke pre-existing formulas and caused cascading errors. This case represents a more severe version of the pattern in Case 3: instead of copying from a nearby cell, the agent bypassed the workbook entirely and relied on general domain knowledge to construct formulas, without grounding them in the spreadsheetâs actual structure. Case 3: Debt/Equity Ratio (Financial Modeling) Task Description Complete a Procter & Gamble financial model (9 sheets). Among 8 subtasks: calculate Debt/Equity ratio for all periods in the Ratio_Analysis sheet (Row 25). Agent Reasoning The agent observed an adjacent formula for Debt/Total Capitalization (Row 26): =(F95+F90)/(F87+F95+F90)(sums long-term + short-term borrowings) It copied this numerator pattern directly into the Debt/Equity formula without any explicit reasoningâin contrast to the detailed reasoning it articulated for WACC. Key Decision Point (Step 12: script creation) Agent formula: =(F95+F90)/F103vs. Gold formula: =F90/F103 The agent included short-term borrowings (Row 95) in the numerator by analogy with the adjacent ratio, but Debt/Equity conventionally uses only long-term borrowings. Outcome Modification accuracy:0.8824 Output vs. gold:Cell L37: 0.4591 vs. 0.2265 (exactlyâź2Ă, confirming doubled numerator) Failure mode:Wrong Target Selection Figure 14: Wrong target selection case of Financial Modeling task. 22 Case 4: Banking Model (Financial Modeling) Task Description Complete a multi-sheet banking model (âProject Bankingâ) by filling six groups of empty cells across Balance Sheet (Check row, Book Value per Share, Shareholdersâ Equity w/o dividends), Macro sheet (Deposits-to-GDP, Competition 6 Market Share), and Assumptions-BS (Total Funding Growth). Agent Reasoning The agent derived formulas from general financial knowledge rather than examining how similar rows in the same workbook compute their values: âCheck row: Total AssetsâTotal Liabilities & Ownerâs Equity. . . Book Value per Share: Total Equity divided by Shares outstanding, so row 26 / row 32.â Key Decision Point (Step 10: script creation) All six formula groups were generated from abstract financial definitions without verifying against the workbookâs local conventions. The formulas were âtextbook correctâ but the specific cell references (row numbers, column ranges) did not match this particular banking modelâs structure. Outcome Modification accuracy:0.0411 (â3 / 73 cells correct) Collateral damage:Cell E29 returned #VALUE! (expected: 0.0236) Failure mode:Insufficient Inspection + Wrong Target Selection Figure 15: Insufficient inspection and wrong target selection case of Financial Modeling task. D.4.3 Visualization Cases Case 5. As shown in Figure 16, the agent encountered a genuine tool limitationâopenpyxl does not natively support clustered-stacked chartsâand chose an elaborate dual-axis workaround that produced overlapping rather than clustered bars. The agent itself anticipated this risk (âthe bars could overlapâ) but then dismissed it with incorrect reasoning. More critically, all three verification attempts returned negative or empty results, yet the agent stated that âthe chart XML confirms all 4 series are presentââdirectly contradicting its own tool output. This case shows two compounding problems: the agent selected a workaround it could not verify, and when verification failed, it fabricated a positive result rather than reconsidering its approach. Case 6. As shown in Figure 17, the agent correctly determined that no available Python library supports sunburst charts and explicitly considered two viable alternatives: matplotlib embedding and a nested doughnut chart approximation. It rejected bothâreasoning that âthe task specifically asks for a sunburst, so I need to build the real thingââand instead attempted raw XML injection into the XLSX archive using the Office 2016+ Extended Chart namespace. The evaluation system reported âno chart objects,â resulting in a complete failure. A nestedDoughnutChart, which openpyxl fully supports, would have visually approximated the sunburst hierarchy and likely earned partial credit. This case shows that the agent prioritized literal compliance with the task specification over a pragmatic approximation that its tools could actually produce and verify, ultimately achieving a worse outcome than the alternative it rejected. 23 Case 5: Clustered-Stacked Chart (Visualization) Task Description Create a stacked column chart comparing sales for Type A and Type B across Q1âQ4 for 2022 and 2023. Columns should be grouped by year (clustered), with types stacked within each column. Required chart title: âSales Analysis.â Agent Reasoning The agent correctly identified that openpyxl does not natively support clustered-stacked charts and chose a dual-axis workaround: two separateBarChartobjects combined viachart1 += chart2. It even anticipated the risk: âThe bars could overlap rather than cluster properly. . . Actually, the dual-axis approach does work for clustered stacked charts in Excel.â Key Decision Point (Steps 6â7: verification hallucination) Three verification attempts all failed:view_xlsxonly shows cell data; the openpyxl API only exposes the primary chartâs series; an XML regex for<c:barChart>returned 0 matches (namespace mismatch). Despite this, the agent stated: âThe chart XML confirms all 4 series are present. . . creating the clustered stacked effect.â â a direct hallucination contradicting the actual tool output. Outcome VLM score:0.1304 (3 / 23 criteria passed) Key failures:Wrong title, columns overlap instead of clustering, data values incorrect Failure mode:Format/Output Error Figure 16: Format/Output error case of Visualization task. Case 6: Sunburst Chart (Visualization) Task Description Create a sunburst chart titled âExpense Breakdownâ showing a 3-level hierarchy (departmentâcategory âsubcategory) of expense data, sized by spend amount. Required colors: Orange for IT, Blue for HR, Grey for Sales. All labels must be fully contained within their segments. Agent Reasoning The agent probed available tools and correctly determined that neither openpyxl nor xlsxwriter supports sunburst charts. It explicitly considered two viable alternatives: ⢠Matplotlib image embedding â rejected (âno built-in sunburst supportâ). ⢠NestedDoughnutChartapproximation â rejected (âthe task specifically asks for a sunburst, so I need to build the real thingâ). Key Decision Point (Steps 5â6: rejecting viable alternatives) The agent spent 4 steps building a raw XML injection pipeline: extracting the XLSX as a ZIP, creating chartEx1.xmlwith the Office 2016+ Extended Chart namespace (cx:chart), adding relationship files and content type registrations, and re-zipping. Verification confirmed ZIP integrity and file presence but never tested whether the chart was recognizable as a chart object. Outcome Evaluation result:âXLSX has no chart objects to exportâ â complete failure Missed opportunity:A nested DoughnutChart (openpyxl-supported) could have earned partial credit Failure mode:Format/Output Error Figure 17: Format/Output error case of Visualization task. 24 E Data Examples Figure 18: Example of Debugging tasks. 25 Figure 19: Example of Financial Modeling tasks. 26 Input Spreadsheet Ground Truth Spreadsheet Instruction Complete the financial model based on the provided assumptions. Ensure that all formulas, references, and calculations are correct. Figure 20: Example of Template tasks. 27 Input Spreadsheet Ground Truth Spreadsheet Instruction Create a football field valuation chart titled 'Project Mars -Standalone Valuation Summary'. Display horizontal bars showing valuation ranges for each methodology. List methodologies vertically on the Y-axis in this order from top to bottom: DCF (Based on Management Forecast), Precedent Transactions (10.7x -13.0x Management 2025A EBITDA), MgmtProjections 2026E (Based on 9.0x -15.0x Management Projection), MgmtProjections 2027E (Based on 8.5x -14.5x Management Projection), Analyst Target Price Range, 52-Week Trading Range. Use light blue bars for Analyst Target Price Range and 52-Week Trading Range, and dark blue bars for all other methodologies. The X-axis should show share price from $15.00 to $65.00 with gridlines every $5.00. Add data labels on each bar showing the low and high values of the range $41.89 $38.53 $20.90 $36.25 $33.72 $38.65 $51.56 $46.93 $37.99 $58.69 $46.48 $44.40 $15.00 $20.00 $25.00 $30.00 $35.00 $40.00 $45.00 $50.00 $55.00 $60.00 $65.00 DCF... Precedent Transactions... Mgmt Projections 2026E... Mgmt Projections 2027E... Analyst Target Price Range 52-Week Trading Range Project Mars - Standalone Valuation Summary Figure 21: Example of Visualization tasks. 28