Paper deep dive
DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis
Chuhan Zhang, Qi Xie, Ziyue Wang, Jianing Yin, Yunfan Zhou, Dazhen Deng, Yingcai Wu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, to our knowledge the first benchmark to diagnose these failures at a fine-grained level within the same dashboard task. DashAct contains 357 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its progressive diagnostic cascade evaluates end-to-end execution, restores verified context for next-action prediction, and provides target semantics and a local view for visual grounding. By progressively restoring the conditions for success, DashAct measures the minimum support an agent needs to recover rather than scoring isolated skills. Experiments show that current models struggle even as support is added. The cascade outcomes reveal bottlenecks hidden by end-to-end scores and provide actionable guidance for improving GUI agents.
Tags
Links
- Source: https://arxiv.org/abs/2609.32385v1
- Canonical: https://arxiv.org/abs/2609.32385v1
Trouble viewing inline? Open PDF directly →
Full Text
48,853 characters extracted from source content.
Expand or collapse full text
DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis Chuhan Zhang Qi Xie Ziyue Wang Jianing Yin Yunfan Zhou Dazhen Deng Yingcai Wu Abstract Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, to our knowledge the first benchmark to diagnose these failures at a fine-grained level within the same dashboard task. DashAct contains 357 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its progressive diagnostic cascade evaluates end-to-end execution, restores verified context for next-action prediction, and provides target semantics and a local view for visual grounding. By progressively restoring the conditions for success, DashAct measures the minimum support an agent needs to recover rather than scoring isolated skills. Experiments show that current models struggle even as support is added. The cascade outcomes reveal bottlenecks hidden by end-to-end scores and provide actionable guidance for improving GUI agents. 1State Key Laboratory of CAD&CG, Zhejiang University 2College of Computer Science and Technology, Zhejiang University Hangzhou, China chuhanzhang, 3230100951, 3230100860, yinjianing, yf.zhou, dengdazhen, ycwu@zju.edu.cn Introduction Figure 1: Overview of DashAct and its progressive diagnostic cascade. DashAct represents each dashboard task with a human-verified interaction trajectory and milestone-level process annotations. The same task process is evaluated under progressively stronger support, from end-to-end task execution to verified-context action prediction and target-specified local grounding. Recent advances in multimodal foundation models have driven growing interest in graphical user interface (GUI) agents that perceive rendered interfaces, interpret user instructions, and perform actions on behalf of users (Koh et al. 2024; Xie et al. 2024; Cheng et al. 2024). While many GUI tasks involve transactional operations such as navigating webpages, filling forms, or controlling software, dashboard-based data analysis requires agents to interact with visualizations to obtain and connect analytical evidence. Dashboards integrate charts, tables, filters, and coordinated views that support operations such as selecting, filtering, and inspecting data (Yi et al. 2007; Heer and Shneiderman 2012). Answering an analytical question may therefore require an agent to configure a view, filter the data, inspect a visual mark, and interpret the resulting state before producing an answer. In this setting, interaction constitutes the analytical process through which evidence becomes available. Dashboard interaction is challenging because these operations are stateful and interdependent. At each point, an agent must determine what information is needed, choose an appropriate operation, locate its target, and interpret the resulting dashboard state before continuing. An action may change filters, selections, highlights, or coordinated views, while many actionable targets are chart marks, table cells, legends, or visual regions with weak textual anchors (Heer and Shneiderman 2012; Cheng et al. 2024; Kartha et al. 2026). An incorrect action can therefore move the dashboard away from the state required by subsequent analysis. Failures may accumulate along the interaction process, making a later error difficult to separate from mistakes that occurred earlier. This dependency presents a central evaluation challenge. End-to-end evaluation can determine whether an agent completes the task and which required analytical milestones it reaches, but a failed trajectory does not directly reveal what prevented success. The agent may struggle to maintain a valid analytical process across multiple interactions. It may fail to produce the appropriate action even when given the relevant interaction history and dashboard state. It may also understand the intended action but fail to localize its visual target. These limitations are intertwined during online execution and cannot be distinguished from task outcomes alone. We introduce DashAct, to our knowledge the first benchmark designed to diagnose, at a fine-grained level, where GUI agents fail in interactive dashboard analysis. DashAct contains 357 analytical questions and corresponding human-verified interaction trajectories collected from 175 real-world dashboards. The trajectories are annotated with step goals, milestone effects and dependency relations, action types, and view- and item-level target bounding boxes. Together, these annotations link final task outcomes to milestone-level progress, action decisions, and hierarchical visual targets. As illustrated in Figure 1, DashAct organizes evaluation into a progressive diagnostic cascade. End-to-End Task Execution evaluates autonomous dashboard interaction and task completion. Verified-Context Action Prediction provides the correct interaction history and current dashboard state, separating failures inherited from preceding execution from those in the current action decision. Target-Specified Local Grounding further provides a description of the intended target and a screenshot of its containing view, removing next-action selection and cross-view search to focus on local target grounding. Experiments across general-purpose multimodal models, GUI-specialized agents, and grounding-specialized models reveal substantial limitations at all three stages. Agents often complete some required milestones without completing the full analytical process or producing the correct answer. Next-action prediction remains unreliable even when the verified interaction context is provided, and local grounding remains difficult after the intended target and its containing view are specified, particularly for visualization marks. The progressive support analysis further reveals substantial variation in where different models recover, exposing capability profiles hidden by aggregate task success. These findings identify promising directions for future work, including multi-step state and evidence management, target discovery across dashboard views, and feedback-guided grounding correction. Related Work Visual Data Understanding. Visual data understanding has been widely studied through chart question answering and summarization tasks. Existing benchmarks evaluate models’ ability to recognize visual encodings, read chart text, estimate data values, perform numerical or logical reasoning, and summarize high-level insights from charts (Kahou et al. 2018; Kafle et al. 2018; Methani et al. 2020; Masry et al. 2022; Kantharaj et al. 2022; Tang et al. 2023). Recent work has also begun to examine models’ ability to ground visual evidence in charts, requiring them to localize chart regions or points that support visual reasoning (Vogel et al. 2026; Xu et al. 2025). These works mainly focus on static chart images, where the relevant evidence is directly available in the input. DashAct instead evaluates agents in interactive dashboards, where evidence may need to be revealed or connected through filtering, selection, hovering, and coordinated-view navigation. GUI Agents for Interactive Interfaces. GUI agents have emerged as a general paradigm for automating user tasks through interaction with digital interfaces (Nguyen et al. 2025). They have been developed for web navigation and online tasks (He et al. 2024; Hong et al. 2024), as well as broader computer-use scenarios spanning desktop and mobile environments (Lin et al. 2025; Qin et al. 2025; Wang et al. 2025). In visual analytics, LightVA supports collaborative task planning and execution (Zhao et al. 2025), while ProactiveVA provides context-aware assistance based on users’ ongoing analytical interactions (Zhao et al. 2026a). VACP explores a complementary direction by exposing visualization states and available interactions through an agent-oriented protocol (Stähle et al. 2026). These works demonstrate how LLMs can assist users during visual analysis or make visualization systems more accessible to agents. DashAct instead systematically evaluates whether current agents can autonomously discover and execute the interactions required to answer analytical questions on existing dashboards. Its screenshot-only setting requires agents to reason over changing dashboard states and ground actions on rendered controls and visualization marks. Benchmarks for Interactive Agents. Interactive-agent benchmarks commonly use task success as their primary evaluation signal in web environments (Zhou et al. 2024; Koh et al. 2024), desktop, enterprise workflows (Xie et al. 2024; Drouin et al. 2024), and mobile-device control (Rawles et al. 2024). Other benchmarks provide finer-grained signals: Mind2Web and WebLINX use human demonstrations to evaluate next-action prediction (Deng et al. 2023; Lù et al. 2024), VideoGUI decomposes planning and execution (Lin et al. 2024), WebCanvas evaluates intermediate task progress (Pan et al. 2024), and ScreenSpot isolates visual grounding (Cheng et al. 2024). The most closely related benchmark, DashboardQA primarily evaluates whether agents can answer questions after interacting with real dashboards (Kartha et al. 2026), while VizAgentBench evaluates coordinated multi-view analysis through declarative interaction commands (Wu et al. 2026). Building on these works, DashAct adopts screenshot-based interaction on real-world dashboards, enabling unified evaluation across heterogeneous interfaces without relying on platform-specific commands or interface structures. Its milestone annotations capture intermediate analytical progress, while hierarchical target annotations support diagnosis from relevant views to fine-grained targets, allowing failures in dashboard exploration, action selection, and visual grounding to be distinguished beyond task-level outcomes. Benchmark Problem Setup Interactive Dashboard Tasks. We define an interactive dashboard task as a tuple =(D,q,y)T=(D,q,y), where D is an interactive dashboard, q is a natural-language question, and y is the reference answer. A dashboard consists of one or more views, where each view is a spatially distinct interface region that groups related visual content or controls, such as a chart, table, filter panel, or toolbar. A view may contain multiple interactive items, and interactions with an item may update the same view or other views in the dashboard. Starting from the initial state of D, an agent observes the rendered interface and performs a sequence of actions to obtain the information required by q. These actions may include clicking, hovering, dragging, scrolling, typing, and other interface operations, followed by submitting an answer y y. An interaction trajectory is represented as τ=(o0,a0,o1,…,aT−1,oT,y^)τ=(o_0,a_0,o_1,…,a_T-1,o_T, y), where oto_t is the visual observation at step t and ata_t is the action performed on the interface. Dashboard interactions are stateful: an action may change filters, selections, visible views, or the information displayed in subsequent observations. Consequently, the evidence required to answer a question may not be available in the initial interface and must instead be revealed and connected through multiple interactions. Task Milestones. We represent the required intermediate progress of an interaction trajectory using milestones. Each milestone defines a required subgoal in the annotated task process and specifies the action type, the bounding boxes of the relevant dashboard view and target item, and the expected interface state change after the action. Milestones may be subject to precedence constraints, where one must be reached before another, or may remain unordered when they can be reached independently. For each task, the milestones and their precedence relations form a directed acyclic graph, where each directed edge indicates that the source milestone must precede the destination milestone. Benchmark Construction Dashboard sources and trajectory collection. We construct DashAct from real-world multi-view interactive dashboards, primarily sourced from Tableau Public. The selected dashboards cover diverse domains, including retail, finance, logistics, marketing, human resources, public services, healthcare, education, and sustainability. Metadata retrieved through APIs provided by Tableau is used during offline data construction and verification. Three annotators with extensive experience in dashboard analysis conduct purposeful explorations of the selected dashboards with specific analytical intent. For each exploration, the annotator records an executable interaction trajectory, including the performed actions and the rendered interface states throughout the interaction process. A dashboard may contribute multiple questions and corresponding trajectories. Annotation drafting and verification. Given each recorded trajectory and the construction-time metadata, GPT-5 first drafts the task question and candidate answer, together with preliminary process annotations. Human annotators then review and revise all drafted components. During verification, annotators replay each trajectory to ensure that its actions are reproducible and that the recorded screenshots correctly reflect the interaction process. They determine which recorded steps constitute milestones, verify the corresponding step-goal descriptions, milestone effects, and dependency relations, and manually annotate or correct the bounding boxes of the relevant dashboard view and target item for each milestone. They also verify the reasonableness of the task question and whether its answer is obtainable from the recorded trajectory and factually correct. Each completed annotation is subsequently reviewed by another annotator to ensure trajectory and annotation quality. DashboardQA adaptation. We also adapt DashboardQA (Kartha et al. 2026) to the same interaction-based setting. As part of this adaptation, we exclude questions that require combining information across multiple dashboards, because we focus on exploratory analysis within a single dashboard. For each remaining question, annotators collect an executable interaction trajectory and produce the same process annotations and human verification as for DashAct. Dataset Statistics DashAct contains 357 questions and corresponding human-verified interaction trajectories collected from 175 real-world dashboards. The dashboards contain an average of 11.07 views, reflecting the multi-view structure of the interfaces in our benchmark. The reference trajectories contain an average of 2.52 annotated milestones, ranging from 1 to 11 milestones per trajectory. The adapted DashboardQA set contains 242 questions and corresponding human-verified interaction trajectories from 83 dashboards. These dashboards contain an average of 2.45 views, and their reference trajectories contain an average of 3.99 annotated milestones. Detailed statistics and distributions of dashboard domains, view counts, action types, and milestone lengths are provided in the supplementary material. Evaluation Figure 1 illustrates the progressive diagnostic cascade of DashAct. The three stages are derived from the same annotated tasks and trajectories but progressively provide more verified information. Stage 1 evaluates complete end-to-end execution; Stage 2 restores the verified interaction context, removing the need to correctly execute the preceding steps and reach the current intermediate dashboard state; and Stage 3 additionally specifies the intended target and its containing view, removing next-action selection and cross-view search. Performance changes across the stages therefore help localize bottlenecks in understanding and advancing multi-step tasks, predicting the next action, and grounding its target within a local view. End-to-End Task Execution. Given a task question and a live dashboard in its initial state, the agent performs a sequence of interface actions and submits a final answer. This stage evaluates the complete interactive analysis process, including exploring the dashboard, interpreting the intermediate results produced by previous actions, advancing the task across multiple steps, selecting and grounding actions, and synthesizing the final answer. We report Milestone Coverage, defined as the proportion of annotated milestones completed across all online rollouts; All-Milestone Completion, defined as the percentage of tasks in which every annotated milestone is completed; and Answer Accuracy, which measures whether the submitted answer is semantically consistent with the reference answer. Milestone Coverage is computed over milestone instances, whereas All-Milestone Completion and Answer Accuracy are computed over complete tasks. Online milestone evaluation. Because dashboard layouts and states may change dynamically during online interaction, we use Gemini 3.1 Flash-Lite to evaluate milestone completion in Stage 1. The evaluator receives the executed action, the observed interface change, the annotated milestone, and a rule-based matching result as reference. It determines whether the action matches the intended action type and interface target, or whether it produces the intended milestone effect. A milestone is considered completed when either condition is satisfied, allowing valid interaction paths that differ from the reference trajectory. The complete evaluation prompt and agreement analysis with human judgments are provided in the supplementary material. Verified-Context Action Prediction. For each annotated milestone step, the model receives the task question, the human-verified preceding action history, and the correct full-dashboard screenshot immediately before the reference action. It predicts the next action type, a normalized target point for spatial actions, and any required action-specific argument. Providing the verified history and current screenshot removes errors introduced by the model’s preceding actions. This stage therefore evaluates whether the model can understand the current progress of a multi-step task and predict and locate the appropriate next action. We report Action Type Accuracy, which measures whether the predicted action type matches the reference action; Target Hit Rate, which measures whether the predicted point falls within the human-annotated fine-grained target bounding box; and Action Success, which requires all components applicable to the reference action to be correct in a single prediction, including the action type, any required argument, and, for spatial actions, a target point within the annotated bounding box, as determined by a deterministic point-in-box test. Target Hit Rate is computed over milestone steps with valid spatial annotations, regardless of whether the predicted action type is correct, and predictions without a valid target point are counted as misses. Target-Specified Local Grounding. For each milestone target, the model receives a human-verified semantic description of the intended target, such as its visible name or role in the interface, together with a cropped view containing it. The model then predicts a single normalized point. Specifying both the intended target and its containing view removes the need to determine the next action and search across multiple dashboard views, thereby isolating local target grounding. We report Target Hit Rate as the proportion of predictions that fall within the human-annotated target bounding box, as determined by a deterministic point-in-box test. Experiments Experimental Setup Evaluated models. We evaluate nine core multimodal models across the three diagnostic stages. The closed-source models are Gemini 3 Flash Preview, GPT-5, and Claude Sonnet 5. The open-source models are OpAgent-32B (Guo et al. 2026), GUI-Owl-1.5-32B (Xu et al. 2026), Qwen3-VL-8B-Instruct (Qwen Team 2025), GUI-Owl-1.5-8B (Xu et al. 2026), OpenCUA-7B (Wang et al. 2025), and UI-TARS-1.5-7B (Qin et al. 2025). This selection covers both general-purpose multimodal models and GUI-specialized computer-use models; the open-source models range from 7B to 32B. For Stage 3, we additionally evaluate two grounding-specialized models, UGround-V1-7B (Gou et al. 2025) and POINTS-GUI-G (Zhao et al. 2026b), to assess the performance of grounding-specialized models. Unified evaluation environment. All stages rely exclusively on rendered screenshots, without DOM, HTML, accessibility-tree, or page-source information. Stage 1 and Stage 2 use full-dashboard screenshots, while Stage 3 uses annotated local-view crops. Screenshots retain their original resolution and aspect ratio. All models use semantically equivalent prompts, a shared action schema, and coordinates normalized to [0,1000][0,1000], rather than model-specific action formats or official computer-use prompts. For Stage 1, Playwright controls the live dashboard, while the OpenAI Agents SDK and Hugging Face inference stack provide a unified interaction interface. At each turn, the model receives the current screenshot and its accumulated history of actions and textual observations. Each task starts from its specified Tableau URL and initial state, is executed once with a maximum of 20 turns, and ends when the model invokes the answer action or reaches the turn limit. Inference settings. API models use a temperature of zero, while local Hugging Face models use greedy decoding (do_sample=False), with a maximum output length of 512 tokens per call. Environment failures are rerun and are not counted as model failures. Complete prompts, model configurations, image-processing settings, software versions, and failure-handling rules are provided in the supplementary material. Main Results on DashAct Stage 1: End-to-End Stage 2: Verified Context Stage 3: Local Model Milestone Coverage All-Milestone Completion Answer Accuracy Action Type Accuracy Target Hit Rate Action Success Target Hit Rate Closed-source models Gemini 3 Flash Preview 62.7 42.9 42.6 63.5 37.1 31.3 58.4 GPT-5 29.9 14.9 9.6 62.2 6.0 6.6 32.5 Claude Sonnet 5 25.4 10.6 8.1 64.7 2.3 3.0 3.8 Open-source models OpAgent-32B 41.4 21.6 16.3 66.6 29.3 26.3 27.2 GUI-Owl-1.5-32B 36.1 21.0 13.8 92.6 21.7 22.2 27.7 Qwen3-VL-8B-Instruct 26.0 10.1 10.6 47.8 16.5 16.0 31.8 GUI-Owl-1.5-8B 17.2 5.4 9.1 49.0 7.0 6.4 29.7 OpenCUA-7B 7.7 1.7 3.4 98.8 1.5 2.5 4.7 UI-TARS-1.5-7B 11.3 3.9 1.1 43.8 0.3 1.0 4.1 Table 1: Main results on DashAct (%) for the three evaluation stages: end-to-end execution, verified-context action prediction, and local grounding. Bold shows the best value in each column, and underlined shows the second-best. Table 1 presents model performance across the three evaluation stages on DashAct. Results on the adapted DashboardQA set are provided in the supplementary material. We next discuss the main findings. End-to-end interactive dashboard analysis remains difficult. Gemini 3 Flash Preview achieves the strongest Stage 1 performance, with 62.7% milestone coverage, 42.9% all-milestone completion, and 42.6% answer accuracy. Nevertheless, it still misses more than one-third of the required milestones and fails to complete the full process on more than half of the tasks. Among the open-source models, OpAgent-32B performs best online, reaching 41.4% milestone coverage and 21.6% all-milestone completion, followed closely by GUI-Owl-1.5-32B at 36.1% and 21.0%. The smaller models struggle further: Qwen3-VL-8B-Instruct completes all milestones on 10.1% of tasks, while the remaining 7–8B models remain at 5.4% or below. Across models, milestone coverage substantially exceeds all-milestone completion, indicating that agents often make partial progress but much less often complete the full set of required analytical milestones. Providing a verified interaction state is not sufficient for reliable next-action prediction. Gemini 3 Flash Preview achieves the highest Stage 2 Action Success at 31.3%, meaning that even the strongest model correctly predicts fewer than one-third of the required actions in a single attempt. Among the open-source systems, OpAgent-32B performs best at 26.3%, but still fails on nearly three-quarters of the milestone actions. Thus, providing the correct interaction history and dashboard state does not by itself enable reliable prediction of the complete required action. Fine-grained visual grounding remains a major unresolved bottleneck. The grounding bottleneck is already evident in Stage 2, where even the highest Target Hit Rate reaches only 37.1%, while most models remain below 30%; OpenCUA-7B and UI-TARS-1.5-7B achieve only 1.5% and 0.3%, respectively. Stage 3 provides the intended target description and a crop of the relevant view, removing the need to infer the target and search across the full dashboard. Even under this stronger support, Gemini 3 Flash Preview reaches only 58.4% target-hit accuracy, followed by the grounding-specialized POINTS-GUI-G at 49.1% and UGround-V1-7B at 46.7% (Table 3); all remaining core models are at 32.5% or below, and Claude Sonnet 5, OpenCUA-7B, and UI-TARS-1.5-7B remain below 5%. Thus, local grounding remains difficult even after the intended target is specified and the search is restricted to the relevant view. We next move beyond aggregate stage-level results to diagnose failures from three perspectives: process complexity, grounding granularity and target type, and recoverability under progressively stronger support. Diagnostic Analyses Performance by Required Milestone Count Figure 2 stratifies Stage 1 Milestone Coverage and Answer Accuracy by the number of required milestones, which serves as a coarse proxy for the amount of intermediate progress and coordination required by a task. Comparing the two metrics distinguishes partial process progress from final task resolution. Milestone Coverage degrades more gradually than Answer Accuracy as milestone count increases. The average across models shows its largest drop in Answer Accuracy between one- and two-milestone tasks, after which accuracy remains low across the multi-milestone groups. Milestone Coverage declines more gradually, indicating that agents continue to complete some required subgoals even when they fail to produce the correct final answer. Thus, higher milestone count is associated with a much sharper decline in final task resolution than in partial process progress. Models exhibit different profiles on multi-milestone tasks. OpAgent-32B maintains comparatively stable Milestone Coverage, but its Answer Accuracy remains low, showing that continued intermediate progress does not necessarily translate to successful task resolution. Gemini 3 Flash Preview retains the highest Answer Accuracy across the multi-milestone groups, although its performance also drops substantially relative to one-milestone tasks. Figure 2: Stage 1 Milestone Coverage (left) and Answer Accuracy (right), grouped by required milestone count. The black dashed line denotes the average across evaluated models within each milestone-count group. Model View Hit Target Hit Target Hit ∣ Closed-source models Gemini 3 Flash Preview 67.6 37.1 54.9 GPT-5 55.5 6.0 10.8 Claude Sonnet 5 33.7 2.3 6.8 Open-source models OpAgent-32B 47.3 29.3 61.9 GUI-Owl-1.5-32B 52.2 21.7 41.6 Qwen3-VL-8B-Instruct 39.7 16.5 41.6 GUI-Owl-1.5-8B 27.1 7.0 25.8 OpenCUA-7B 35.2 1.5 4.3 UI-TARS-1.5-7B 13.5 0.3 2.2 Table 2: Stage 2 spatial localization performance (%). Target Hit∣ denotes target hit rate conditioned on the predicted point falling within the correct view. Model Overall Controls Marks Closed-source models Gemini 3 Flash Preview 58.4 72.6 48.7 GPT-5 32.5 44.3 24.4 Claude Sonnet 5 3.8 6.2 2.1 Open-source models OpAgent-32B 27.2 39.2 14.0 GUI-Owl-1.5-32B 27.7 34.4 18.3 Qwen3-VL-8B-Instruct 31.8 45.4 16.7 GUI-Owl-1.5-8B 29.7 42.0 15.8 OpenCUA-7B 4.7 5.6 3.3 UI-TARS-1.5-7B 4.1 5.9 2.1 Grounding-specialized models POINTS-GUI-G 49.1 64.8 35.9 UGround-V1-7B 46.7 62.8 33.3 Table 3: Stage 3 local grounding accuracy (%), reported overall and by dashboard target type: interface controls and visualization marks. Bold and underlining indicate the best and second-best results in each column, respectively. Grounding across Spatial Levels and Target Types Tables 2 and 3 diagnose grounding failures along two axes: spatial level and target type. Table 2 separates locating the target’s containing view from identifying the target within that view, while Table 3 compares interface controls with visualization marks. View selection and within-view localization constitute distinct grounding bottlenecks. Gemini 3 Flash Preview achieves the highest View Hit and Target Hit, at 67.6% and 37.1%, respectively. OpAgent-32B exhibits a different profile: although it reaches the correct view less often, with a View Hit of 47.3%, it achieves the highest Target Hit∣ at 61.9%. Thus, Gemini more frequently places its prediction in the correct view, whereas OpAgent is more precise once its prediction falls within that view. By contrast, GPT-5 attains a View Hit of 55.5% but only 10.8% Target Hit∣ , showing that reaching the correct view does not necessarily lead to identifying the intended item. These profiles separate failures in finding the relevant view from those in pinpointing the target within it. Visualization marks are consistently harder to ground than interface controls. All eleven evaluated models perform better on controls than on visualization marks. Gemini 3 Flash Preview achieves the highest accuracy on both categories, at 72.6% on controls and 48.7% on marks. The grounding-specialized models rank next: POINTS-GUI-G reaches 64.8% on controls and 35.9% on marks, while UGround-V1-7B reaches 62.8% and 33.3%, respectively. Both outperform all open-source core models, highlighting the promise of grounding-specific training. The control–mark gap varies across models, but the consistent advantage on controls shows that visualization marks remain substantially more difficult to ground. This pattern persists across all target-size bins, as shown in the supplementary material. This gap is practically important because conventional controls are often exposed as semantically addressable DOM or accessibility elements, whereas visualization marks may lack stable, individually addressable nodes and therefore still require visual grounding. Because these marks must often be hovered over or selected to reveal values and trigger coordinated interactions, reliable mark grounding remains a central requirement for interactive dashboard agents. Figure 3: Milestone-level recovery profiles under progressively stronger support. Each stacked bar groups comparable milestones by the earliest condition under which they succeed; milestones failing under all conditions are unresolved. Bottlenecks Revealed by Progressive Support Figure 3 partitions 889 milestone instances that can be evaluated in all three stages by the minimum support under which each model succeeds. A milestone is categorized as autonomous if it is completed during online execution, context-recoverable if it first succeeds with verified interaction context, target-support-recoverable if it first succeeds with the intended target and containing view provided, and unresolved if it fails under all three conditions. These categories are mutually exclusive and should not be interpreted as independent stage accuracies. Verified context recovers a meaningful share of online failures for several stronger models. The context-recoverable segment accounts for 13.1% of milestones for Gemini 3 Flash Preview, 11.0% for OpAgent-32B, and 10.9% for GUI-Owl-1.5-32B. These milestones fail during autonomous execution but succeed once the preceding history and current dashboard state are restored, pointing to difficulty reaching or preserving the interaction state required for the next decision. Target specification and view isolation recover a second set of failures. This segment is particularly large for GUI-Owl-1.5-8B, GPT-5, and Qwen3-VL-8B-Instruct, at 21.1%, 20.0%, and 17.6%, respectively. These milestones remain unsuccessful with verified context but become solvable when the intended target and its containing view are supplied. Because this support jointly removes next-action selection, target inference, and cross-view search, the result points to a combined bottleneck in determining what to act on and where to find it. Several models remain largely unresolved even under the strongest support. The unresolved segment reaches 69.5% for Claude Sonnet 5, 82.7% for UI-TARS-1.5-7B, and 85.5% for OpenCUA-7B. For these models, neither verified context nor explicit target and view information is sufficient for success, indicating persistent limitations in next-action prediction and local target grounding. By contrast, Gemini 3 Flash Preview leaves only 8.8% of comparable milestones unresolved, showing that models differ substantially not only in autonomous performance but also in how effectively they use additional support. Qualitative Case Analysis To complement the aggregate analyses, we inspect illustrative trajectories showing how the identified bottlenecks manifest in agent behavior. The cases examine multi-step state and evidence management, acquisition of hidden interface states, and the use of interaction feedback for grounding correction. Additional cases and detailed trajectory analyses are provided in the supplementary material. Agents fail to preserve both interface state and intermediate evidence across steps. In some online rollouts, an action successfully changes the dashboard, but the agent fails to recognize the resulting state and repeats the same action, closing a panel or undoing a completed selection. In other rollouts, the agent completes all annotated milestones but still submits an incorrect answer, including in tasks that require combining values observed in different dashboard states or performing a final calculation. This pattern is consistent with failures to retain or synthesize the collected evidence. Together, these cases expose two distinct multi-step demands: tracking the current interface configuration and carrying analytical evidence forward to final answer synthesis. Hidden interface structure can cause state-acquisition failures. Some dashboard targets become accessible only after opening a panel, tab, or menu. In one OpAgent-32B case, the model fails during online execution because it continues searching the visible dashboard without clicking the control that reveals the relevant panel. When Stage 2 provides the verified context with the panel already open, however, it correctly predicts the required action and target. This comparison localizes the failure to reaching the required interface state rather than selecting and grounding the next action once the target is visible. Interaction feedback can support iterative grounding correction in dense views. Dashboard views often contain densely packed controls and visualization marks, making precise localization difficult even after the relevant region has been identified. Selected Gemini 3 Flash Preview rollouts illustrate how online interaction can compensate for an initially imprecise prediction: an initial hover lands near the intended mark, and the model uses the resulting tooltip, highlight, or state change to make small coordinate adjustments until the target is reached. By contrast, unsuccessful rollouts sometimes repeat the same inaccurate click or hover without using the observed feedback to refine the next attempt. These cases show that online grounding depends not only on initial localization accuracy, but also on the ability to recognize unsuccessful attempts and use interaction feedback to refine subsequent actions. Conclusion In this work, we introduced DashAct, a progressive diagnostic benchmark for evaluating GUI agents on interactive dashboard analysis. DashAct contains 357 human-verified interaction trajectories collected from 175 real-world dashboards, together with milestone dependencies and view- and target-level annotations for diagnosing failures across end-to-end execution, verified-context action prediction, and local target grounding. Experiments across general-purpose multimodal models, GUI-specialized agents, and grounding-specialized models show that current agents struggle to complete multi-step analyses, predict actions from verified context, and ground fine-grained visualization targets. The resulting diagnostic profiles reveal distinct bottlenecks across models and provide guidance for developing more capable GUI agents. References Cheng et al. (2024) K. Cheng, Q. Sun, Y. Chu, F. Xu, Y. Li, J. Zhang, and Z. Wu SeeClick: harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 9313–9332. External Links: Link, Document Cited by: Introduction, Introduction, Benchmarks for Interactive Agents.. Deng et al. (2023) X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su Mind2Web: towards a generalist agent for the web. In Advances in Neural Information Processing Systems, Vol. 36, p. 28091–28114. External Links: Link Cited by: Benchmarks for Interactive Agents.. Drouin et al. (2024) A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, D. Vazquez, N. Chapados, and A. Lacoste WorkArena: how capable are web agents at solving common knowledge work tasks?. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 11642–11662. External Links: Link Cited by: Benchmarks for Interactive Agents.. Gou et al. (2025) B. Gou, R. Wang, B. Zheng, Y. Xie, C. Chang, Y. Shu, H. Sun, and Y. Su Navigating the digital world as humans do: universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, Cited by: Evaluated models.. Guo et al. (2026) Y. Guo, W. Yang, S. Yang, Z. Liu, C. Chen, Y. Wei, Y. Hu, Y. Huang, G. Hao, D. Yuan, J. Wang, X. Chen, H. Yu, L. Lei, and P. Di OpAgent: operator agent for web navigation. arXiv preprint arXiv:2602.13559. Cited by: Evaluated models.. He et al. (2024) H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu WebVoyager: building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 6864–6890. External Links: Link, Document Cited by: GUI Agents for Interactive Interfaces.. Heer and Shneiderman (2012) J. Heer and B. Shneiderman Interactive dynamics for visual analysis. Communications of the ACM 55 (4), p. 45–54. External Links: Document Cited by: Introduction, Introduction. Hong et al. (2024) W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y. Wang, Z. Wang, Y. Zhang, J. Li, B. Xu, Y. Dong, M. Ding, and J. Tang CogAgent: a visual language model for GUI agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: GUI Agents for Interactive Interfaces.. Kafle et al. (2018) K. Kafle, B. Price, S. Cohen, and C. Kanan DVQA: understanding data visualizations via question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 5648–5656. External Links: Link Cited by: Visual Data Understanding.. Kahou et al. (2018) S. E. Kahou, V. Michalski, A. Atkinson, A. Kadar, A. Trischler, and Y. Bengio FigureQA: an annotated figure dataset for visual reasoning. In International Conference on Learning Representations Workshop, External Links: Link Cited by: Visual Data Understanding.. Kantharaj et al. (2022) S. Kantharaj, R. T. Leong, X. Lin, A. Masry, M. Thakkar, E. Hoque, and S. Joty Chart-to-text: a large-scale benchmark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, p. 4005–4023. External Links: Link, Document Cited by: Visual Data Understanding.. Kartha et al. (2026) A. Kartha, A. Masry, M. S. Islam, T. Lang, S. Rahman, R. Mahbub, M. Rahman, M. Ahmed, M. R. Parvez, E. Hoque, and S. Joty DashboardQA: benchmarking multimodal agents for question answering on interactive dashboards. In Findings of the Association for Computational Linguistics: EACL 2026, Rabat, Morocco, p. 3385–3407. External Links: Link, Document Cited by: Introduction, Benchmarks for Interactive Agents., DashboardQA adaptation.. Koh et al. (2024) J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried VisualWebArena: evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 881–905. External Links: Link, Document Cited by: Introduction, Benchmarks for Interactive Agents.. Lin et al. (2024) K. Q. Lin, L. Li, D. Gao, Q. Wu, M. Yan, Z. Yang, L. Wang, and M. Z. Shou VideoGUI: a benchmark for GUI automation from instructional videos. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Document, Link Cited by: Benchmarks for Interactive Agents.. Lin et al. (2025) K. Q. Lin, L. Li, D. Gao, Z. Yang, S. Wu, Z. Bai, S. W. Lei, L. Wang, and M. Z. Shou ShowUI: one vision-language-action model for GUI visual agent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19498–19508. External Links: Link Cited by: GUI Agents for Interactive Interfaces.. Lù et al. (2024) X. H. Lù, Z. Kasner, and S. Reddy WebLINX: real-world website navigation with multi-turn dialogue. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 33007–33056. External Links: Link Cited by: Benchmarks for Interactive Agents.. Masry et al. (2022) A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, p. 2263–2279. External Links: Link, Document Cited by: Visual Data Understanding.. Methani et al. (2020) N. Methani, P. Ganguly, M. M. Khapra, and P. Kumar PlotQA: reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 1527–1536. External Links: Link Cited by: Visual Data Understanding.. Nguyen et al. (2025) D. Nguyen, J. Chen, Y. Wang, G. Wu, N. Park, Z. Hu, H. Lyu, J. Wu, R. Aponte, Y. Xia, X. Li, J. Shi, H. Chen, V. D. Lai, Z. Xie, S. Kim, R. Zhang, T. Yu, M. Tanjim, N. K. Ahmed, P. Mathur, S. Yoon, L. Yao, B. Kveton, J. Kil, T. H. Nguyen, T. Bui, T. Zhou, R. A. Rossi, and F. Dernoncourt GUI agents: a survey. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 22522–22538. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: GUI Agents for Interactive Interfaces.. Pan et al. (2024) Y. Pan, D. Kong, S. Zhou, C. Cui, Y. Leng, B. Jiang, H. Liu, Y. Shang, S. Zhou, T. Wu, and Z. Wu WebCanvas: benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373. External Links: Link Cited by: Benchmarks for Interactive Agents.. Qin et al. (2025) Y. Qin, Y. Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y. Li, S. Huang, W. Zhong, K. Li, J. Yang, Y. Miao, W. Lin, L. Liu, X. Jiang, Q. Ma, J. Li, X. Xiao, K. Cai, C. Li, Y. Zheng, C. Jin, C. Li, X. Zhou, M. Wang, H. Chen, Z. Li, H. Yang, H. Liu, F. Lin, T. Peng, X. Liu, and G. Shi UI-TARS: pioneering automated GUI interaction with native agents. arXiv preprint arXiv:2501.12326. External Links: Link Cited by: GUI Agents for Interactive Interfaces., Evaluated models.. Qwen Team (2025) Qwen Team Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: Evaluated models.. Rawles et al. (2024) C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, D. Toyama, R. Berry, D. Tyamagundlu, T. Lillicrap, and O. Riva AndroidWorld: a dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573. External Links: Link Cited by: Benchmarks for Interactive Agents.. Stähle et al. (2026) T. Stähle, P. F. Gyarmati, T. Spinner, R. Sevastjanova, D. Moritz, and M. El-Assady VACP: visual analytics context protocol. arXiv preprint arXiv:2603.29322. External Links: Link Cited by: GUI Agents for Interactive Interfaces.. Tang et al. (2023) B. J. Tang, A. Boggust, and A. Satyanarayan VisText: a benchmark for semantically rich chart captioning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, p. 7268–7298. External Links: Link, Document Cited by: Visual Data Understanding.. Vogel et al. (2026) A. Vogel, O. Moured, Y. Chen, J. Zhang, and R. Stiefelhagen RefChartQA: grounding visual answer on chart images through instruction tuning. In Document Analysis and Recognition – ICDAR 2025, Lecture Notes in Computer Science, Vol. 16026, p. 523–537. External Links: Document, Link Cited by: Visual Data Understanding.. Wang et al. (2025) X. Wang, B. Wang, D. Lu, J. Yang, T. Xie, J. Wang, J. Deng, X. Guo, Y. Xu, C. Wu, et al. OpenCUA: open foundations for computer-use agents. In Advances in Neural Information Processing Systems, Vol. 38. External Links: Link Cited by: GUI Agents for Interactive Interfaces., Evaluated models.. Wu et al. (2026) G. Wu, C. Xu, E. Wu, Z. Yao, and Y. He VizAgentBench: benchmarking multimodal agent reasoning on coordinated multi-view visual analytics tasks. OpenReview preprint. External Links: Link Cited by: Benchmarks for Interactive Agents.. Xie et al. (2024) T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Introduction, Benchmarks for Interactive Agents.. Xu et al. (2026) H. Xu, X. Zhang, H. Liu, J. Wang, Z. Zhu, S. Zhou, X. Hu, F. Gao, J. Cao, Z. Wang, Z. Chen, J. Liao, Q. Zheng, J. Zeng, Z. Xu, S. Bai, J. Lin, J. Zhou, and M. Yan Mobile-agent-v3.5: multi-platform fundamental gui agents. arXiv preprint arXiv:2602.16855. Cited by: Evaluated models.. Xu et al. (2025) Z. Xu, S. Du, Y. Qi, S. Lu, C. Xu, C. Yuan, and J. Guo ChartPoint: guiding MLLMs with grounding reflection for chart reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 426–436. External Links: Link Cited by: Visual Data Understanding.. Yi et al. (2007) J. S. Yi, Y. a. Kang, J. T. Stasko, and J. A. Jacko Toward a deeper understanding of the role of interaction in information visualization. IEEE Transactions on Visualization and Computer Graphics 13 (6), p. 1224–1231. External Links: Document Cited by: Introduction. Zhao et al. (2026a) Y. Zhao, X. Shu, L. Fan, L. Gao, Y. Zhang, and S. Chen ProactiveVA: proactive visual analytics with LLM-based UI agent. IEEE Transactions on Visualization and Computer Graphics 32 (1), p. 451–461. External Links: Document, Link Cited by: GUI Agents for Interactive Interfaces.. Zhao et al. (2025) Y. Zhao, J. Wang, L. Xiang, X. Zhang, Z. Guo, C. Turkay, Y. Zhang, and S. Chen LightVA: lightweight visual analytics with LLM agent-based task planning and execution. IEEE Transactions on Visualization and Computer Graphics 31 (9), p. 6162–6177. External Links: Document, Link Cited by: GUI Agents for Interactive Interfaces.. Zhao et al. (2026b) Z. Zhao, Y. Liu, Y. Liu, H. Wang, L. Tian, X. Zhou, Y. You, Z. Yu, Y. Yu, and J. Zhou POINTS-gui-g: gui-grounding journey. arXiv preprint arXiv:2602.06391. Cited by: Evaluated models.. Zhou et al. (2024) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig WebArena: a realistic web environment for building autonomous agents. In International Conference on Learning Representations, External Links: Link Cited by: Benchmarks for Interactive Agents..