Paper deep dive

WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics

Yutong Chen, Zhike Tang, Zhihao Mai, Zhihao Shuai, Danli Luo, Jing Xu, Weikai Yang

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 87

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 9/27/2026, 3:44:33 AM

Summary

The paper introduces WidgetVA, a widget-centric framework for agentic visual analytics that standardizes interactive components into structured widgets with unified action and perception APIs. It enables vision-language models (VLMs) to operate as autonomous agents by wrapping existing systems or composing new ones from modular widgets. The authors also present WidgetVABench, a benchmark for evaluating multi-step visual analytics tasks, and demonstrate that the framework improves VLM performance while exposing specific limitations in perception, grounding, and planning.

Entities (8)

WidgetVA · framework · 98%WidgetVABench · benchmark · 97%Visual Analytics · domain · 95%Vision-Language Models · technology · 92%Widget · concept · 90%DashboardQA · benchmark · 88%VizAgentBench · benchmark · 88%Vega-Lite · technology · 85%

Relation Signals (5)

WidgetVA → supports → WidgetVABench

confidence 95% · Building on this framework, we present WidgetVABench

WidgetVA → standardizes → Widget

confidence 93% · WidgetVA... standardizes interactive components as structured widgets

WidgetVABench → evaluates → Vision-Language Models

confidence 92% · Experiments across multiple VLMs show that our framework provides an effective scaffold... WidgetVABench... can score Answer, Reference Trace Similarity, and State

WidgetVA → enables → Vision-Language Models

confidence 90% · We study whether modern vision-language models (VLMs) can take on this role as autonomous VA operators

WidgetVA → improves → Visual Analytics

confidence 85% · The results show consistent gains over the baselines... indicating that the framework is effective at making VA interfaces agent-operable

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Visual analytics (VA) enables sensemaking through interactive visualization, but effective analysis often requires experts to translate high-level intents into long sequences of interface operations and iteratively interpret visual feedback. We study whether modern vision-language models (VLMs) can take on this role as autonomous VA operators that observe the interface, plan multi-step exploration, execute interactions, and adapt based on intermediate visual feedback. To support systematic development and evaluation, we first introduce WidgetVA, a widget-centric agentic VA framework that standardizes interactive components as structured widgets with unified action (e.g., filter and zoom) and perception-query (e.g., selection summaries) APIs. This standardization supports two modes of system construction: wrapping an existing VA system to make it agent-operable without rebuilding it, and composing a new system from widgets as modular building blocks. To help agents coordinate across widgets rather than plan each interaction from scratch, each widget further packages reusable analytical workflows, giving agents more than a bare set of callable functions to plan over. Building on this framework, we present WidgetVABench, a benchmark of single- and multi-widget VA tasks that require agents to perform multi-step interactions to uncover evidence and produce verifiable results. Each task also provides fine-grained reference annotations so that WidgetVABench can score Answer, Reference Trace Similarity, and State separately rather than collapsing agent performance into one success score. Experiments across multiple VLMs show that our framework provides an effective scaffold for agentic VA, while the diagnostic measures expose persistent limitations for future work. The WidgetVA framework and WidgetVABench have been released in this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,757 characters extracted from source content.

Expand or collapse full text

WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics Yutong Chen Zhike Tang Zhihao Mai Zhihao Shuai Danli Luo Jing Xu and Weikai Yang* †thanks: Y. Chen, Z. Tang, Z. Shuai, J. Xu, and W. Yang are with the Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.†thanks: Z. Mai is with Nanyang Technological University, Singapore.†thanks: D. Luo is with the School of Electronics and Information Technology (School of Microelectronics), Sun Yat-sen University, Guangzhou, China.†thanks: Y. Chen and Z. Tang contributed equally. W. Yang is the corresponding author. Email: weikaiyang@hkust-gz.edu.cn Abstract Visual analytics (VA) enables sensemaking through interactive visualization, but effective analysis often requires experts to translate high-level intents into long sequences of interface operations and iteratively interpret visual feedback. We study whether modern vision-language models (VLMs) can take on this role as autonomous VA operators that observe the interface, plan multi-step exploration, execute interactions, and adapt based on intermediate visual feedback. To support systematic development and evaluation, we first introduce WidgetVA, a widget-centric agentic VA framework that standardizes interactive components as structured widgets with unified action (e.g., filter and zoom) and perception-query (e.g., selection summaries) APIs. This standardization supports two modes of system construction: wrapping an existing VA system to make it agent-operable without rebuilding it, and composing a new system from widgets as modular building blocks. To help agents coordinate across widgets rather than plan each interaction from scratch, each widget further packages reusable analytical workflows, giving agents more than a bare set of callable functions to plan over. Building on this framework, we present WidgetVABench, a benchmark of single- and multi-widget VA tasks that require agents to perform multi-step interactions to uncover evidence and produce verifiable results. Each task also provides fine-grained reference annotations so that WidgetVABench can score Answer, Reference Trace Similarity, and State separately rather than collapsing agent performance into one success score. Experiments across multiple VLMs show that our framework provides an effective scaffold for agentic VA, while the diagnostic measures expose persistent limitations for future work. The WidgetVA framework and WidgetVABench have been released in https://github.com/Hiverwin/widgetva. Index Terms: Visual analytics, agentic system I Introduction Figure 1: In WidgetVA, interactive views are exposed through a unified widget so that humans and agents can explore and analyze together in the same workspace. WidgetVA supports wrapping an existing VA page to make it agent-operable or composing a new system from widgets, and it provides reusable workflows to support multi-step planning. We also construct WidgetVABench to evaluate agent performance and exposes persistent limitations for future work. Visual analytics (VA) combines interactive visualization with computational analysis to support sensemaking over complex data [1, 2, 3]. However, effective VA requires analysts to iteratively formulate hypotheses, translate high-level intents into sequences of interface operations (e.g., filtering, brushing, zooming), interpret intermediate visual feedback, and revise their analytical strategy. This iterative loop is cognitively demanding and requires expertise in both the application domain and the VA system itself [4, 5]. As a result, it limits who can effectively use VA systems and how quickly insights can be reached. Recent progress in large language models (LLMs) and vision-language models (VLMs) has created an opportunity to delegate parts of this interaction loop to an agent. Letting an agent analyze raw data through code is one way to reduce this burden for many purely computational questions, but it bypasses the exploration tools that VA already provides, which structure search and leave an inspectable trace for a human analyst to follow, correct, or build on. Rather than bypass these tools, a growing number of recent systems augment themselves with large models to facilitate tool onboarding [6], data exploration [7, 8], and automated insight summarization [9]. However, models in these systems are largely positioned as assistants that do not autonomously operate VA systems [10], and their reasoning remains tightly coupled to application-specific states, functions, and front-end operations, which limits reuse across VA systems. On the other hand, general-purpose GUI agents provide broader application coverage, but screenshots and low-level mouse or keyboard actions expose limited analytical semantics for planning and state verification [11, 12, 13]. Neither route yet provides a systematic, reusable way to make VA interfaces agent-operable across systems, or evidence of how far current VLMs can already carry out such visual analysis tasks. Our goal is therefore to build the former and measure the latter: to make interactive VA environments agent-operable for human–agent co-analysis, where agents share, revise, and continue analysis from the same views and interaction state as a human. Realizing these two goals, however, raises four linked technical challenges. First, different VA implementations expose incompatible, provider-specific states and operations, so it is unclear how to define a single agent-facing representation that captures the same analytical semantics, such as encodings, transforms, and selections, without depending on any one rendering library. Second, when multiple widgets are coordinated, such as a selection in one view filtering another, the resulting state changes must remain attributable to the action that caused them, rather than appearing as untraceable side effects. Third, translating a high-level analytical goal into a correct sequence of concrete operations requires the agent to verify each intermediate outcome and adapt its plan accordingly, rather than committing to an unverified action sequence end-to-end. Finally, final-answer accuracy alone cannot reveal whether a failure arises from answer synthesis, operation grounding, or workspace-state construction, yet scoring against one fixed reference trace would unfairly penalize an agent that reaches an equally valid state through a different path. To address these challenges, we introduce a widget-centric agentic VA framework that separates a model-facing interaction abstraction from the host system’s visualization grammar and rendering implementation (Figure 1). Each widget is represented as Wi=(Si,i,i)W_i=(S_i,A_i,P_i), comprising its semantic state, typed action capabilities, and side-effect-free perception queries. Individual widgets compose into a workspace representation that records shared state, semantic links, and coordination effects across views. The same contract is therefore usable in two ways: wrapping an existing VA system so that it becomes agent-operable without being rebuilt, and composing widgets to assemble a task-specific VA workspace quickly. Operating over this workspace, a closed-loop scaffold repeatedly observes the current widget or workspace state, selects and executes an action or perception query, incorporates the returned evidence, and plans the next step. To facilitate agent planning, an analytical workflow library supplies reusable analysis workflows for translating recurring visual-analysis goals into widget-level action and perception sequences. Built on our framework, we develop WidgetVABench, a benchmark of representative VA tasks where interaction is necessary to surface evidence and reach verifiable results. Each task packages an expected answer, an executable reference trace, and task-relevant conditions on the final semantic state. Tasks are further organized by whether their analytical target and path are known, following the knowledge-gap characterization of analytic guidance [14]. Together, these reference annotations and the Target–Path task taxonomy support fine-grained diagnosis: rather than reducing agentic VA to a single end-task score, WidgetVABench scores Answer, Reference Trace Similarity, and State as separate dimensions and further breaks results down by knowledge-gap category. These signals help diagnose failures in answer reasoning, operation grounding, and analytical-state construction, respectively. The resulting failure patterns can inform the next generation of agent-ready VA systems, for example by indicating whether action APIs, exposed state, or verification feedback should be redesigned. Moreover, since human and agent operations already share the same contract, ordinary analysis sessions can be recorded as structured traces for later data collection, skill construction, or agent training. To assess the framework’s contribution, we compare multiple state-of-the-art VLMs across four settings: No Tools, Direct Tools, Widget Abstraction, and Widget Abstraction + Workflow. The results show consistent gains over the baselines across all three VLMs, indicating that the framework is effective at making VA interfaces agent-operable. Our diagnostic analysis further identifies recurring errors in chart perception, tool grounding, and multi-step planning that persist even under the framework. We use these findings to characterize current limitations of agent–VA interaction rather than to present a durable ranking of rapidly evolving models. In summary, this paper makes three contributions: • A scoped, model-facing widget and workspace abstraction, together with a concrete implementation that makes interactive VA components agent-operable and their state transitions inspectable. • WidgetVABench, an interaction-centric benchmark spanning single-widget and coordinated multi-widget VA systems, with Target–Path task categories and separate annotations for answers, reference traces, and final states. • A diagnostic evaluation of multiple VLMs that compares levels of interaction support and identifies limitations in perception, operation grounding, state construction, and multi-step planning. I Related Work I-A Agentic Visual Analytics and Model-Facing Interfaces The study of conversational visualization has long explored the combination of natural language and visualization for analysis (e.g., when and how to embed visual context in conversation) [15]. Recent breakthroughs in large language models (LLMs) and vision-language models (VLMs) have pushed this area forward, and VIS/TVCG research has shown that LLMs can be integrated into VA systems to support onboarding, exploration, and summarization. For instance, LEVA interprets visualization designs and view relationships from system specifications, suggests insights based on system status, and assists in report generation based on interaction history [6]. To complement this, ProactiveVA points out that even the latest LLM-assisted VA systems are only useful when explicitly asked for assistance, and proposes an LLM-assisted UI agent that monitors interaction logs to proactively identify when the user needs assistance, reason about what to provide, and execute that assistance through the interface in an interpretable and controllable manner [8]. Other related work also investigates how to improve context and intermediate insight management in LLM-assisted analysis systems (e.g., InsightLens) or integrate text and visualization for interactive data analysis (e.g., DASH) [16, 17]. Overall, these works indicate a clear trend: VA systems are moving from passive systems to increasingly agentic systems. However, most of these systems are tightly coupled to a specific front-end and interaction design, which makes it hard to reuse the same model in different tools or to compare agent behaviors. Other work has therefore sought more general and reusable methods that are not tied to a single VA front end. A visualization grammar such as Vega-Lite provides a portable specification of how data, visual encodings, and interactions are compiled into views [18], but such a grammar alone does not provide agentic capabilities for inspecting and operating an instantiated visualization. Multimodal UI agents instead pursue generality by operating directly on screenshots with low-level actions. SeeClick and ScreenAgent improve GUI grounding and computer control by learning to identify and manipulate GUI elements from visual input [11, 12, 13]. These examples illustrate the potential of computer-using VLMs, but pixel-level interfaces are often opaque about intermediate state and debugging, particularly for analytical tasks that demand precise state monitoring, verification, and user control. Our widget abstraction therefore sits between these two directions: it reuses instantiated visualization components rather than replacing their host grammar, and exposes them through structured states, intent-level actions, and perception queries with consistent inputs and outputs. I-B Benchmarks for Visualization Reasoning Current visualization benchmarks and studies are largely based on static charts, focusing on visual question answering and visualization literacy. For example, Xu and Wall investigate LLMs’ capability to execute low-level analytic tasks directly on SVG visualizations of charts [19]. However, practical dashboards involve complex multi-view visualizations and rich interactions, where important evidence is often hidden until filtered, drilled down, or hovered over. DashboardQA brings agentic question answering to real-world Tableau dashboards in a virtual desktop environment, requiring VLM GUI agents to execute multi-step GUI tasks while preserving intermediate state [20]. VizAgentBench also assesses multimodal agents on real-world, interactive, and coordinated dashboards with a declarative interaction API that decouples perception and action [21]. These benchmarks have greatly contributed to the area by recognizing interactivity as a fundamental aspect of dashboard reasoning. However, the state of evaluation is still fragmented: existing benchmarks are QA-focused and mostly treat the interface as a pixel-level domain, which makes it hard to analyze agent failures: whether they arise from visual perception, fragile state management, poor long-horizon planning, or inadequate error recovery. Moreover, they offer little leverage for a systematic investigation of the role of interface representations and feedback mechanisms, i.e., what the system reveals to the agent beyond the raw pixel data, in facilitating robust autonomous operation, which is a problem that directly relates to VA system design. To address this problem, WidgetVABench emphasizes tool-mediated interaction with visualization widgets, provides both objective QA and exploratory questions, and records structured intermediate tool traces and final states to facilitate fine-grained analysis of agent behavior. I-C Reusable Analytical Workflows Visualization research has developed task and interaction taxonomies to describe recurring analytical goals. Amar et al. identify and classify low-level activities as retrieving values, filtering, finding extrema, characterizing distributions, detecting anomalies, and examining correlations [22]. Brehmer and Munzner connect an analyst’s motivation, target, and method through a multi-level task typology [23], while interaction taxonomies describe intents [24, 25]. These abstractions establish the analytical operations that a VA system should support. They provide less guidance on how several operations should be ordered, when an intermediate result should be inspected, or how evidence should be transferred between coordinated views during an agent’s execution. Prior work has addressed parts of this problem. Provenance models capture how an analysis develops over time [26], and grammar-based methods map low-level interaction logs to higher-level analytical activities [27]. Gadhave et al. represent interactions at a semantic level so that completed analyses can be reused with updated data [28]. These approaches are effective for preserving and reapplying previously demonstrated procedures, but generally assume that a suitable analysis has already been carried out and recorded. ATWL instead represents workflows at a higher level as typed artifacts connected by intent-bearing transformations, supporting their description, comparison, and reuse across analytical domains [29]. Such representations characterize the structure of analytical processes, while offering limited guidance on which recurring analysis pattern an agent should adopt for a new visual-analysis query. Our workflow library addresses this gap by summarizing common visual-analysis patterns as planner-facing knowledge. It covers patterns within single views and across multiple views, drawing on established analytical-task and coordinated-view practices [30, 31]. Making this library available to the agent’s planner reduces the need to derive an analysis strategy from scratch and helps agents complete a broad range of classical visual-analysis tasks more consistently. I Widget-Centric Agentic VA Framework We aim to develop and evaluate VLM agents that can complete VA tasks by operating interactive interfaces. However, current VA tools expose heterogeneous interfaces that make agent behavior hard to reproduce and diagnose. A standardized interface is therefore necessary to make VA systems easier for agents to operate, as well as to make agent behavior reproducible and diagnosable. To this end, we present WidgetVA, a widget-centric framework that standardizes interactive visualization components as widgets with explicit state, action APIs, and perception APIs. I-A Design Requirements The framework was iteratively refined through biweekly interviews with three experts (E1–E3) in developing visual analytics systems and agent systems. E1 is a visualization researcher with extensive experience in building VA systems. E2 and E3 are Ph.D. students in computer science with one year of experience building tool-using agents. None of them are coauthors of this paper. We distilled their feedback from these discussions into three design requirements. R1: Intent-aligned, structured interaction. The supported interactions should be both model-friendly and analytically meaningful. Rather than requiring pixel-level control (e.g., mouse clicks and keyboard shortcuts), the framework exposes high-level interactions that match analysts’ intent (e.g., filter, zoom, select, and re-encode), which are more meaningful and easier to reason about for both humans and agents. E2 commented, “a model-friendly interface should expose intent-level operations with typed parameters, so an agent can plan and act in data space rather than chase pixels.” E3 also argued that while pixel-level control seems more robust, it may fail with unreliable grounding and introduce unnecessary complexity for the model. Moreover, this design reduces ambiguity for the model and yields cleaner, comparable traces. The trace of interaction history, including tool calls and parameters, can be directly inspected and evaluated across runs. R2: Queryable intermediate states. VA tasks are typically performed through a sequence of interactions rather than a one-shot problem [32, 33]. After each operation, users need to verify what changed, extract evidence, and decide what to do next. E2 emphasized: “The agent must be able to immediately read back outcomes after taking an action. Otherwise, the agent cannot reliably recover from mistakes or refine hypotheses.” E1 added that it might be useful to provide the results in both text and visual format: “while the text format is more precise, the visual format is more intuitive and easier to understand for some tasks, such as identifying clusters in a scatterplot.” Therefore, the framework is designed to pass both the updated state (text format) and the new rendered view (visual format) into the next interaction run. In addition, this design supports a more robust evaluation by comparing the interface’s intermediate states rather than relying solely on the final answer and the trace of the interaction history. R3: Portable and unified interface. A portable and unified interface is necessary to abstract heterogeneous VA systems into consistent, model-friendly schemas and tool signatures. Such uniformity reduces the learning and control burden for VLM agents. E1 commented, “By making each interactive visualization component a plug-and-play object with a standardized API, developers can easily build VA systems and directly reuse the agent capabilities inherited from the framework.” Meanwhile, this design makes the framework extensible: new visualizations can be easily supported by implementing the widget schema. I-B Widget Abstraction Guided by the design requirements, we take a widget as the unit of agent-facing abstraction. A widget is self-contained enough to expose a coherent analytical state and intent-level operations (R1–R2), and compositional enough to serve as a building block for coordinated workspaces (R3). Each widget therefore exposes its interaction-relevant context and capabilities through an implementation-agnostic contract =(S,,)W=(S,A,P), comprising widget state S, action A, and perception queries P. I-B1 Widget State The widget state specifies the widget’s view and interaction configuration, enabling a visualization to be rendered from the state. Inspired by the visualization pipeline proposed by Card et al. [34], we represent the state of a widget as Si=(Di,Ei,Ti,Vi,Seli,Fi),S_i=(D_i,E_i,T_i,V_i,Sel_i,F_i), (1) with the following components: • Data Context (DiD_i) records references to the source and important information about the current data view, such as the numbers of visible and selected records. • Encoding Context (EiE_i) records the semantic bindings between data fields and visual channels such as position (x, y), color, size, and shape. • Transformation Context (TiT_i) records transformations, such as filtering, sorting, and aggregation, as well as their source references when a transformation is induced by an action issued on a linked widget. • View Context (ViV_i) specifies view-level configuration and transforms, such as scale domains and ranges, axis configuration, and pan/zoom extents when applicable. • Selection Context (SeliSel_i) records active point, interval, or predicate selections. • Interaction Feedback (FiF_i) records transient evidence returned by the widget, such as hovered items and tooltips. In practice, a VA system usually contains multiple coordinated widgets, so we support composing individual widget states into a workspace state that captures shared state across widgets, their link topology, and the coordinated interactions. I-B2 Action Each action corresponds to an intent-level interaction (e.g., filter, zoom), which takes the current widget state and typed parameters as input and returns an updated widget state. Rather than enumerating all concrete actions per widget, we organize them by their underlying intent, drawn from interaction techniques commonly supported by interactive visualizations, into six primitives: data transformation (Filter, Sort, Drill-down, Aggregate), visual mapping (Re-encode), view transformation (Zoom), interactive selection (Brush/Select, Highlight, Focus), analytic augmentation (Overlay/Annotate), and navigation (Navigate, Add/Remove). These intents are instantiated as widget-specific tools. For example, the Zoom primitive becomes line.zoomXRegion for a line chart and scatter.zoomDomain for a scatterplot. Not every widget supports every primitive, and some widgets may provide additional actions beyond existing primitives (e.g., expanding stacked bars). I-B3 Perception Queries Perception queries take the current widget state as input and return structured, machine-readable information about what the widget currently shows. Unlike actions, perception queries are side-effect-free, i.e., they deterministically return the same result given the same state and parameters. This separation enables closed-loop exploration, where the agent alternates between executing actions and issuing perception queries to validate intermediate results. Similar to actions, we derive perception primitives and organize them into three categories: Inspect exposes structured evidence such as view configuration and raw data, Summarize returns lightweight aggregates for quick comparison and sanity checks, and Compute derives higher-level analytical results such as correlations. Together, they form a common interaction contract between agents and VA systems. By decoupling this contract from implementation-specific view representations (e.g., Vega, D3, ECharts), agents gain a unified interface for perceiving and manipulating widgets across heterogeneous toolkits. I-C Orchestrating Agent-Widget Interaction Interaction Orchestration The widget abstraction describes the analytical state and the capabilities available during analysis. However, visual analysis usually requires multiple interactions: an action changes the state, while a perception query inspects the resulting view or obtains further evidence needed for subsequent analysis. We therefore introduce a lightweight orchestration layer to coordinate the use of the widget contract across successive agent turns. At each turn, the orchestrator provides the agent with the user query, the current rendered view, the corresponding widget or workspace state, and the available capabilities. The agent may answer directly or invoke an action or perception capability based on this context. After the call is executed, the updated state or returned evidence is incorporated into the next turn. In coordinated views, changes propagated through widget links are also reflected in the workspace state and are visible to the agent at each turn. Consequently, each subsequent plan is grounded in the actual outcome of the preceding interactions. Analytical Workflow Library During the development of our work, we repeatedly observed common sequences for recurring analytical goals, both in agent runs and in human annotation. These patterns were useful because they turned a multi-step analysis into a reusable procedure, rather than requiring the agent to reconstruct a plan from a bare tool list at every turn. We therefore compiled a workflow library from two complementary sources. First, we reviewed recurring task abstractions and interaction intents reported in visualization task and interaction taxonomies [22, 24, 23]. Second, we examined human-annotated interaction traces collected during task generation and identified common patterns that repeatedly appeared across annotated instances. We then summarized these general patterns and common analytical procedures to create a workflow library. The resulting library contains 38 single-widget workflows and 38 multi-widget workflows spanning two to six widgets. Each workflow records its relevant widget types, analytical intent, representative scenarios, and guidance on how to perform the analysis. The single-widget workflows include procedures such as constrained ranking and interval-specific anomaly search, while the multi-widget workflows cover procedures such as aggregate-to-detail investigation and repeated cohort comparison. Figure 2: Overview of our integrated analysis environment. (A) The Data Panel initializes an analysis session from uploaded tabular data or a supported visualization specification. (B) The Analysis Panel supports mixed-initiative analysis by allowing users to record useful insights and inspect the agent’s iterative analysis process. (C) The Visualization Workspace renders the current visualization state. (D) The Interaction Trace Panel records the analysis as a branch-aware provenance graph and links each analysis step to its corresponding iteration card and workspace state. I-D Implementation We implement the proposed framework as WidgetVA Kit, a JavaScript toolkit that realizes the widget abstraction and exposes it for integration with practical VA environments. It supports both adapting existing VA systems to be agent-operable without rebuilding them from scratch, and constructing new VA systems by composing widgets as reusable building blocks. I-D1 WidgetVA Kit Architecture The implementation follows the same widget-centric abstraction introduced in Section I-B. The kit defines shared contracts for widget and workspace states, actions, perceptions, and inter-widget links, so that these elements have a consistent representation across different visualization implementations. For each visualization kind, a widget family instantiates this contract by defining the actions and perception queries supported by that kind, together with the interaction information needed to operate the widget. A provider denotes the underlying visualization implementation used to realize a widget, such as Vega-Lite, D3, ECharts, or VGPlot. The corresponding provider adapter connects the standardized widget semantics to the provider’s native view and interaction mechanisms. It applies widget-state changes to the rendered view and captures provider-specific human interactions so that their effects are reflected back in the widget state. This separation allows the analysis workflow to operate on a consistent widget interface while each visualization retains its native rendering and interaction logic. Multiple widget instances can further be composed through the workspace representation, enabling the same abstraction to support both individual visualizations and coordinated multi-view systems. Extensibility Our implementation currently supports six widget types: bar charts, line charts, scatterplots, heatmaps, parallel coordinates, and Sankey diagrams. We include parallel coordinates and Sankey diagrams to demonstrate that the widget abstraction can accommodate more complex interactive visualizations beyond conventional statistical charts. Developers can extend existing widget types with a new action or perception query by providing the descriptor that specifies its name, applicable widget kinds, typed parameters, and expected effects. Developers can also register a new widget type by providing a widget-family module that declares the widget kind, supported actions, perception queries, and executable implementations. I-D2 Integration We demonstrate two ways to integrate WidgetVA with practical VA environments. Integration with Existing VA Systems For existing VA systems, such as interactive multi-view examples in the Vega-Lite gallery11 1 https://vega.github.io/vega-lite/examples/interactive_seattle_weather.html, we provide lightweight integration through a browser extension. The extension connects to the page’s native visualizations and maps interaction-relevant information to the corresponding widget and workspace representations without modifying the original implementation. This integration exposes a common interface for reading the current state, invoking actions, issuing perception queries, and accessing interaction history, with communication handled through a message bridge. Once connected, the existing page state serves as the initial analytical context, while subsequent interactions and observations are mediated through the same widget interface used by natively integrated VA systems. Composing New VA Systems Users can also compose widgets to build a new VA system directly. Specifically, they describe the workspace in a specification listing each widget’s provider, native view specification, and any links among coordinated views. The kit then instantiates the corresponding widgets via provider adapters and connects them into a shared workspace. Similar to Vega and ECharts, we use a declarative workspace specification, which is well suited for assembly. To better facilitate human–agent collaborative analysis, we also develop an integrated analysis environment. Users first upload datasets and visualization specifications in the Data Panel (Figure 2A). The result is then rendered in the Visualization Workspace (Figure 2C), where both human and agent can freely interact and explore. The Analysis Panel (Figure 2B) acts as the conversational and procedural control surface of the system. Users switch control between the human and the agent: humans can annotate insights directly, while the agent plans and acts step by step, with each iteration shown as an Observe–Plan–Act–Verify–Reason card followed by grounded suggestions for the next step. The Interaction Trace Panel (Figure 2D) visualizes the analysis as a branch-aware provenance graph, where each node is a widget state labeled by its producing action, with human- and agent-initiated nodes visually distinguished. Nodes link to their corresponding iteration card and workspace state, so selecting one synchronizes all panels and lets users jump to that point in the history, with additional controls for navigating longer traces. IV WidgetVABench Built on this widget abstraction, we construct WidgetVABench, a benchmark for evaluating whether agents can solve VA tasks in both single- and multi-widget workspaces. Each instance consists of a workspace, a task prompt, and multiple layers of reference annotations, including a target answer, a reference tool trajectory, and the expected final workspace state. These annotations enable us to evaluate not only the correctness of an agent’s conclusion, but also the process by which it reaches that conclusion. We next describe the construction of the widget pool and task suite, followed by the evaluation protocol. IV-A Workspace Construction Dataset Sourcing and Selection We construct the benchmark from a combination of real-world and synthetic datasets. For real-world data, we source candidate datasets from Kaggle and retain those that are compatible with at least one supported widget type and provide sufficient analytical structure to support meaningful interaction, excluding datasets whose typical questions can be answered directly from a single static view. This process yields 96 real-world datasets spanning business, public services, education, and other applied domains. To complement these datasets, we additionally construct 304 synthetic datasets with realistic application contexts. The synthetic datasets allow us to broaden the coverage of data characteristics and analytical relationships while providing controlled and verifiable task settings that are difficult to obtain consistently from independently collected real-world data. Workspace Instantiation For each dataset, we generate the visualization specification using data-type-aware field-mapping rules. The converter identifies numerical, categorical, and temporal fields and assigns them to the semantic roles required by each widget. When a benchmark instance requires specific analytical fields, these mappings are provided explicitly instead of using the default selection. For a single-widget workspace, the resulting visualization specification is rendered and bound as one widget to form one instance. For a multi-widget workspace, we generate the visualization specifications for two to six widgets and instantiate them together as one coordinated workspace. The workspace records the widgets and their coordination relations, and serves as the visual analysis environment of that multi-widget benchmark instance. Additional construction details, including the single-widget field-to-encoding conversion and the multi-widget workflow instantiation procedure, are provided in Supplementary Sec. S2.1, and the shared benchmark-instance schema is documented in Supplementary Sec. S2.2. IV-B Task Construction Given a workspace, we construct analytical tasks that require agents to interact with its visualizations to derive an answer. To capture different levels of task specification and analytical autonomy, we characterize tasks along the target and path dimensions following the knowledge-gap formulation of Ceneda et al. [14], which is also consistent with visualization task typologies that distinguish analytical goals from the means used to achieve them [23]. The target dimension indicates whether the expected form of the analytical finding is specified, whereas the path dimension indicates whether the relevant evidence and major analytical steps are provided. Their combinations yield four task types, summarized in Table I. We use this taxonomy to guide task construction across workspaces, ensuring systematic coverage of tasks ranging from fully specified analyses to open-ended exploration. Specifically, for each workspace, the authors draft seed tasks covering the four task types. This design allows the benchmark to assess not only whether an agent can execute prescribed analytical procedures, but also how well it can determine appropriate analytical paths and identify relevant findings when less guidance is provided. Table I: Task taxonomy based on target and path specification. Type Target Path Characteristic Example TK–PK Known Known Specified analysis Select the largest and smallest wind-farm regions, compare their average vibration values and proportions of high-vibration turbines, and identify the region that performs worse on both indicators. TK–PU Known Unknown Goal-directed analysis Between the largest and smallest wind-farm regions, which requires a more urgent maintenance review? Support the answer with evidence. TU–PK Unknown Known Guided discovery Inspect region size, vibration distributions, and anomalous turbine records in sequence, and report any noteworthy pattern discovered. TU–PU Unknown Unknown Open-ended exploration Explore the wind-turbine dashboard and identify a phenomenon that warrants further investigation. IV-C Annotation Construction Human Annotation We recruited eight annotators from a local university, all with computer-science backgrounds and prior experience using visualization for data analysis, to solve the designed tasks through an annotation interface that exposes the same widget operations used during evaluation. The system records the final answer, interaction trace, final widget or workspace state, and annotator-provided reasoning and key insights. We convert these records into structured reference annotations for Answer, Reference Trace Similarity, and State evaluation. This process yields 516 single-widget instances and 608 multi-widget instances. Detailed interface implementation and annotation validation procedures are provided in Supplementary Sec. S2.2. Unified Instance Representation All task sources are converted into a shared instance representation. Each instance contains the analytical query and workspace specification, together with three evaluation-facing annotations: the expected answer or reference insights, a successful reference interaction trace with accepted alternatives, and task-relevant final-state checks. The instance additionally records its Target–Path type, workspace scope, and provenance. This unified representation allows human-authored instances to be evaluated with the same protocol. IV-D Evaluation Protocol A single end-task accuracy score cannot fully reflect an agent’s performance, since it conflates a correct answer with the validity of the interaction process that produced it. Because each task instance already packages an expected answer, a reference tool trace, and task-relevant state conditions, we evaluate along three matching dimensions: Answer, Reference Trace Similarity, and State. Reference Trace Similarity checks whether the agent’s interaction trace is consistent with a valid evidence-gathering process, so a plausible answer produced without a supporting trace is not mistaken for genuine interaction competence. However, scoring the trace alone against one human-annotated reference would penalize an agent that reaches an equally useful state through a different, still-valid trace, so State credits any trace whose final workspace state is adequate to answer the query, independent of how that state was reached. Answer Score The Answer Score evaluates the final response against each task’s expected answer using an evaluation procedure suited to the task type. For tasks with deterministic ground-truth answers, we apply answer-format-specific matching. Specifically, numeric answers are checked with a tolerance threshold, boolean answers with normalized string matching [35], and categorical answers with exact matching plus an LLM judge for a semantic-equivalence check [36, 37]. For Target-Unknown (TU) tasks, we score against a set of reference insights. The evaluator computes claim precision P, claim recall R, and uses a rubric to score groundedness G, which assesses how faithfully the response preserves available analytical evidence. The detailed rubric is presented in Supplementary Sec. S2.3.1. The score is calculated as Sanswer=2PRP+R⋅G.S_answer= 2PRP+R· G. (2) We use Claude Sonnet 4.6 [38] as the judge model and compare its scores with two independent human raters on open-ended tasks [39, 40, 41, 42]. The judge reaches ICC(2,1) values of 0.760.76 and 0.730.73 with the two raters, showing good agreement with human ratings. The details are presented in Section V-A2. Reference Trace Similarity This dimension evaluates whether the agent completes the required interaction steps in an order consistent with the reference trace. Each required reference step specifies an operation, a target widget, the parameters, and additionally provides accepted alternative operations. Because an agent may issue exploratory calls that are not part of the reference trace, comparing the two sequences position by position would unfairly penalize otherwise correct behavior. We therefore align the agent’s successful calls with the reference steps in the spirit of longest-common-subsequence matching: each reference step may be matched to at most one agent call, matched pairs must preserve the original order of the reference steps, and any extra agent calls between two matches are simply skipped rather than penalized. Among all such order-preserving alignments, we take the one that maximizes the total similarity between matched pairs, and report the trace score as this maximum total similarity divided by the number of required reference steps. Optional reference steps and unmatched agent calls are reported separately for diagnostic purposes but do not affect the score. State Score This dimension assesses whether the executed interactions produce a final state that appropriately supports the task, rather than rewarding an interaction trace that looks plausible but leaves the visualization in an ineffective or incorrect state. State Score and Reference Trace Similarity are therefore complementary rather than redundant: Reference Trace Similarity checks whether the agent’s process resembles a valid evidence-gathering procedure, while State Score checks whether the agent actually ended up in a state that supports the task, regardless of which procedure it followed to get there. Annotators specify the conditions that the final state should satisfy to solve the task, and the evaluator checks whether the final state satisfies these conditions and reports the fraction of the conditions that are satisfied. V Evaluation We evaluate the effectiveness of the framework on WidgetVABench by comparing four execution settings that isolate tool access, widget abstraction, and workflow guidance. The comparison is also complemented by a human reference study, two usage scenarios, and an analysis of failure modes. V-A Benchmark Evaluation V-A1 Evaluation Setup Baselines We evaluate three VLM backbones covering proprietary and open-source families (Gemini 3.6 Flash [43], GPT-5.6-Luna [44], and Qwen3-VL-30B-A3B-Instruct [45]). The Claude family is not included because it is used as the judge model for open-ended tasks. We also include a human baseline on a 10% subset with balanced coverage of the different task types. Two participants complete the tasks through the benchmark annotation system, and the system records each participant’s final answer, interaction log, and final application state for evaluation. Agentic Execution Settings We compare four settings that isolate the effect of the widget abstraction framework: • S0S_0 (No Tools): the model receives the task query and the initial view context but cannot invoke tools. • S1S_1 (Direct Tools): the model has access to a set of widget tools through multi-turn interaction. • S2S_2 (Widget Abstraction): the model has access to the same interactive tools, but these tools are organized through the widget-level state–action–perception abstraction described in Section I-B. • S3S_3 (Widget Abstraction + Workflow): we additionally provide workflows to structure the VA analysis process. Environment Setup All experiments were executed using the benchmark’s standard evaluation runner. Each instance was launched as an independent session, preventing interaction state from carrying over across runs. Agents were given a maximum budget of 16 turns for each task. Table I: Experimental results for single- and multi-widget tasks. N/A indicates that Trace and State are not applicable under S0S_0. TK-PK TK-PU TU-PK TU-PU Setting Answer Trace State Answer Trace State Answer Trace State Answer Trace State Model (Avg. A) Single-widget tasks S0S_0 52.5 N/A N/A 56.4 N/A N/A 35.2 N/A N/A 35.9 N/A N/A GPT (45.0) 54.2 N/A N/A 56.7 N/A N/A 37.3 N/A N/A 34.5 N/A N/A Gemini (45.7) 33.3 N/A N/A 31.6 N/A N/A 16.2 N/A N/A 23.0 N/A N/A Qwen (26.0) S1S_1 70.8 68.1 67.2 63.7 49.1 39.0 64.5 58.3 55.5 46.9 37.8 34.3 GPT (61.4) 59.4 62.9 67.2 57.3 44.7 39.8 43.0 55.0 53.3 39.8 31.0 33.3 Gemini (49.9) 36.2 44.9 48.0 37.9 32.7 29.8 21.6 41.1 40.3 27.7 21.1 18.7 Qwen (30.9) S2S_2 78.2 73.6 72.4 76.3 60.5 50.0 65.0 66.9 60.1 51.2 42.8 40.5 GPT (67.7) 62.9 66.7 74.0 67.7 52.2 45.6 46.0 56.4 56.8 46.1 33.3 38.1 Gemini (55.6) 37.4 50.8 56.1 33.2 36.0 32.5 24.7 45.4 42.5 25.5 26.6 24.5 Qwen (30.2) S3S_3 84.3 80.7 73.5 80.7 73.0 62.4 68.8 78.0 69.8 56.6 66.4 63.1 GPT (72.6) 76.2 76.0 73.8 65.3 66.2 59.5 48.8 69.6 69.5 44.3 55.7 53.8 Gemini (58.7) 46.3 67.7 69.6 43.0 50.5 50.8 30.3 59.6 57.2 27.5 44.3 45.1 Qwen (36.8) 83.6 84.8 71.7 79.4 76.1 64.6 69.7 74.3 59.3 66.5 64.4 52.7 Human (74.8) Multi-widget tasks S0S_0 67.8 N/A N/A 61.8 N/A N/A 29.8 N/A N/A 28.6 N/A N/A GPT (47.0) 57.9 N/A N/A 59.9 N/A N/A 20.2 N/A N/A 18.6 N/A N/A Gemini (39.2) 51.3 N/A N/A 42.8 N/A N/A 13.2 N/A N/A 12.5 N/A N/A Qwen (30.0) S1S_1 71.7 55.0 70.4 77.6 25.9 25.7 24.6 50.4 67.8 29.8 4.5 6.0 GPT (51.0) 58.6 46.2 53.3 60.5 15.8 11.8 13.1 44.9 54.7 17.7 1.4 1.3 Gemini (37.6) 54.6 38.8 36.5 46.7 8.1 9.9 9.1 35.3 36.9 9.2 2.4 2.0 Qwen (30.0) S2S_2 88.2 75.3 94.6 78.3 35.6 37.7 33.3 76.5 90.4 32.7 10.3 13.5 GPT (58.1) 59.2 68.2 96.0 64.5 28.5 26.9 22.4 71.5 94.1 25.2 2.4 2.8 Gemini (43.0) 61.8 55.1 75.9 52.0 19.4 24.3 11.9 54.2 75.8 13.8 13.2 20.8 Qwen (35.0) S3S_3 88.8 67.6 90.1 81.6 39.9 40.7 36.9 71.6 88.2 35.3 34.7 36.1 GPT (60.7) 63.8 68.0 93.5 66.4 32.6 36.1 23.2 69.5 89.3 23.2 26.5 29.7 Gemini (44.2) 63.8 53.1 76.1 57.2 20.1 23.6 12.8 52.1 70.4 12.8 18.1 23.2 Qwen (36.8) 76.9 56.2 70.8 91.7 40.0 56.3 26.3 39.2 56.1 33.9 44.9 53.8 Human (57.1) V-A2 Result Analysis Table I reports Answer, Trace, and State scores across models, settings, and Target–Path types. The four settings isolate tool access, widget organization, and workflow guidance, so we read S1S_1–S2S_2 and S2S_2–S3S_3 as interface effects rather than as changes in what the agent can execute. Single-Widget Tasks All three models obtain their strongest overall scores under S3S_3. Moving from No Tools to Direct Tools improves Answer, as expected: many tasks require evidence that is not visible in the initial view, so the agent has to interact before it can form a correct conclusion. The more diagnostic comparison is S1S_1–S2S_2, where the executable tools are held fixed. Widget Abstraction raises Answer for GPT and Gemini and slightly lowers it for Qwen, while Qwen’s Trace and State still rise. An explanation is that the widget abstraction makes the current state, applicable actions, and perception queries easier to select, so Qwen can ground interactions more reliably without yet mapping the acquired evidence onto the required answer. This reading is consistent with the evidence-to-answer errors reported later (Section V-C), which remain more frequent for Qwen than for GPT even under S3S_3. Workflow then improves Trace and State more than Answer. The pattern suggests that reusable procedures mainly help the agent choose and sequence operations when the path is underspecified, whereas synthesizing a correct final answer still depends more on the model capability. Widget abstraction and workflow are therefore complementary: the former organizes what the agent can do in the current view, and the latter organizes what it should do next. Multi-Widget Tasks Multi-widget tasks do not follow the same progression. Direct Tools add little to Answer, and Gemini’s overall Answer even decreases slightly relative to No Tools. A likely reason is that a coordinated workspace exposes many more operations than a single widget, so a flat tool list increases selection difficulty without making cross-view state changes attributable to the originating action. The main step is therefore S1S_1–S2S_2: averaged across models, Answer rises by 5.85.8 points, while Trace and State rise by 15.115.1 and 23.123.1 points. The much larger interaction gains indicate that agents already had the needed operations, but lacked an inspectable account of widget state and of how an action in one view affects another. Workflow adds a smaller further gain, suggesting that once coordination is inspectable, additional path templates help less than they do on single-widget tasks. Where Trace and State already rise sharply, the remaining Answer gap is more consistent with insight selection than with missing step order. Across both scopes, target-unknown tasks remain harder on Answer, because the agent must decide which finding is worth reporting rather than match a specified target. Path-unknown tasks remain harder on Trace and State, because the agent must discover a useful interaction sequence rather than follow a stated procedure. Both difficulties concentrate on multi-widget TU-PU, where the agent has to choose what to look for and how to look across views at the same time. Relative to the Human Baseline On single-widget tasks, agents remain below the human Answer reference, so interface support improves operability without closing the gap in conclusion quality. On multi-widget tasks, GPT under S3S_3 reaches an Answer score of 60.7%, slightly above the human reference of 57.1%. This observation is consistent with the broader pattern reported above: multi-widget tasks show the largest gains from S1S_1 to S2S_2, indicating that structured widget support is particularly useful when analysis requires coordination across multiple views. This difference in evidence-acquisition strategy is also reflected in the human evaluation scores. The human Trace score is lower than the State score because participants often manipulate the visualization and interpret the view directly, rather than invoking the perception queries recorded in the reference trace. In contrast, perception queries account for 41.0%41.0\%, 29.2%29.2\%, and 33.9%33.9\% of successful calls under S3S_3 for GPT, Gemini, and Qwen, respectively. The more explicit use of structured perception may help agents maintain and integrate evidence across coordinated views, particularly in multi-widget analysis. Agreement with Human Ratings To assess the reliability of the LLM-based evaluator for open-ended tasks, we compare Claude Sonnet 4.6 scores with two independent human raters on a sample of 200 responses. Claude achieved ICC(2,1) values of 0.760.76 and 0.730.73 and mean absolute errors (MAEs) of 0.120.12 and 0.140.14, respectively. These results support the reliability of Claude-based scoring for open-ended tasks. More details are reported in Supplementary Sec. S2.3.4. Stability Analysis To assess result stability, we repeated each model under all four settings three times on fixed 10% subsets of both the single- and multi-widget benchmarks. Across the repeated runs, the results exhibited limited variation, with an overall average standard deviation of approximately 2.42.4 points. Complete repeated-run results, including the mean and sample standard deviation for each model, execution setting, and workspace scope, are reported in Supplementary Sec. S3.2. V-B Usage Scenarios To complement the quantitative evaluation, we present two usage scenarios that illustrate how the widget abstraction supports practical visual analysis in the two integration settings described in Section I-D. V-B1 Scenario 1: Operational Risk Analysis This scenario illustrates how users and agents conduct collaborative analysis in our integrated analysis environment. Open-Ended Operational Triage Daniel worked through a service-reliability triage task in which he needed to determine which operational regions needed deeper investigation. The bubble scatterplot encoded latency on the x-axis, error rate on the y-axis, traffic volume by bubble size, and operational region by color. The initial overview contained both a dense central band and several distant outlier groups. Although the chart suggested that operational risk increased toward the upper-right region, the outliers stretched the visual scale and made it difficult to determine whether the dense center contained distinct regional risk profiles. Rather than manually testing a sequence of filters and zoom regions, Daniel switched the Analysis Panel to Agent mode and issued an open-ended request: “Please explore this scatterplot autonomously and summarize risk patterns.” The agent subsequently expressed its exploration through semantic widget actions such as filtering and zooming (R1). The iteration cards exposed the agent’s current action and next analytical sub-goal, allowing Daniel to understand how the exploration was being organized rather than receiving only its final result (Figure 2 1.1–1.2). View Refinement and Verification The agent first applied scatter.filterCategorical to remove Legacy-Core and Factory-IoT, the two most isolated outlier regions in the initial view (Figure 2 2.1). This operation reduced the visual influence of the extreme points and made the dense operating region easier to inspect, while retaining Global-Backbone as a potentially relevant secondary risk group. Based on the filtered state, the agent then applied scatter.zoomDomain to focus on the dense center of the chart (Figure 2 2.2). The Verification panel confirmed that the filtering had been executed successfully and that the resulting state provided a new and deeper perspective for the analysis (Figure 2 2.3). Daniel could inspect the updated views and how they were produced. The Interaction Trace recorded the transition from the initial view to the filtered and zoomed states (Figure 2 D1). Each trace node corresponded to an iteration card in the Analysis Panel and to the resulting state in the Visualization Workspace (Figure 2 D3). Daniel could therefore revisit a step and inspect its plan, executed operation, and visual outcome together, rather than reconstructing the analysis from a disconnected sequence of interface changes (R2). Findings and Follow-Up In the zoomed view, three regional risk tiers became distinguishable: Central-Cloud formed a lower-risk tier, Edge-Store a moderate-risk tier, and Healthcare-Edge a higher-risk tier. The refined view also showed a clear gradient in which higher latency and error rates were associated with increased risk, with larger-traffic operations concentrated toward the higher-risk portion of the chart. The agent summarized these findings in the final response, grounding its claims in the state produced through the preceding filtering and zooming operations (Figure 2 2.4). For Daniel, this transformed the original cluttered overview into a more concrete basis for deciding which service regions and high-traffic operations merited further examination. The analysis did not end with a static summary. Based on the current findings, the system proposed four follow-up analyses: comparing the three identified tiers in terms of latency and traffic, examining Global-Backbone as a secondary risk group, identifying high-traffic outliers, and comparing the previously filtered Legacy-Core and Factory-IoT regions (Figure 2 3.1). Because these suggestions were grounded in the current workspace state, Daniel could continue directly from the refined view without recreating the filters, zoom region, or analytical context. The system thus helped him move from an ambiguous overview to an inspectable set of findings and a reusable operational triage plan. Figure 3: Scenario 2: finding an atmospheric filming window on an existing Vega-Lite page. (A) WidgetVA is connected directly to the existing visualization and receives the user’s filming goal (R3). (B) Cross-widget filtering and temporal brushing identify the relevant atmospheric conditions and progressively narrow the candidate period to September (R1). (C) Comparing early and late September reveals an atmosphere–reliability trade-off: late September provides more fog and drizzle but also substantially more rain. (D) The agent revisits the saved September states and drills down into rainy-day records (R2), verifying the higher disruption risk in late September and supporting Sept. 1–15 as the recommended filming window. V-B2 Scenario 2: Finding an Atmospheric Filming Time This scenario illustrates how the same widget abstraction can be applied to an existing VA page and support state-preserving cross-widget analysis. Connecting WidgetVA and Expressing the Filming Goal Ethan was a photographer preparing an outdoor shoot in Seattle. She wanted to show the city’s characteristic moody and misty atmosphere, but frequent or heavy rain would interrupt filming. Therefore, she needed to find a suitable filming period that preserved the desired atmosphere while avoiding frequent heavy rain. Ethan opened the Seattle Weather Exploration example on the official Vega-Lite website to find the ideal time through visual analysis.22 2 https://vega.github.io/vega-lite/examples/interactive_seattle_weather.html The page contained an upper scatterplot showing date and maximum daily temperature, with precipitation encoded by point size, and a lower bar chart summarizing the number of records for each weather condition. Without exporting the data or reconstructing the visualization, Ethan activated the WidgetVA browser extension on the existing page (R3) (Figure 3 A1–A2). The two linked views were imported into a two-widget workspace. Ethan then sent the question: “We want Seattle’s moody, misty atmosphere for an outdoor shoot, but frequent heavy rain would disrupt filming. Can you help identify a time that satisfies this need?” (Figure 3 A2). Cross-Widget Exploration The agent first translated the requested atmosphere into the relevant weather categories, fog and drizzle, and selected them in the bar chart (R1). The cross-view linkage filtered the scatterplot to the corresponding records, revealing several candidate periods from late summer to fall (Figure 3 B1). Through subsequent temporal brushing and comparison of the weather distributions, the agent progressively narrowed to September (Figure 3 B2). September Trade-Off The agent divided September into two possible time periods and brushed Sept. 1–15 and Sept. 16–30 separately (Figure 3 C1–C2). The linked bar chart showed that the first half had fewer rainy and foggy days. The second half contained more fog and drizzle records, producing a stronger misty appearance. However, rain also increased substantially and became almost as frequent as sun. The comparison therefore isolated the remaining decision: Sept. 1–15 offered more reliable filming conditions, whereas Sept. 16–30 better matched the desired atmosphere but might impose greater disruption (Figure 3 C). Rain-Severity Verification Weather composition alone did not reveal whether the additional rain in late September was likely to cause substantial disruption. The Interaction Trace preserved both September states, allowing the agent to revisit each interval. For each interval, the agent selected rain in the linked bar chart and inspected the remaining rainy-day records in the scatterplot (Figure 3 D1–D2). The Sept. 16–30 state contained more rainy days, several larger precipitation marks, and lower rainy-day temperatures than the Sept. 1–15 state (Figure 3 D1–D2). This evidence suggested that the stronger misty atmosphere in late September came with a higher risk of filming disruption. Based on this additional evidence, Ethan selected Sept. 1–15 as the primary shooting period because it preserved some fog while providing fewer rainy days and better filming conditions. She left Sept. 16–30 as an alternative for scenes requiring a stronger misty atmosphere (Figure 3 D). V-C Failure Modes Analysis Beyond the aggregate scores, both the benchmark evaluation and the usage scenarios reveal several recurring failure patterns. We report important ones here in the hope that they can help guide future research on agentic visual analytics. Evidence-to-answer grounding errors. Although recent VLMs have made notable progress in visual analysis, they still sometimes derive wrong conclusions from the available evidence. Across the S3S_3 condition, this pattern—defined as Answer <0.5<0.5 with both Trace and State scores of at least 0.60.6—occurred in 3.8% of GPT episodes, 13.5% of Gemini episodes, and 16.8% of Qwen episodes. For example, in a constrained-ranking task, GPT correctly filtered the requested categories, sorted the bars, highlighted the leading candidates, and compared the visible groups (Trace = State = 1.0), but answered West when the correct answer was East. In a cohort task that also exhibited planning drift, Qwen correctly brushed the requested region but returned only Male, omitting the required comparison between Male (36 records) and Female (8 records). These examples show that successful interaction does not by itself guarantee that the model selects the correct visible statistic or answers the query correctly. Tool-grounding errors. Tool-grounding errors arise when the model invokes tools, but not the ones that best match the user’s analytical intent. In the S3S_3 condition, 6.5% of GPT episodes, 9.5% of Gemini episodes, and 17.0% of Qwen episodes had Answer, Trace, and State scores all below 0.50.5. For example, in several tasks whose reference trace required brushing a specified price-revenue region, models applied only a categorical filter and then answered from the unbrushed view. The issue is therefore not tool availability, but grounding the semantic request that makes the relevant evidence observable. Planning drift and non-convergent exploration. Planning drift appears when the model remains active in the tool loop but fails to converge on the interaction that actually resolves the task. In one scatter-cohort task under S3S_3, Qwen exhausted the 16-turn interaction budget. It brushed the requested region and then issued 13 repeated perception.summarizeVisible calls, along with view-configuration checks. It ultimately returned only Male, omitting the required comparison between Male: 36 records and Female: 8 records. The trace suggests that stronger evidence-sufficiency and stopping criteria are needed. VI Discussion and Limitations VI-A Discussion After the evaluation, we conducted follow-up interviews with E1–E3 to collect their reflections on WidgetVA, which we report as formative reflections rather than as independent validation. Observability and Provenance. Overall, the experts found our system easy to work with, not merely because it automates parts of the analysis, but because it makes the analytical process visible and inspectable. E2 especially valued the Interaction Trace view, which helped him “understand the exploration logic and how the current state was reached”. E3 also added, “The transparency of the provenance makes the agent easier to follow.” These reflections suggest that transparency is not merely a presentation feature, but an important condition for making mixed-initiative collaboration feel controllable and comfortable. In our framework, agent-issued actions are reflected in the shared widget and workspace states and recorded in the analysis history. Users can therefore connect an intermediate view to the interaction that produced it, revisit previous states, and continue the analysis from an earlier point. Therefore, the analytical process becomes explicit and replayable, making the co-analysis between human and agent more controllable and trustworthy. Beyond Model Capability. Our evaluation results suggest that effective agentic VA depends not only on model capability, but also on how the VA environment exposes interaction state and capabilities to the agent. Across the three evaluated model families, overall Answer performance generally follows the same ordering over the four execution settings. With executable capabilities fixed, Widget Abstraction further improves interaction grounding, while Workflow provides its strongest support when the analytical path is underspecified. The larger gains in Trace and State than in Answer for some models further indicate that interface-side support can improve evidence acquisition before those gains are fully translated into final conclusions. Qualitatively, the usage scenarios and expert reflections further show that effective agentic VA requires interfaces that support observability, intervention, and continuation during analysis. These results imply that, alongside continued advances in model capability, more effort should also be devoted to designing systems that can better utilize those models for VA-assisted analysis. This includes not only stronger agentic system design, but also interface and interaction designs that better involve humans in practice. Diagnostic Benchmarking. As agentic systems continue to evolve rapidly, evaluation should do more than report end-task success. Final accuracy alone provides limited insight into why an agent fails or which component has improved under a new model or scaffold. In contrast, WidgetVABench is designed to provide more diagnostic feedback by evaluating behavior along multiple dimensions, including Answer, Reference Trace Similarity, and State, together with intermediate view states and reference traces. This diagnostic capability makes it possible to localize bottlenecks more precisely, which helps researchers move beyond coarse comparison and more directly target the limitations of autonomous VA agents and their supporting systems. VI-B Limitations and Future Work We also identify several limitations and outline promising extensions to the framework and benchmark. Non-Unique Interaction Traces. In many cases, annotators recorded the interaction trace that an efficient human analyst would naturally take, such as invoking bar.sortBar for a ranking task. However, there can be multiple valid traces for the same analysis intent. To reduce the dependence on a single reference trace, our current Reference Trace Similarity already supports annotated alternative steps and partial parameter matches. In addition, the separate State evaluation checks whether the required semantic conditions are satisfied in the resulting workspace, allowing successful trajectories that differ from the primary reference trace to receive credit. Nevertheless, these mechanisms can only account for alternatives anticipated during benchmark annotation and cannot exhaustively represent all valid analytical strategies or assess their relative quality. Future benchmark designs should expand the coverage of alternative interaction paths and provide multiple valid traces and graded preferences over interaction paths in terms of efficiency, interpretability, or analytical groundedness. This may also serve as a useful foundation for future RL-based optimization. Agent-Side Learning from Interaction Traces. The evaluation still shows remaining agent failures in evidence-to-answer grounding, tool selection, and planning, even under S3S_3. The present work diagnoses these gaps and studies interface-side support, but it does not specialize or train agents for visual analysis. One obstacle is the scarcity of high-quality traces aligned with analytical intent, rather than with pointer events or screenshots. Because human and agent operations already share the same widget state–action–perception contract, ordinary analysis sessions can be recorded as structured traces without a separate logging protocol. Such traces could later be post-processed and annotated, then reused to construct memory and reusable skills or to train agents for VA tasks. For remaining agent-side failures, these traces may be more directly useful than evaluation-oriented reference paths alone. Data and Visualization Scope. Our current framework and benchmark focus on structured tabular data and a controlled set of six registered widget families: bar charts, line charts, scatterplots, heatmaps, parallel coordinates, and Sankey diagrams. In particular, the current benchmark does not cover unstructured or multimodal data, whose use would require additional mechanisms for extracting task-relevant semantics and representing them through agent-operable states, actions, and perceptions. Future work should therefore extend both the widget library and benchmark to broader data representations and visualization forms [46, 47, 48, 49], and investigate whether the current abstraction remains sufficient as their interaction and state semantics become more heterogeneous. VII Conclusion We introduced WidgetVA, a widget-centric framework for agentic VA. By standardizing interactive visual components as structured widgets with unified action and perception interfaces, our framework makes agent interaction more observable and reusable. Built on this abstraction, we construct WidgetVABench, which enables fine-grained evaluation of interaction-centric VA behavior through answers, traces, and final states. Experiments with state-of-the-art VLMs show that our framework provides an effective scaffold for agentic VA, but also reveal substantial remaining challenges in grounding, state management, and multi-step analytical control. We hope this work provides a useful foundation for developing more capable and more collaborative agentic VA systems. References [1] D. Keim, G. Andrienko, J. Fekete, C. Görg, J. Kohlhammer, and G. Melançon (2008) Visual analytics: definition, process, and challenges. In Information visualization: Human-centered issues and perspectives, p. 154–175. Cited by: §I. [2] M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu (2017) Towards better analysis of deep convolutional neural networks. IEEE transactions on visualization and computer graphics 23 (1), p. 91–100. Cited by: §I. [3] S. Liu, W. Yang, J. Wang, and J. Yuan (2025) Visualization for artificial intelligence. Springer Nature Switzerland. External Links: ISBN 9783031753404, ISSN 2159-5178, Link, Document Cited by: §I. [4] J. Yuan, C. Chen, W. Yang, M. Liu, J. Xia, and S. Liu (2021) A survey of visual analytics techniques for machine learning. Computational Visual Media 7 (1), p. 3–36. External Links: Link, Document, ISSN 2096-0433 Cited by: §I. [5] W. Yang, M. Liu, Z. Wang, and S. Liu (2024) Foundation models meet visualizations: challenges and opportunities. Computational Visual Media 10 (3), p. 399–424. External Links: ISSN 2096-0662, Link, Document Cited by: §I. [6] Y. Zhao, Y. Zhang, Y. Zhang, X. Zhao, J. Wang, Z. Shao, C. Turkay, and S. Chen (2025) LEVA: using large language models to enhance visual analytics. IEEE Transactions on Visualization and Computer Graphics 31 (3), p. 1830–1847. External Links: Document Cited by: §I, §I-A. [7] Y. Zhao, J. Wang, L. Xiang, X. Zhang, Z. Guo, C. Turkay, Y. Zhang, and S. Chen (2025) LightVA: lightweight visual analytics with LLM agent-based task planning and execution. IEEE Transactions on Visualization and Computer Graphics 31 (9), p. 6162–6177. External Links: Document Cited by: §I. [8] Y. Zhao, X. Shu, L. Fan, L. Gao, Y. Zhang, and S. Chen (2026) ProactiveVA: proactive visual analytics with LLM-based UI agent. IEEE Transactions on Visualization and Computer Graphics 32 (1), p. 451–461. External Links: Document Cited by: §I, §I-A. [9] F. Wang, B. Wang, X. Shu, Z. Liu, Z. Shao, C. Liu, and S. Chen (2025) ChartInsighter: an approach for mitigating hallucination in time-series chart summary generation with a benchmark dataset. IEEE Transactions on Visualization and Computer Graphics 31 (6), p. 3733–3745. External Links: Document Cited by: §I. [10] S. Amershi, D. Weld, M. Vorvoreanu, A. Fourney, B. Nushi, P. Shea, J. Ju, and E. Horvitz (2019) Guidelines for human-ai interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, New York, NY, USA, p. 1–13. External Links: Document Cited by: §I. [11] K. Cheng, Q. Sun, Y. Chu, F. Xu, Y. Li, J. Zhang, and Z. Wu (2024) SeeClick: harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 9313–9332. External Links: Document, Link Cited by: §I, §I-A. [12] R. Niu, J. Li, S. Wang, Y. Fu, X. Hu, X. Leng, H. Kong, Y. Chang, and Q. Wang (2024) ScreenAgent: a vision language model-driven computer control agent. External Links: 2402.07945, Document Cited by: §I, §I-A. [13] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu (2024) OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, p. 52040–52094. External Links: Link, Document Cited by: §I, §I-A. [14] D. Ceneda, T. Gschwandtner, T. May, S. Miksch, H. Schulz, M. Streit, and C. Tominski (2017) Characterizing guidance in visual analytics. IEEE Transactions on Visualization and Computer Graphics 23 (1), p. 111–120. External Links: Document Cited by: §I, §IV-B. [15] M. Hearst and M. Tory (2019) Would you like a chart with that? incorporating visualizations into conversational interfaces. In 2019 IEEE Visualization Conference (VIS), p. 36–40. External Links: Document Cited by: §I-A. [16] L. Weng, X. Wang, J. Lu, Y. Feng, Y. Liu, H. Feng, D. Huang, and W. Chen (2025) InsightLens: augmenting LLM-powered data analysis with interactive insight management and navigation. IEEE Transactions on Visualization and Computer Graphics 31 (6), p. 3719–3732. External Links: Document Cited by: §I-A. [17] D. Bromley and V. Setlur (2024) Dash: a bimodal data exploration tool for interactive text and visualizations. In Proceedings of the 2024 IEEE Visualization and Visual Analytics (VIS), St. Pete Beach, FL, USA, p. 256–260. External Links: Document Cited by: §I-A. [18] A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer (2017) Vega-lite: a grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics 23 (1), p. 341–350. External Links: Document Cited by: §I-A. [19] Z. Xu and E. Wall (2024) Exploring the capability of LLMs in performing low-level visual analytic tasks on SVG data visualizations. In 2024 IEEE Visualization and Visual Analytics Conference (VIS), p. 126–130. External Links: Document Cited by: §I-B. [20] A. Kartha, A. Masry, M. S. Islam, T. Lang, S. Rahman, R. Mahbub, M. Rahman, M. Ahmed, M. R. Parvez, E. Hoque, and S. Joty (2026) DashboardQA: benchmarking multimodal agents for question answering on interactive dashboards. In Findings of the Association for Computational Linguistics: EACL 2026, p. 3385–3407. External Links: Document, Link Cited by: §I-B. [21] G. Wu, C. Xu, E. Wu, Z. Yao, and Y. He (2026) VizAgentBench: benchmarking multimodal agent reasoning on coordinated multi-view visual analytics tasks. Note: OpenReview (ICLR 2026 submission) External Links: Link Cited by: §I-B. [22] R. Amar, J. Eagan, and J. Stasko (2005) Low-level components of analytic activity in information visualization. In Proceedings of the 2005 IEEE Symposium on Information Visualization, p. 111–117. External Links: Document Cited by: §I-C, §I-C. [23] M. Brehmer and T. Munzner (2013) A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics 19 (12), p. 2376–2385. External Links: Document Cited by: §I-C, §I-C, §IV-B. [24] J. S. Yi, Y. Kang, J. Stasko, and J. A. Jacko (2007) Toward a deeper understanding of the role of interaction in information visualization. IEEE Transactions on Visualization and Computer Graphics 13 (6), p. 1224–1231. External Links: Document Cited by: §I-C, §I-C. [25] J. Heer and B. Shneiderman (2012) Interactive dynamics for visual analysis. Communications of the ACM 55 (4), p. 45–54. External Links: Document Cited by: §I-C. [26] D. Gotz and M. X. Zhou (2009) Characterizing users’ visual analytic activity for insight provenance. Information Visualization 8 (1), p. 42–55. External Links: Document Cited by: §I-C. [27] S. Gathani, S. Monadjemi, A. Ottley, and L. Battle (2022) A grammar-based approach for applying visualization taxonomies to interaction logs. Computer Graphics Forum 41 (3), p. 489–500. External Links: Document Cited by: §I-C. [28] K. Gadhave, Z. Cutler, and A. Lex (2022) Reusing interactive analysis workflows. Computer Graphics Forum 41 (3), p. 133–144. External Links: Document Cited by: §I-C. [29] N. Andrienko, G. Andrienko, J. Bernard, and M. Sedlmair (2026) ATWL: a formal language for representing, comparing, and reusing visual analytics workflows. External Links: 2605.25489, Document Cited by: §I-C. [30] M. Q. W. Baldonado, A. Woodruff, and A. Kuchinsky (2000) Guidelines for using multiple views in information visualization. In Proceedings of the Working Conference on Advanced Visual Interfaces, p. 110–119. External Links: Document Cited by: §I-C. [31] J. C. Roberts (2007) State of the art: coordinated and multiple views in exploratory visualization. In Fifth International Conference on Coordinated and Multiple Views in Exploratory Visualization, p. 61–71. External Links: Document Cited by: §I-C. [32] W. Yang, Z. Li, M. Liu, Y. Lu, K. Cao, R. Maciejewski, and S. Liu (2020) Diagnosing concept drift with visual analytics. In 2020 IEEE Conference on Visual Analytics Science and Technology (VAST), p. 12–23. External Links: Document, Link Cited by: §I-A. [33] W. Yang, X. Wang, J. Lu, W. Dou, and S. Liu (2021) Interactive steering of hierarchical clustering. IEEE Transactions on Visualization and Computer Graphics 27 (10), p. 3953–3967. External Links: Document, Link, ISSN 2160-9306 Cited by: §I-A. [34] S. K. Card, J. Mackinlay, and B. Shneiderman (1999) Readings in information visualization: using vision to think. Morgan Kaufmann. Cited by: §I-B1. [35] A. Masry, M. Islam, M. Ahmed, A. Bajaj, F. Kabir, A. Kartha, M. Laskar, M. Rahman, S. Rahman, M. Shahmohammadi, M. Thakkar, M. Parvez, E. Hoque, and S. Joty (2025) ChartQAPRo: a more diverse and challenging benchmark for chart question answering. In Findings of the Association for Computational Linguistics: ACL 2025, p. 19123–19151. External Links: Link, Document Cited by: §IV-D. [36] Z. Wang, M. Xia, L. He, H. Chen, Y. Liu, R. Zhu, K. Liang, X. Wu, H. Liu, S. Malladi, A. Chevalier, S. Arora, and D. Chen (2024) CharXiv: charting gaps in realistic chart understanding in multimodal llms. In Advances in Neural Information Processing Systems, p. 113569–113697. External Links: Link, Document Cited by: §IV-D. [37] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao (2024) MathVista: evaluating mathematical reasoning of foundation models in visual contexts. External Links: 2310.02255, Link Cited by: §IV-D. [38] Anthropic (2026) Introducing claude sonnet 4.6. Note: https://w.anthropic.com/news/claude-sonnet-4-6Accessed: 2026-08-19 Cited by: §IV-D. [39] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque (2022) ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, p. 2263–2279. External Links: Document Cited by: §IV-D. [40] S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi (2023) FActScore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 12076–12100. External Links: Document Cited by: §IV-D. [41] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 2511–2522. External Links: Document Cited by: §IV-D. [42] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, Vol. 36, p. 46595–46623. Cited by: §IV-D. [43] Google (2026) Gemini 3.6 flash. Note: https://ai.google.dev/gemini-api/docs/models/gemini-3.6-flashOfficial model documentation; accessed: 2026-08-19 Cited by: §V-A1. [44] OpenAI (2026) GPT-5.6 luna. Note: https://developers.openai.com/api/docs/models/gpt-5.6-lunaOfficial model documentation; accessed: 2026-08-19 Cited by: §V-A1. [45] Qwen Team (2025) Qwen3-vl-30b-a3b-instruct. Note: https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-InstructOfficial model card; accessed: 2026-08-19 Cited by: §V-A1. [46] D. Li, J. Yuan, X. Guo, X. Wang, Y. Liu, W. Yang, and S. Liu (2026) NCP: neighborhood-preserving non-uniform circle packing method for visualization. Computational Visual Media. Note: to appear Cited by: §VI-B. [47] Y. Zhou, W. Yang, J. Chen, C. Chen, Z. Shen, X. Luo, L. Yu, and S. Liu (2024) Cluster-aware grid layout. IEEE Transactions on Visualization and Computer Graphics 30 (1), p. 240–250. External Links: Document, ISSN 2160-9306, Link Cited by: §VI-B. [48] Y. Zhou, C. Chen, Z. Shen, J. Zhu, J. Chen, W. Yang, and S. Liu (2025) Hierarchical fuzzy-cluster-aware grid layout for large-scale data. IEEE Transactions on Visualization and Computer Graphics 31 (10), p. 8200–8213. External Links: Document, ISSN 2160-9306, Link Cited by: §VI-B. [49] Z. Li, W. Yang, J. Yuan, J. Wu, C. Chen, Y. Ming, F. Yang, H. Zhang, and S. Liu (2025) RuleExplorer: a scalable matrix visualization for understanding tree ensemble classifiers. IEEE Transactions on Visualization and Computer Graphics 31 (9), p. 6370–6384. External Links: ISSN 2160-9306, Link, Document Cited by: §VI-B.

Instant research discovery