Paper deep dive
AnnoBench: A Benchmark for Visualization Annotation Generation
Md Rahat-uz-Zaman, Md Dilshadur Rahman, Andrew McNutt, Paul Rosen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Annotation is among the most demanding visualization tasks to automate, as it simultaneously requires correctly navigating visual, semantic, and stylistic constraints. Failure to meet any of these conditions severely undermines the utility of an annotation, rendering it challenging to read, inaccurate, or visually discordant. Despite a growing body of annotation tools and automations, no existing benchmark or evaluation framework tests whether these conditions are met because of their scope and annotation not being the focus of their studies. We introduce AnnoBench, a benchmark for visualization annotation that materializes the inherent challenges of this domain in a structured and testable manner. AnnoBench pairs visualizations from professional data journalism and visualization galleries with annotation tasks, spanning four representation formats, five chart description conditions, and two prompt specification levels. The benchmark is executed via VLM-as-a-judge, using models aligned with manual human assessment. We evaluate the benchmark via four one-factor-at-a-time experiments, exploring the effects of input representation, semantic context, and prompt specificity, and model selection on annotation quality. This work provides a foundation for advancing annotation automation, tooling, and visualization-generation pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2607.25911v1
- Canonical: https://arxiv.org/abs/2607.25911v1
Trouble viewing inline? Open PDF directly â
Full Text
90,722 characters extracted from source content.
Expand or collapse full text
Md Rahat-uz-Zaman is with the University of Utah. E-mail: rahatzamancse@gmail.com. Md Dilshadur Rahman is with the University of Utah. E-mail: dilshadur@sci.utah.edu. Andrew McNutt is with the University of Utah. E-mail: andrew.mcnutt@utah.edu. Paul Rosen is with the University of Utah. E-mail: paul.rosen@utah.edu. AnnoBench: A Benchmark for Visualization Annotation Generation Rahat-uz-Zaman0000-0001-6728-7569 * Dilshadur Rahman0009-0008-5467-615X * McNutt0000-0001-8255-4258 and Rosen0000-0002-0873-9518 * equal contribution Abstract Annotation is among the most demanding visualization tasks to automate, as it simultaneously requires correctly navigating visual, semantic, and stylistic constraints. Failure to meet any of these conditions severely undermines the utility of an annotation, rendering it challenging to read, inaccurate, or visually discordant. Despite a growing body of annotation tools and automations, no existing benchmark or evaluation framework tests whether these conditions are met because of their scope and annotation not being the focus of their studies. We introduce AnnoBench, a benchmark for visualization annotation that materializes the inherent challenges of this domain in a structured and testable manner. AnnoBench pairs visualizations from professional data journalism and visualization galleries with annotation tasks, spanning four representation formats, five chart description conditions, and two prompt specification levels. The benchmark is executed via VLM-as-a-judge, using models aligned with manual human assessment. We evaluate the benchmark via four one-factor-at-a-time experiments, exploring the effects of input representation, semantic context, and prompt specificity, and model selection on annotation quality. This work provides a foundation for advancing annotation automation, tooling, and visualization-generation pipelines. keywords: Annotations, Visualizations, Benchmark study, Large Language Model We introduce a three-dimensional benchmark for evaluating chart annotation in large language models (A), along with a five-dimensional evaluation framework to assess annotation quality. Using AnnoBench, we reveal systematic failure modes in modern models, including incorrect targeting, data distortion, and avoidance of certain annotation types (middle column in (B)). To support reproducible experimentation and comparison, we provide an interactive web-based interface for configuring, running, and analyzing benchmark trials (C). 1 Introduction Annotations are among the most critical components of a visualization [43], guiding readers through content via careful placement and contextualization that must be both graphically and semantically appropriate. For these reasons, annotation is among the most demanding visualization tasks to automate. An annotation must target the correct visual element [27, 50], assert only what the underlying encoding supports [43, 20], and fit within the visualizationâs existing layout without occluding evidence [24]. These conditions compound in ways that make failures hard to anticipate. For example, an annotation labeling a bar as âpeak revenueâ is incorrect if that bar encodes the second-highest value, even if the annotation text is well-phrased and clearly positioned. Conversely, a factually correct annotation may have no viable placement in a densely labeled region without occluding the data it references. Annotations are visually anchored to the elements they describe, and so such errors can be difficult to detect [30, 14] and can shift reader conclusions in otherwise accurate visualizations [55]. A range of tools have taken up this challenge, including interactive environments [50], semi-automatic insight externalization systems [9], automated generation pipelines [20, 15], and specification grammars [48, 10]. Prior work has also characterized annotation forms and targets [50, 45, 27], the communicative purposes annotations serve [45, 20, 47], and their measurable effects on reader comprehension and bias perception [54, 55, 39, 13, 14]. Yet across both the tools and the research literature, evaluable conditions for annotation correctness remain undefined [43, 53]. Without such conditions, annotation quality can only be judged informally, and failures cannot be systematically identified or compared across systems. Existing benchmarks for chart-related tasks evaluate chart comprehension [28, 4, 18], text answers [33, 35], natural-language descriptions [56, 22], or chart specifications [8, 49], but none considers annotation quality. These problems further compound as we increasingly draw on generative AI models for visualization construction support [29, 58]. We address these gaps with an evaluation framework, discussed in section 3, that operationalizes three necessary conditions for annotation correctnessâtarget correctness, task accuracy, and spatial feasibilityâalongside rubric dimensions for stylistic coherence and chart preservation. We instantiate this framework as AnnoBench, discussed in section 4 and section 5, a benchmark that evaluates Large Language Models (LLMs) and Vision Language Models (VLMs) under controlled variation in chart representation, semantic description, task specification, and model class. This design enables us to analyze how access to chart structure, contextual support, and prompt specificity affects annotation quality across different model types. We evaluate AnnoBench through a set of controlled experiments, reported in section 6. We make three contributions: âą Evaluation framework: An evaluation framework for chart annotation that distinguishes per-instance correctness criteria from broader rubric dimensions. We operationalize this framework as a five-dimensional rubric consisting of target correctness, task accuracy, spatial feasibility, stylistic coherence, and chart preservation, enabling comparable evaluation of annotations produced by visualization tools, grammars, and generation pipelines. âą AnnoBench dataset: A benchmark dataset of about 342 charts drawn from four source collections, with over 650 annotation tasks defined at both intent and execution levels. The dataset spans six formats across four representation families (D3, Vega, Vega-Lite, ggplot2, SVG, and raster) and five chart description conditions: no description, construction and encoding, statistical and relational, perceptual and cognitive, and contextual and domain-specific. This design enables controlled analysis of how representation, semantic context, and task specification affect annotation performance. âą Benchmark system: An end-to-end benchmarking pipeline for running annotation evaluations across built-in and custom models, with format-aware rendering, automated VLM-as-a-judge scoring, and human evaluation support. This system provides reusable infrastructure for applying the benchmark to future annotation methods under a consistent evaluation setup. With this work, we establish chart annotation as a more explicit and testable target for evaluation, enabling more rigorous analysis of automated systems, informing the development of future annotation methods, tools, and evaluation practice, and ultimately helping improve the quality of annotations more generally. Our source code is available at anonymous.4open.science/r/AnnoBench â , and an interactive browser for the benchmark is available at tinyurl.com/annobench â . 2 Related Work We situate AnnoBench at the intersection of two bodies of work: visualization research on the structure and effects of annotations, and benchmark evaluation of LLMs and VLMs on visualization tasks. 2.1 Annotations in Visualization Research on annotations addresses three aspects relevant to AnnoBench: how annotations are structured, how they are formalized, and how their design affects readers. Annotation structure and formalization. Annotations are chart-bound additions that attach text or graphical cues to specific chart elements to explain, emphasize, compare, or contextualize data features [27, 45, 46]. Two structural dimensions organize this spaceâannotation forms and annotation targets. Form refers to the graphical vehicle of the annotation, such as text labels, connectors, enclosures, and highlights. Target refers to the chart element the annotation addresses, such as individual data items, regions, or broader chart structures [27, 50]. Several systems and grammars make these dimensions explicit. For instance, ChartAccent organizes annotation design around paired form and target choices [50]. Contextifier distinguishes annotations that restate visible patterns from those that introduce external context and models anchoring at multiple levels of the visualization [20]. Recent grammar-based work treats annotation semantics, targets, and placement as first-class components of visualization specifications [48, 10]. In practice, a single form is often insufficient to unambiguously identify its target. Professional graphics, therefore, frequently combine forms into ensembles such as connector+text and enclosure+connector+text, with cue selection varying by task and display context [45, 46, 25]. Observed practice suggests that form-target choices remain dependent on task, display density, and communicative purpose [45, 46, 25]. Effects on readers. Annotation design shapes how information is attached to a chart and, by extension, what readers notice, how they navigate the display, and what they take away. Whether an annotation restates a pattern visible in the chart or introduces external context constrains both the appropriate form and the claims it can legitimately make [20, 45]. Presentation choices more broadly influence memorability, comprehension, and recall [6, 5, 3], and within annotated views, specifically, placement, linking, and layout affect gaze behavior, comprehension of referenced evidence, and cognitive load [7, 39, 63, 2]. These effects are design-dependent. Augmentation choices vary by form, and cue effectiveness changes across presentation settings [11, 17, 26]. The same mechanisms can also distort interpretation. Text amount, framing, and wording shift reader takeaways and perceived bias even when the underlying chart is accurate [55, 54, 13, 14, 30]. Because these effects are both design-dependent and consequential, annotation errors are not merely stylistic. Incorrect placement, misleading phrasing, or semantically inappropriate context can change what readers conclude from an otherwise accurate chart. Prior work provides a vocabulary for describing annotations, formal tools for specifying them, and evidence that annotation design shapes reader outcomes. But it does not establish whether a particular annotation is correct for a given chart and task. Doing so requires verifiable per-instance conditions: whether an annotation targets the intended visual element, asserts only what the encoding supports, and can be placed without occluding evidence. Design guidelines, reader outcome measures, and editorial conventions operate at a broader level and therefore do not provide such conditions. Rahman et al. [43] and Spivak & Tory [53] identify this absence as an open problem. In section 3, we address it by defining per-instance correctness criteria and operationalizing them in a structured evaluation framework. 2.2 LLM and VLM Benchmarks for Visualization A growing body of benchmarks evaluates LLMs and VLMs on chart-related tasks, spanning chart comprehension, natural language description, and chart generation and editing. Chart comprehension benchmarks. Chart comprehension benchmarks take the form of question answering: a model receives a chart and a natural language question, then is evaluated on whether it can produce a short, correct answer. Chart QA benchmarks cover numerical, relational, encoding-level, and multi-chart reasoning [35, 33, 21, 36, 64]. Visualization literacy instruments assess chart reading through controlled item sets with known answers [28, 40, 16], and recent work uses them to probe VLM generalization under systematic chart modifications [18, 41]. These benchmarks surface one capability central to chart annotation: precise identification and reading of chart elements. Evaluations report systematic difficulty in locating and reading particular values or marks [4, 36], a prerequisite for annotation placement, even though these benchmarks do not evaluate chart modification itself. Chart captioning and generation benchmarks. Captioning benchmarks evaluate chart-level natural language descriptions. For instance, VisText represents each chart as a rasterized image, a backing data table, and a scene graph, and studies how representation type and semantic level affect captioning quality [56]. Chart-to-text benchmarks address chart summarization, and Lundgard and Satyanarayan [31] show that what counts as an adequate description depends on the reader and the task. NL2VIS benchmarks address chart construction from natural language [32, 8, 49]. Chart editing benchmarks instead require modifying an existing chart under textual or multimodal instructions [62, 60, 65]. Across these benchmarks, the output is a description, a specification, or a modified chart. Annotation introduces a more specific evaluation target: a chart-bound addition that must select the intended referent, state a supported claim, and fit within the chart layout. Prior benchmark work suggests three dimensions that are especially relevant to annotation evaluation: chart representation, semantic context, and task specification. Annotation depends on all three simultaneously because it requires a system to identify the intended referent, determine what the chart supports, and incorporate the requested addition into the visualization. section 3 develops these dimensions as the basis for the benchmark design. 3 Benchmark Design and Evaluation Framework Figure 1: Overview of the AnnoBench dataset (top), execution pipeline (middle), and evaluation criteria (bottom). The top row illustrates the dataset design, where each chart has multiple representations (raster, vector, grammar, and code) and is paired with multi-level semantic captions ranging from no description to domain-specific context. Each chart is also associated with multiple annotation tasks defined at both the intent and execution levels. The middle row illustrates AnnoBenchâs automatic prompt construction, running with the constructed prompt and providing environments for human and VLM-as-Judge evaluation. The bottom row shows our five-dimensional evaluation framework, with a failing example in each criterion. section 2 identifies both the factors a benchmark must control to support valid annotation experiments and the properties that matter for evaluating whether a generated annotation is correct. We organize AnnoBench around these two observations: benchmark design dimensions that control the conditions under which annotation is attempted (subsection 3.1), and an evaluation framework that defines what a correct annotation must satisfy under those conditions (subsection 3.2). 3.1 Dataset Design Dimensions Prior benchmark work discussed in subsection 2.2 on chart understanding, captioning, and editing points to three factors that are especially consequential for annotation generation: chart representation, semantic context, and task specification. These factors correspond to three capabilities annotation requires simultaneouslyâidentifying the intended target, determining what the chart supports, and integrating the requested addition into the visualization. AnnoBench treats each as an independent design dimension, so that failures can be attributed to the specific capability they stress: chart reading, semantic grounding, or annotation execution. Multiple Chart Representations. Annotations, being a supplementary design element, are inherently bound to the form of the input they modify. Different representations change how difficult each of these steps becomes. Raster inputs preserve rendered appearance and support visual reasoning, but they do not expose chart components as directly editable elements. Marks, axes, and labels must instead be inferred from appearance, and any modifications must be expressed at the pixel level, which may be beneficial or detrimental to models. Structured representations, such as vector markup, grammar-based specifications, and imperative code, expose chart components more explicitly and support direct manipulation, but require the model to reason over abstract structure rather than rendered appearance. A model may therefore succeed under one representation and fail under another for reasons that are separate from its underlying annotation performance. This distinction matters for evaluation. A failure on a raster input may reflect weak visual localization, difficulty producing pixel-level modifications, or both. A failure on a structured representation may instead reflect difficulty parsing the format or producing well-formed edits, even when the required annotation is understood. AnnoBench, therefore, treats chart representation as a controlled design dimension allowing the benchmark to distinguish failures in target identification, structural reasoning, and editing, and to attribute those failures more clearly to the criteria defined in subsection 3.2. Task Specification. The second axis of variation in AnnoBench is how the annotation goal is specified. The same objective can be expressed at different levels of detailâfrom a loose communicative intent to a precise specification of what to add, where to place it, and how it should connect to chart elements. Because prompt wording directly determines how much freedom a model has in deciding what to annotate and how, leaving this variation uncontrolled conflates failures of annotation strategy with failures of instruction following. AnnoBench makes it an explicit variable instead. A task pairs a chart input with an annotation objective: the model receives one or more chart representations and is asked to produce the chart with the specified annotation integrated. A single chart may be paired with multiple tasks, each targeting a different annotation objective. Each task is defined at two prompt levels: intent level prompts describe the annotation goal without specifying its form or placement; execution level prompts specify what to add, where to place it, and how it should relate to chart elements. This distinction separates two failure modes. A model that fails at the intent level cannot map a communicative goal to an appropriate annotation strategy. A model that succeeds only under execution-level prompts can follow explicit instructions but cannot reason about annotation strategy independently. Failures at both levels point to deeper problemsâweak chart understanding, incorrect target identification, or unreliable placement decisions. We draw tasks from the annotation taxonomy of Rahman et al. [44], which characterizes annotations along three dimensions: why a given annotation type is used, what data it targets, and how it connects to chart elements and other annotations. Grounding tasks in this taxonomy connects AnnoBench to annotation practices observed in real-world visualizations rather than an ad hoc selection of task types. The specific task distribution and taxonomy coverage are described in section 4. Multi-level Semantic Descriptions. How much a model knows about a chart directly shapes its annotation output [56]. Task accuracy depends not only on whether a model can read the chart visually, but on whether it understands what the chart is about. A model given only a chart must infer the chartâs subject, domain, and key features from visual appearance alone. However, a model given a rich textual description (such as encoding details about axis, marks, and channels, and other relevant context like trends and perceptual details) has that context explicitly available. Therefore, a failure in task accuracy might be mitigated by additional information, such as a caption or external domain information regarding the chart. Hence, we use the chart description semantics as a dimension of annotation capability. Each chart in AnnoBench is paired with multiple levels of semantic detail, applied independently of the selected representation type. We follow the semantic hierarchy of Lundgard et al. [31]: level 1 covers chart type, encodings, and axis ranges; level 2 adds descriptive statistics and point-wise comparisons; level 3 adds perceptual patterns and trend synthesis; and level 4 adds domain-specific interpretation. The Lundgard framework was designed for accessibility, in which text entirely substitutes for the chart; here, descriptions augment the chart input rather than replace it. Therefore, we introduce level 0 (No Description), in which we omit any additional context and serve the chart as a no-context baseline. All five levels are summarized in Figure 1 along with an example. These design dimensions define the conditions under which annotation is attempted. To evaluate model outputs under those conditions, we next define what a correct annotation must satisfy. 3.2 Annotation Evaluation Framework Evaluating annotation correctness requires per-instance conditions that can be checked directly: whether the annotation targets the intended visual element, asserts only what the underlying encoding supports, and fits within the chart layout without occluding evidence. As identified in subsection 2.1, prior work does not define such conditions, leaving annotation quality without a basis for systematic comparison across systems. Because annotation tasks often admit multiple valid realizations [45, 46], evaluating against a single reference output is also insufficient. A single aggregate score would conflate two fundamentally different failure types: an annotation that targets the wrong element, and one that is correct but styled differently than the reference. We developed the framework by examining what prior work on annotation systems [50, 20], chart overlays [27], annotation grammars [48, 10], and annotation practice [43, 45] implies about correctness. That work characterizes annotation through target binding, claim scope, and spatial placement, but as design dimensions rather than testable conditions. Through iterative discussion among the authors, we examined how each property applies when the question shifts from design to evaluation, yielding a set of criteria that apply consistently across annotation types, representations, and authoring contexts. Target Correctness (Precise). An annotation makes a claim about a specific visual element or the chart as a whole, and that claim is only correct if the annotation is attached to the intended element [27, 45]. For target-bound annotations, the annotation must refer to the correct mark, series, or region. Protocols that check only annotation text without verifying attachment cannot detect this class of error. In the failing example in Figure 1, the task asks for the smallest bar to be labeled âSmallest (<value>)â. The annotation produces the correct label text but attaches it to one of the taller bars instead. The content is plausible, but the target is wrong. Chart-level annotations do not refer to a specific visual target. For these, target correctness is satisfied by construction, and evaluation shifts to task accuracy. We operationalize this criterion as the rubric dimension precise. Task Accuracy (Relevant). Correct target binding does not guarantee a correct annotation: an annotation may be attached to the intended element while still making an inaccurate or irrelevant claim about it. The relevant failing example in Figure 1 shows this clearly: the label is attached to the correct smallest bar, but the text is wrongââSmallest (80)â instead of âSmallest (22)â. The target is correct, but the claim is not. Task accuracy asks whether the annotation makes an accurate and relevant claim about its target. For target-bound annotations, the claim must be accurate with respect to the specific mark, series, or region. For chart-level annotations, it must be consistent with the visualization as a whole. We evaluate this criterion independently of communicative properties such as phrasing and framing, which remain preference-sensitive. We use relevant as the corresponding rubric dimension. Spatial Feasibility (Visible). An annotation must fit within the chart without obscuring evidence or creating ambiguity about what it refers to. It fails this condition if it overlaps axis labels, data marks, or other annotations, or if it is placed far enough from its target that the connection becomes unclear [27]. As shown in the visible example in Figure 1, the label âSmallestâ is placed outside the chart area and connected to the correct bar only by a long line, making the association unclear. The annotation is neither occluding evidence nor incorrectly labeledâit simply cannot be unambiguously read as belonging to its target. Chart-level annotations face a weaker version of this condition: they must be placeable without obscuring evidence, but do not require proximity to a specific mark. We define this criterion as visible in the evaluation rubric. Chart Preservation (Preservant). The underlying chart must remain faithful to its original encoding after the annotation is applied. Any automated system that generates or modifies a chart representation as part of an annotationâwhether through code generation, SVG editing, or raster regenerationârisks inadvertently altering data encodings, axis scales, or mark positions in the process. For LLM-based systems specifically, this risk is compounded by the tendency of generative models to produce plausible-looking but factually incorrect content [19], which can potentially manifest as modified bar heights, shifted axis ranges, or reordered categories in the output chart. The preservant failing example in Figure 1 captures this well: the label âSmallest (22)â targets the correct bar with the correct value, but the bar heights in the output chart have shifted relative to the original, changing the data the visualization encodes. The annotation itself is correct; the chart it was added to is not. Chart preservation evaluates whether the annotated output maintains the original visualizationâs data, structure, and visual mappings independent of the added annotation. We operationalize this as preservant in the evaluation rubric. Stylistic Coherence (Consistent). An annotation should remain visually coherent with the chart it augments, including its typography, color use, and overall visual style [50]. Consider the consistent failing example in Figure 1: the label âSmallest (22)â targets the correct bar and states the correct value, but is rendered in a large, bold style that clashes with the chartâs minimal design. The annotation is factually correct and spatially feasible, yet visually discordant. Stylistic coherence is preference-sensitive: multiple realizations may be valid, so it is evaluated holistically rather than against a single reference. We use consistent as the corresponding rubric dimension. For each generated output, the evaluator receives the input chart, the model output, and the task description. The five-dimensional rubric supports both manual human evaluation and automated VLM-as-a-Judge scoring, as described in Sect. 6. We further suggest scoring a gradesheet for each rubric item, which can be found in annobench/rubrics â . 4 AnnoBench Benchmark Dataset We now describe the construction of AnnoBench dataset. The dataset draws from two complementary sources: professionally produced charts from data journalism and charts collected from established visualization library galleries. This combination enables us to capture both the complexity and editorial nuance of real-world visualizations, as well as the controllability and reproducibility of library-based examples. For each chart included in the benchmark, we construct instances that vary along the benchmark dimensions introduced in section 3, including chart representation, description condition, and task specification. After constructing each dataset, we conducted an iterative review process to ensure data quality and consistency, shown in Figure 2. Figure 2: Overview of the AnnoBench datasetâs iterative review process. Dataset items are constructed from online visualization galleries and journalism websites, then reviewed by two co-authors across chart representations, prompt levels, and multi-level chart descriptions. Finally, the same co-authors sampled benchmark items in the AnnoBench Browser and verified the final dataset entries. Dataset # Charts # Tasks Chart Types Professional Charts â 58 58 Bar (25), Line (26), Scatter (9) Vega Gallery â 92 214 Scatter (13), Heatmap (9), Line (9), Bar (9), Others (61) Vega-Lite Gallery â 192 394 Line (27), Bar (24), Distribution (18), Scatter (13), Maps (12), Others (97) Table 1: AnnoBench dataset combines professional charts with library-based examples, providing diverse chart types, multiple representations, varying description levels, and task prompt variants. 4.1 Professional Charts Our first dataset is a corpus of charts drawn from professional data journalism [46]. We organize the corresponding annotation tasks using the taxonomy introduced by Rahman et al. [45]. The corpus draws from four newsroom sources with distinct visual and editorial conventions: 12 charts from The Washington Post, 18 from The New York Times, 15 from The Wall Street Journal, and 15 from The Economist. Because these visualizations were designed and published by professional teams at major news organizations, they reflect characteristics that make annotation difficult in practice, including editorial styling, custom typography, non-standard layouts, and dense labeling. Unlike the standardized outputs of popular visualization libraries, they also emerge from heterogeneous workflows involving tools such as Excel, Illustrator, Photoshop, proprietary newsroom pipelines, and custom in-house systems [42]. Relative to the library-gallery examples (subsection 4.2), they introduce layout irregularities and design conventions that are uncommon in controlled gallery examples. The corpus consists of 25 bar charts, 26 line charts, and 9 scatterplots. These charts also provide strong reference annotations for evaluation. Many already contain carefully designed highlights, callouts, labels, and explanatory overlays created by domain experts [46]. These human-authored annotations reflect editorial intent and established practice. They serve as strong reference points, though a chart may admit other valid annotations. Starting from the annotated source visualizations, we first create an unannotated version of each chart in D3.js, which serves as the benchmarkâs canonical source representation. From this D3 implementation, we derive additional representations, including SVG and grammar-based specifications, through a combination of manual conversion and LLM-assisted translation. Because many source articles do not provide the original data, we reconstruct or synthesize data that closely match the source visualization while preserving the perceptual and structural properties required for the annotation tasks. We then derive benchmark tasks from the original annotated charts. In the professional corpus, each chart contributes a single base annotation objective corresponding to the published annotation. From this objective, we construct both intent-level and execution-level prompt variants and assign taxonomy-based metadata tags. Finally, we manually author five chart description conditions for each unannotated chart based on the source article. The full construction process is described in the supplementary materials. We make the final charts publicly accessible at annobench/charts â . 4.2 Charts in Visualization Library Galleries To complement the professional charts dataset, we also collect charts from widely used visualization libraries. Gallery examples are common in chart understanding and visualization research [61, 23, 34], and they provide controlled examples with stable underlying specifications. Although many visualization galleries could be included [61], supporting every ecosystem would require substantial engineering effort and evaluation cost. We therefore selected the Vega and Vega-Lite galleries because they provide diverse declarative chart specifications, stable source representations, and broad coverage of chart types while keeping the benchmark practical to build and maintain [51, 52, 61]. Consistent with the chart-representation dimension described in section 3, these sources provide canonical code-based and grammar-based specifications, from which we also derive raster and vector representations. Across the collected visualizations from Vega and Vega-Lite galleries, our library corpus covers a broad range of chart types, including bar, line, area, scatterplot, heatmap, histogram, map, node-link, radial, pie, and faceted chart variants (see Table 1). We also authored description conditions for the library charts. However, many gallery examples are designed primarily as usage demonstrations rather than domain-grounded visualizations, making it difficult to support richer contextual descriptions. As a result, the library corpus includes only the lower-level description conditions. To ensure compatibility with model input constraints, we kept chart data inline where possible. For very large representations exceeding 16,384 tokens, we externalized the data and included a minimal example in the prompt to communicate the data schema. We also retained a small number of large inline examples to test long-context behavior. We adopt a semi-automated task generation pipeline grounded in the annotation taxonomy of Rahman et al. [45]. Instead of authoring all tasks manually, we designed three generation templates for each annotation type and used them to guide LLM-based task generation. To preserve a realistic distribution of annotation types, we used stratified sampling [37] using the empirical distribution reported by Rahman et al.. For each chart, we randomly assign sampled annotation types and prompt templates, then repeat this process four times, yielding four candidate tasks per chart. During generation, we provide the LLM with the chart representations and available description conditions to ensure the tasks remain consistent with the chart content. We then manually review the generated tasks, remove ambiguous or incompatible cases, and rewrite the remaining tasks as paired intent-level and execution-level prompts corresponding to the same annotation objective. The Vega gallery contains 92 charts after excluding one example that demonstrates a video game; from these charts, we manually derive 214 annotation tasks. From the Vega-Lite gallery, after removing duplicates and filtering out small demonstration-only examples, we retain 192 charts and derive 394 annotation tasks. AnnoBench datasets are designed to be extensible. Users can incorporate new datasets by following the standardized dataset schema without requiring additional code changes. To facilitate this, we also provide a curated set of charts from the D3 Observable Gallery, including multi-level semantic descriptions and multiple chart representations (D3, SVG, and raster). These charts can be directly explored in annobench/playground â , where users can experiment with custom prompts and annotation tasks. This dataset curation process is discussed in the supplementary document. 5 AnnoBench Browser AnnoBench is designed as a multi-dimensional benchmark that captures the complexity of chart annotation across representations, semantic contexts, and task formulations. To fully leverage this richness, researchers must be able to systematically explore, configure, and analyze experiments across these dimensions. We therefore introduce the AnnoBench Browser, an interactive web-based interface that operationalizes the benchmark as an executable and navigable system. The browser, as shown in Abstract, enables users to explore datasets, configure experiments, and analyze model behavior within a unified framework, allowing them to study annotation performance across diverse conditions in a structured, interpretable manner. The browser exposes AnnoBench as a structured design space, allowing users to select datasets, tasks, chart representations, semantic description levels, and prompt variants to define controlled experimental conditions. Given the diversity of representations and output formats in the benchmark, the system automatically generates prompts and standardizes model outputs, enabling direct comparisons across configurations and models. A key feature of the browser is its ability to render and inspect model outputs across heterogeneous representations. For vector, grammar, and code-based charts, the system provides automatic compilation, visualization and error reporting, allowing users to evaluate both the rendered chart and its underlying representation. The browser also supports simultaneous human and VLM-based evaluation, result aggregation across experimental conditions, and analysis of performance trends and evaluator agreement. The AnnoBench browser execution pipeline automatically compiles supported chart representations into static images and uses VLM-Judge for automated evaluation. To support larger-scale evaluation, the system allows exporting results to ReVISit [12] studies, enabling standardized human evaluation workflows or crowdsourcing. The system can further import study results from ReVISit back into the AnnoBench browser for unified analysis. Together, these capabilities enable reproducible, large-scale benchmarking and fine-grained analysis of annotation behavior. Additional implementation details and system capabilities are described in the supplementary document. 6 Experiments We describe a series of experiments designed to evaluate the dimensions of the AnnoBench benchmark through controlled studies with contemporary models. Our goal is to understand how different design dimensions of AnnoBench influence annotation performance, rather than solely benchmarking models. Specifically, we conduct experiments that isolate and analyze the effects of chart representation (SectionË6.1), task-prompt specificity (SectionË6.2), and semantic chart descriptions (SectionË6.3), enabling us to assess how each dimension contributes to the difficulty and variability of chart annotation tasks. A key challenge is the scale of the benchmark, as the combinatorial space of configurations makes exhaustive evaluation computationally and practically prohibitive. Even restricting ourselves to the core dimensions in this paper would produce more than 273,960 output charts to generate and assess. Assuming an average of 4,000 input tokens and 4,000 output tokens per task, running this full matrix using the March 2026 Claude Opus 4.6 API rates would cost approximately $98,625 ($15/MTok input and $75/MTok output) [1], before accounting for retries, rendering failures, and human evaluation effort. The resulting manual assessment burden would be 2,283 hours (assuming 30 seconds per chart), which is even less practical. In place of such an enumerative evaluation, we conduct four one-dimension-at-a-time experiments. We conduct four experiments evaluating different dimensions. First, we look at the effect of representation, then prompt form, the effect of captions, and lastly model performance. In each experiment, we use what are believed to be the best configurations for each dimension, except when that dimension is under test. For instance, we use execution level prompting in all experiments besides the second, while in the first experiment, we use the most performant representation (which is then used in other experiments). This strategy keeps human and API costs manageable while still supporting controlled analysis of how individual dimensions, such as representation or prompt specificity, affect performance. Human and VLM-based Evaluation. For the first three experiments, we use stratified sampling [37] with n=15n=15 tasks per experiment condition. To increase sample size and improve statistical accuracy, we use AnnoBenchâs VLM-as-a-Judge feature to score an additional sample of equal size (n=30n=30). Because each study multiplies this base sample by the number of compared levels in its independent variable and by the number of tested models, the final number of scored outputs remains tractable while still supporting controlled comparisons. Under this design, the first experiment contains 15Ă4Ă5=30015Ă 4Ă 5=300 samples, the second contains 15Ă3Ă5=22515Ă 3Ă 5=225 samples, and the third contains 15Ă2Ă5=15015Ă 2Ă 5=150 samples. All samples in these experiments are scored by two authors and three VLM judges using the five-dimensional rubric introduced in subsection 3.2. The three VLM judges are GPT-5.2 (gpt-5.2), Claude Sonnet 4 (claude-sonnet-4-20250514), and Gemini 2.5 Flash (gemini-2.5-flash). The generated annotations are produced by five generation models under evaluation: GPT-5.2, Claude Sonnet 4, Gemini 2.5 Flash, DeepSeek-R1 14B (deepseek-r1:14b), and Gemma 3 27B (gemma3:27b). Thus, GPT-5.2, Claude Sonnet 4, and Gemini 2.5 Flash serve as both generation models and VLM judges, whereas DeepSeek-R1 14B and Gemma 3 27B serve only as generation models. We report inter-rater reliability (IRR) for the human evaluators in each experiment to show consistency in manual scoring. As noted in section 2, prior work does not yet establish how reliably VLM-as-a-Judge evaluation can assess visualization annotations. We therefore analyze the agreement between VLM-based and human ratings and discuss where automated judging is reliable and where it fails at the end of subsection 6.1. The AnnoBench analysis interface supports both manual filtering of runs and configuration-based selection of experiments. For any selected configuration, the interface presents aggregate scores, IRR between human evaluators, agreement between human and VLM-based judgments, and summary visualizations of model performance across dimensions and rubric categories. This analysis workflow is important not only for reporting final results, but also for understanding where annotation failures arise. All model run results, along with their human and VLM-judge evaluation results, are explorable in annobench/analysis â . Sampling Strategy. To avoid bias toward any single annotation type, we use proportionate stratified sampling over annotation categories (e.g., text, highlight, enclosure) for all experiments, following the empirical distribution observed in our corpus and motivated by prior studies of real-world annotation usage [44]. This yields evaluation sets that better reflect the distribution of annotation practices found in the wild. Failure Handling. For non-raster chart representations, models generate text that is compiled to the final annotated chart. While contemporary models are effective at generating text, they sometimes generate invalid or malformed specifications that do not run. When a model output fails, the system can provide execution logs feedback and re-invoke the model to obtain a corrected output. This mechanism helps separate failures of annotation reasoning from failures of format compliance. We allow up to three automated retry attempts per failed generation to correct these incidental errors to avoid an open-ended repair loop. If the output remains non-executable, we assign a score of 0 for that trial across all evaluation categories. This policy is intentionally conservative: a non-renderable output is unusable in practice. 6.1 Experiment 1: Evaluating Representation Robustness Our first experiment evaluates the effect of chart representations. Different representations expose different kinds of information to the model. For instance, while raster images preserve visual appearance, they hide structural editability. Similarly, code-level representations expose the full construction process, with a lower level of abstraction that may not be aware of the semantics of the final image. This experiment sought to answer the questionâWhich chart representations best support annotation generation? We evaluate models across four core representation formats: Raster (provided as base64-encoded image content or image-grid input, depending on model API support), Vector (SVG), Grammar (Vega specification), and Code (JS with D3 implementation). The underlying tasks are matched across other conditions, allowing us to isolate the effect to only the representation dimension. Figure 3: Distribution of rubric scores for Experiment 1â4. In each chart of the experiments, each column group corresponds to each evaluation criterion, and rows correspond to the evaluator (Human or VLM-Judge). Each column corresponds to the dimension we are evaluating in the experiment that it represents. Within each bar, score percentages are shown between 0 (lowest intensity) and 3 (highest intensity). Results. Figure 3 (Experiment 1) summarizes the score distributions across the evaluation rubric using the average scores of two human visualization experts and three VLM judges, applied to outputs from five generation models under evaluation: GPT-5.2, Claude Sonnet 4, Gemini 2.5 Flash, DeepSeek-R1 14B, and Gemma 3 27B. Even with the three-retry-and-fix prompting process, we find DeepSeek-R1 14B to have compilation errors or completely missing annotations in all outputs, scoring zero across all evaluation criteria. Looking at the examples, we see the model continuously providing instructions for adding annotations to the chart, rather than returning the completed chart for each retry. Next, Claude Sonnet 4 had compilation errors in 18% of benchmark runs, Gemini 2.5 Flash in 13%, and GPT-5.2 in 8%. Among these errors from all models, approximately 30% were from SVG, 70% were from Vega specifications, and none were caused by D3 charts. Due to the poor performance of DeepSeek-R1 14B, we exclude it from the remaining experiments. Across the two human evaluators, Cohenâs Îș exceeds 0.93 for all five rubric dimensions (precise, visible, preservant, relevant, and consistent), indicating near-perfect agreement. We further compute the mean absolute distance (MAD) between the two human evaluators and find it to be below 0.22 for all rubric dimensions (scored 0-3), suggesting that disagreements are not only infrequent but also small in magnitude. For human evaluation results, the D3 code representation achieves the highest scores across all dimensions, except visible, where raster performs better. The Vega representation followed D3 on visible, relevant, and preservant, while SVG performed better on precise and consistent than Vega. We adopt the D3 code representation for the remaining experiments as the benchmarkâs default best-effort representation. The VLM-as-a-Judge results show a broadly similar ordering among D3, SVG, and Vega representations (Ï>0.4Ï>0.4 for all, with the Gemini 2.5 Flash judge yielding the highest agreement), but much weaker agreement with human evaluation for raster outputs (average Ï=0.3Ï=0.3). Across VLM judges, raster appears to perform slightly better than D3. To examine this disagreement, we computed humanâVLM correlations for each chart representation and rubric category (see Table 2). For raster outputs, agreement is weak, with Ï<0.4Ï<0.4 in most rubric categories. We discuss qualitative failure cases and possible reasons in section 7. We also separately tested vector, grammar, and code representations for a chart with large inline data. As expected, all models returned truncated outputs in all three cases, yielding zero scores across all evaluation dimensions. This experiment yields three important findings. First, chart annotation capability varies across representation types, with code-based inputs performing best overall in human evaluation. Second, raster chart annotation remains a major weakness for current models: although models can often generate visible annotation-like elements, they often fail to faithfully preserve the underlying charts. Third, VLM-as-a-Judge evaluation is itself representation-sensitive. While it is moderately reliable for structured textual chart outputs, it is much less reliable for raster outputs, where it tends to overestimate quality. Table 2: Spearman correlation (Ï) between human evaluation and the best-performing VLM judge for each chart representation (Numeral colors mapped Ïâ[0,1]Ïâ[0,1] (redâgreen)). Agreement is weakest for raster charts and strongest for vector-based outputs. Precise Visible Relevant Preservant Consistent D3 0.63 0.44 0.54 0.72 0.41 Grammar 0.64 0.48 0.73 0.57 0.54 Vector 0.75 0.80 0.79 0.97 0.84 Raster 0.32 0.22 0.48 0.39 0.47 fair (0.2â0.4) moderate (0.4â0.6) substantial (0.6â0.8) almost perfect (0.8â1.0) 6.2 Experiment 2: Evaluating Task Prompting Variances Our second experiment investigates the effects of task specificity. A given annotation goal can be framed either as a high-level intent or as an explicit procedural instruction (see tasks in Figure 1). To this end, we sought to understandâHow does the level of prompt specificity influence the annotation capability of models? We draw on a similar design to the previous experiment, but with representation fixed and task form varied using the best-performing chart representation (D3). We consider both prompt levels present in the data set, intent-level (which keep task open-ended) and execution-level (which explicitly constrains the annotation type) This experiment probes whether models can independently infer effective annotation strategies or primarily succeed when given concrete guidance. Results. Figure 3 (Experiment 2) shows the score distributions for human and VLM-as-Judge evaluators. We find that the execution-level prompt achieves higher average performance across all evaluation criteria for human-evaluator scores, which aligns with our expectations because explicit instructions reduce ambiguity and constrain the solution space to our expectations for the annotation task. Inter-rater reliability between the two human evaluators was high. They had near-perfect agreement (exact score matches for precise, preservant, and visible) across most samples. For consistent, however, agreement was lower (average Spearman Ïâ0.6Ïâ 0.6), reflecting the inherently subjective nature of stylistic evaluation. Judging styling consistency can be highly subjective, especially with the open-ended nature of intent-level prompts. We find the improvements to be most pronounced in precise, preservant, and visible, suggesting that explicit instructions help models correctly localize annotations, preserve the input chart, and place annotations without overlap or conflict with other chart elements. Since this experiment is performed only with the code representation, where models tend to perform best, most annotation tasks received comparably high scores from both humans and VLM judges across all evaluation factors. However, VLM-Judge tends to score slightly lower on tasks with execution-level prompts and higher on those with intent-level prompts. As a result, intent-level prompts receive slightly higher scores by VLM-judges than execution-level prompts. We suspect that this discrepancy arises from the evaluation behavior of VLM judges. Because intent-level prompts are less constrained, it is easier for a judge model to rationalize that an output satisfies the promptâs intent, even when the annotation may look suboptimal to a human evaluator. In other words, VLM judges tend to reward semantic plausibility over structural correctness. We notice a key behavioral difference across prompt levels in the selection of annotation types. For example, when asked to âhighlight the significant wave duration for Delta waveâŠâ models operating under intent-level prompts typically use simple strategies such as adding a text label or arrow. In contrast, execution-level prompts that explicitly specify an enclosure (e.g., âadd a vertical shaded y-spanning region during the Delta waveâŠâ) consistently produce the intended enclosure-based annotation (see third chart in Abstract (B)). Notably, we find that intent-level prompts almost never result in enclosure annotations unless explicitly requested. These results highlight a limitation in current models: although they can interpret high-level goals, they often default to simpler or more common annotation strategies (e.g., text, connector, and indicator [45]) rather than exploring the full design space of possible annotations. This pattern is consistent with broader observations in LLM research, where models are less reliable under underspecified prompts and remain weaker at autonomous planning than when task requirements are made explicit [59, 57]. 6.3 Experiment 3: Evaluating Dependency of Chart Captions Our third experiment evaluates how performance depends on the accompanying textual chart descriptions. Although AnnoBench includes five semantic caption levels, it is not clear whether models genuinely benefit from richer chart descriptions once they are already provided with a complete chart, such as code or grammar representations. Here we sought to understandâHow much do models rely on textual chart descriptions when generating annotations? We continue to follow the default evaluation setup, but vary the semantic description level across the lowest (No Description), mid (Statistical and Relational), and highest (Contextual and Domain-Specific, or the highest description available). Results. Figure 3 (Experiment 3) shows score distributions across rubric dimensions for both human evaluators and the VLM-as-Judge. Overall, human evaluation shows a moderate improvement in precise, visible, and relevant as the semantic description level increases. preservant remains constant across all conditions, with perfect scores (100%) for all description levels. This is expected as we have seen in subsection 6.1, as preservation primarily depends on the chart representation (constant D3 code in this experiment) rather than semantic context. The consistent scores vary across levels, suggesting that while semantic descriptions help with task alignment, stylistic consistency is influenced more by the prompt specificity and design guidelines (as seen in subsection 6.2). The VLM-as-Judge results exhibit weaker sensitivity to semantic description levels compared to human evaluation. Across all evaluation criteria, score distributions remain relatively similar, with only minor variations. This behavior suggests that VLM judges rely more heavily on the final visual plausibility of the output rather than on how well the annotation aligns with the underlying chart semantics, since the VLM-Judges were not provided with the same chart semantic descriptions to avoid bias. As a result, improvements driven by richer semantic descriptions are less noticed by VLM evaluations. This experiment shows that semantic descriptions provide measurable but secondary benefits for chart annotation. Richer descriptions improve performance mainly in dimensions tied to semantic understanding, such as precise and relevant, but have limited effect on stylistic consistency and chart preservation. Finally, semantic description augmentation cannot fully compensate for limitations in a modelâs ability of annotating visualizaiton charts, and improving robustness in these areas remains a key direction for future work. 6.4 Experiment 4: Model Performance comparison Our final experiment compares the overall annotation capability of state-of-the-art models under the standardized, best-effort configuration. While previous experiments isolate the effects of representation, prompt specificity, and semantic descriptions, it remains unclear how different models perform when these factors are optimized. Here we seek to understandâHow do current state-of-the-art models compare in their ability to generate chart annotations. We use the same results from all previous experiments and compare each experimental setup, both aggregated and individually, to identify whether model performance varies across configurations, where one model performs better in one configuration but poorly in another. Results. Figure 3 (Experiment 4) shows the average distribution of scores across rubric dimensions for all evaluated models. Across human evaluation, we observe a clear separation between higher-performing and lower-performing models. GPT-5.2 and Claude Sonnet 4 consistently achieve the highest scores across most evaluation criteria, with Gemini 2.5 Flash following closely behind. In contrast, smaller or less specialized models such as DeepSeek-R1 14B and Gemma 3 27B perform substantially worse across all criteria. Notably, DeepSeek-R1 14B frequently fails to produce valid or meaningful annotations, resulting in near-zero scores across multiple dimensions. Gemma 3 27B demonstrates partial capability but struggles with precise and relevant, often producing incomplete or misaligned annotations. Across individual experimental conditions, we observe that the relative ranking of models remains largely consistent. Models that perform well in one configuration tend to perform well across others, indicating that improvements are not tied to a specific representation, prompt type, or semantic context. This stability suggests that annotation capability reflects a more general underlying competence rather than sensitivity to a particular dimension. Finally, even the strongest models exhibit variability across evaluation dimensions, suggesting that chart annotation remains a challenging, multifaceted task requiring both semantic understanding and precise visual execution to add annotations to any chart representation. 7 Discussion & Conclusion Chart annotation is a multi-dimensional challenge that depends jointly on representation, instruction specificity, and semantic context. No single factor alone determines success; rather, annotation quality emerges from the interaction between chart understanding and the ability to execute precise visual modifications, which we evaluated with a five-dimensional evaluation framework. While evaluating annotations from various models, we encountered some key observations. Chart representations have the largest impact. Results from Experiment 1 show that D3 substantially outperforms other chart representations for annotation authoring, likely for two key factors: its JavaScript foundation is well represented in LLM training data, making it easier for models to manipulate, and annotations can be added incrementally without regenerating the entire chart. In contrast, raster representations frequently lead to severe preservant failures, where even simple annotation attempts introduce hallucinated marks or distort the chart to the point of unusability; models may also shift data-encoded marks to create space for annotations, artificially inflating visible scores (see Abstract (B) and supplementary document for examples). SVG exhibits a different limitation: while it preserves structure, its verbosity often pushes models beyond context or even token limits, resulting in truncated, malformed, or hallucinated outputs. Overall, the performance gap across representations is strikingâhighlighting that structured representations play a critical role in enabling reliable annotation generation and should be prioritized in future visualization systems. Prompt specificity improves execution but exposes limited design inference. Results from Experiment 2 show that models benefit substantially from explicit execution-level prompts, which consistently outperform intent-level prompts across most evaluation criteria. This suggests that current models are much better at following annotation instructions than at inferring an appropriate annotation strategy from a high-level communicative goal. In particular, intent-level prompts often lead models to default to annotation types that do not better match the intended message. As chart annotation is not only a matter of chart understanding, but also of visual planning and design reasoning, the results suggest that improving annotation quality will require models that can reason more flexibly about alternative annotation strategies, rather than relying primarily on explicit procedural guidance. Semantic chart descriptions help when annotations are relevant to it. Experiment 3 shows that richer chart descriptions provide measurable, though secondary, benefits for annotation generation. Higher-level chart captions improve performance primarily on dimensions tied to semantic understanding, especially precise and relevant, indicating that models use textual context to better identify what should be annotated and why. However, these gains are modest compared with those from chart representation or prompt specificity, and they do not improve preservant. This suggests that semantic descriptions can support chart interpretation, but they cannot compensate for weaknesses in structured editing, spatial reasoning, or visual design. In other words, giving a model more context about the chart helps it understand the task, but does not necessarily help it carry out the annotation well. VLM-based judges overestimate quality on raster outputs. Qualitative analysis of disagreement cases reveals that VLM-based judges tend to over-credit raster outputs, often judging annotations as correctly placed relative to the original chart even when the underlying chart content has been distorted. In such cases, annotations appear visually plausible but no longer correspond to the correct data, leading to low human scores in precise, relevant, and preservant (see supplementary document for examples). While automated judging aligns reasonably well with human evaluation for structured representations, it performs less reliably on raster inputs, where it struggles to detect input corruption and may overestimate semantically incorrect results. These findings suggest that, although VLM-based judging is useful for scaling evaluation, it should be applied with caution in settings involving high design flexibility, such as annotation tasks. Extensibility of AnnoBench Browser. AnnoBench Browser is released as a Python library and is designed to be extensible along several dimensions. Users can add new visualization datasets by following the documentation and standardized data schema. For unsupported chart representations, custom rendering functions can be registered through the API, enabling integration with additional visualization frameworks. Furthermore, the system supports extension of semantic description strategies and annotation prompt levels, allowing researchers to introduce new chart description methodologies or task variations. Importantly, AnnoBench is not limited to LLMs or VLMs; it can be used to evaluate any system that takes a chart and a task prompt as input and produces an annotated chart as output, including rule-based methods, program synthesis approaches, or hybrid human-AI workflows. Limitations. Despite the breadth of AnnoBench, it has several limitations. First, although we consider multiple chart representations and datasets, our evaluation covers only a subset of possible configurations due to practical constraints on API cost and human assessment effort. We also evaluate a limited set of models; while sufficient for comparative analysis, broader model coverage would strengthen generality. Second, while we include a diverse set of annotation types, our tasks are constrained by the distributions present in available datasets and may not fully capture all real-world annotation scenarios. Third, our evaluation varies one dimension at a time, which enables controlled analysis but may miss higher-order interactions between dimensions. Fourth, our evaluation relies on a fixed rubric and a limited number of human evaluators. Although inter-rater reliability is high, some dimensions, particularly stylistic consistency, remain inherently subjective. While we incorporate VLM-based judging to scale evaluation, we observe clear failure modes, and such methods should not be treated as a substitute for human assessment. Designing more reliable automated judges for visual tasks remains an open challenge. Fifth, our experiments focus on static charts and do not consider interactive or animated visualizations, which introduce additional complexities for both annotation design and evaluation. Finally, model capabilities evolve rapidly, which may reduce the long-term relevance of the specific model performance results reported in this paper. For this reason, we do not include model outputs in the benchmark dataset and instead present them as a point-in-time evaluation to demonstrate the utility of AnnoBench for analyzing visualization annotation dimensions. Future work. AnnoBench opens several directions for future research. From a dataset perspective, we plan to expand the benchmark by adding additional chart sources, including multi-level task prompts to existing datasets such as the D3 Observable Gallery (section 4), as well as incorporating new visualization ecosystems, interactive and animated charts, and domain-specific datasets. Increasing the diversity of chart types, data domains, and annotation tasks will further strengthen its role as a general benchmark for understanding and editing visualization. Beyond dataset expansion, AnnoBench provides a foundation for studying visualization tasks more broadly. While this work focuses on chart annotation, the underlying framework supports any task that involves visualizations and produces either textual or visual outputs. We plan to extend AnnoBench into a unified platform for evaluating a wider class of visualization tasks, including chart editing, transformation, and explanation, and to explore hybrid evaluation approaches that combine rule-based validation with model-based judging to better capture the complexities of visualization tasks. Conclusion. We present AnnoBench, a benchmark for evaluating chart annotation through per-instance correctness criteria and controlled variation in chart representation, task specification, and semantic context. Across our experiments, current models show a clear gap between understanding what a chart says and correctly editing it to express that understanding. Performance improves substantially when models receive structured, editable chart representations and explicit execution-level instructions, but raster inputs, underspecified prompts, and chart-preservation failures remain persistent weaknesses. These findings suggest that progress in annotation automation will require more than stronger chart comprehension alone. It will also require methods that support faithful chart editing, stronger inference of annotation strategies, and evaluation procedures that do not reward visually plausible but semantically incorrect outputs. AnnoBench provides a concrete basis for that work. Acknowledgements.We used the generative AI tool GPTâ5 by OpenAI to edit, paraphrase, and restructure the text, which was later reviewed and revised. This work was supported in part by a grant from NSF (#2402719). References [1] Anthropic (2026) Claude api pricing: model costs, prompt caching, and batch processing. Note: https://platform.claude.comAccessed: 2026-03-30 External Links: Link Cited by: §6. [2] M. Bancilhon, A. Wright, S. Ha, R. J. Crouser, and A. Ottley (2023) Why combining text and visualization could improve bayesian reasoning: a cognitive load perspective. In ACM CHI, p. 1â15. External Links: Document Cited by: §2.1. [3] S. Bateman, R. L. Mandryk, C. Gutwin, A. Genest, D. McDine, and C. Brooks (2010) Useful junk? the effects of visual embellishment on comprehension and memorability of charts. In ACM CHI, p. 2573â2582. External Links: Document Cited by: §2.1. [4] A. Bendeck and J. Stasko (2024) An empirical evaluation of the gpt-4 multimodal language model on visualization literacy tasks. IEEE Transactions on Visualization and Computer Graphics. External Links: Document Cited by: §1, §2.2. [5] M. A. Borkin, Z. Bylinskii, N. W. Kim, C. M. Bainbridge, C. S. Yeh, D. Borkin, H. Pfister, and A. Oliva (2015) Beyond memorability: visualization recognition and recall. IEEE Transactions on Visualization and Computer Graphics 22 (1), p. 519â528. External Links: Document Cited by: §2.1. [6] M. Borkin, A. A. Vo, Z. Bylinskii, P. Isola, S. Sunkavalli, A. Oliva, and H. Pfister (2013) What makes a visualization memorable?. IEEE Transactions on Visualization and Computer Graphics 19 (12), p. 2306â2315. External Links: Document Cited by: §2.1. [7] C. Bryan, A. Mishra, H. Shidara, and K. Ma (2020) Analyzing gaze behavior for text-embellished narrative visualizations under different task scenarios. Visual Informatics 4 (3), p. 41â50. External Links: Document Cited by: §2.1. [8] N. Chen, Y. Zhang, J. Xu, K. Ren, and Y. Yang (2024) Viseval: a benchmark for data visualization in the era of large language models. IEEE Transactions on Visualization and Computer Graphics. External Links: Document Cited by: §1, §2.2. [9] Y. Chen, S. Barlowe, and J. Yang (2010) Click2annotate: automated insight externalization with rich semantics. In IEEE VAST, p. 155â162. External Links: Document Cited by: §1. [10] Y. Chen, Y. Wu, S. Shen, Y. Xie, L. Shen, H. Xiong, and Y. Luo (2025) ChartMark: a structured grammar for chart annotation. arXiv preprint arXiv:2507.21810. External Links: Document Cited by: §1, §2.1, §3.2. [11] R. Chun (2020) Giving guidance to graphs: evaluating annotations of data visualizations for the news. Visual Communication Quarterly 27 (2), p. 84â97. External Links: Document Cited by: §2.1. [12] Z. Cutler, J. Wilburn, H. Shrestha, Y. Ding, B. Bollen, K. A. Nadib, T. He, A. McNutt, L. Harrison, and A. Lex (2026) ReVISit 2: a full experiment life cycle user study framework. IEEE Transactions on Visualization and Computer Graphics 32 (1), p. 13â23. External Links: Document Cited by: §5. [13] A. Fan, F. Lei, M. Mancenido, A. M. Maceachren, and R. Maciejewski (2024) Understanding reader takeaways in thematic maps under varying text, detail, and spatial autocorrelation. In ACM CHI, p. 1â17. External Links: Document Cited by: §1, §2.1. [14] A. Fan, Y. Ma, M. Mancenido, and R. Maciejewski (2022) Annotating line charts for addressing deception. In ACM CHI, External Links: Document Cited by: §1, §1, §2.1. [15] T. Gao, J. R. Hullman, E. Adar, B. Hecht, and N. Diakopoulos (2014) NewsViews: an automated pipeline for creating custom geovisualizations for news. In ACM CHI, p. 3005â3014. External Links: Document Cited by: §1. [16] L. W. Ge, Y. Cui, and M. Kay (2023) CALVI: critical thinking assessment for literacy in visualizations. In ACM CHI, p. 1â18. External Links: Document Cited by: §2.2. [17] G. Guo, J. Stasko, and A. Endert (2024) What we augment when we augment visualizations: a design elicitation study of how we visually express data relationships. In ACM Advanced Visual Interfaces, p. 1â5. External Links: Document Cited by: §2.1. [18] J. Hong, C. Seto, A. Fan, and R. Maciejewski (2025) Do llms have visualization literacy? an evaluation on modified visualizations to test generalization in data interpretation. IEEE Transactions on Visualization and Computer Graphics 31 (10), p. 7004â7018. External Links: Document Cited by: §1, §2.2. [19] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. (2025) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1â55. External Links: Document Cited by: §3.2. [20] J. Hullman, N. Diakopoulos, and E. Adar (2013) Contextifier: automatic generation of annotated stock visualizations. In ACM CHI, p. 2707â2716. External Links: Document Cited by: §1, §1, §2.1, §2.1, §3.2. [21] K. Kafle, B. Price, S. Cohen, and C. Kanan (2018) DVQA: understanding data visualizations via question answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 5648â5656. External Links: Document Cited by: §2.2. [22] S. Kantharaj, R. Ramesh, A. Shetty, M. M. Chandrasekaran, and A. Modi (2022) Chart-to-text: a large-scale benchmark for chart summarization. In Findings of the Association for Computational Linguistics (ACL), p. 2694â2706. External Links: Document, Link Cited by: §1. [23] D. H. Kim, E. Hoque, and M. Agrawala (2020) Answering questions about charts and generating visual explanations. In ACM CHI, p. 1â13. External Links: Document Cited by: §4.2. [24] C. Kittivorawong, D. Moritz, and J. Heer (2020) FastLabels: fast and flexible overlap detection for chart labeling with occupancy bitmaps. IEEE Visualization Conference (VIS). External Links: Document Cited by: §1. [25] H. Kong, Z. Liu, and K. Karahalios (2017) Internal and external visual cue preferences for visualizations in presentations. Computer Graphics Forum 36 (3), p. 515â525. External Links: Document Cited by: §2.1. [26] H. Kong, W. Zhu, Z. Liu, and K. Karahalios (2019) Understanding visual cues in visualizations accompanied by audio narrations. In ACM CHI, p. 1â13. External Links: Document Cited by: §2.1. [27] N. Kong and M. Agrawala (2012) Graphical overlays: using layered elements to aid chart reading. IEEE Transactions on Visualization and Computer Graphics 18 (12), p. 2631â2638. External Links: Document Cited by: §1, §1, §2.1, §3.2, §3.2, §3.2. [28] S. Lee, S. Kim, and B. C. Kwon (2016) Vlat: development of a visualization literacy assessment test. IEEE Transactions on Visualization and Computer Graphics 23 (1), p. 551â560. External Links: Document Cited by: §1, §2.2. [29] H. Li, Y. Wang, and H. Qu (2024) Where are we so far? understanding data storytelling tools from the perspective of human-ai collaboration. In ACM CHI, p. 1â19. External Links: Document Cited by: §1. [30] M. Lisnic, C. Polychronis, A. Lex, and M. Kogan (2023) Misleading beyond visual tricks: how people actually lie with charts. In ACM CHI, p. 1â21. External Links: Document Cited by: §1, §2.1. [31] A. Lundgard and A. Satyanarayan (2022) Accessible visualization via natural language descriptions: a four-level model of semantic content. IEEE Transactions on Visualization and Computer Graphics 28 (1), p. 1073â1083. External Links: Document Cited by: §2.2, §3.1. [32] Y. Luo, J. Tang, and G. Li (2021) NvBench: a large-scale synthesized dataset for cross-domain natural language to visualization task. arXiv preprint arXiv:2112.12926. External Links: Document Cited by: §2.2. [33] A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque (2022) Chartqa: a benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics (ACL), p. 2263â2279. External Links: Document Cited by: §1, §2.2. [34] A. M. McNutt and R. Chugh (2021) Integrated visualization editing via parameterized declarative templates. In ACM CHI, p. 1â14. External Links: Document Cited by: §4.2. [35] N. Methani, P. Ganguly, M. M. Khapra, and P. Kumar (2020) Plotqa: reasoning over scientific plots. In IEEE/CVF Winter Conference on Applications of Computer Vision, p. 1527â1536. External Links: Document Cited by: §1, §2.2. [36] K. Mukherjee, D. Ren, D. Moritz, and Y. Assogba (2025) EncQA: benchmarking vision-language models on visual encodings for charts. IEEE Transactions on Visualization and Computer Graphics. Note: Early Access; also on arXiv:2508.04650 External Links: Document, Link Cited by: §2.2. [37] J. Neyman (1992) On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. In Breakthroughs in statistics: Methodology and distribution, p. 123â150. External Links: Document Cited by: §4.2, §6. [38] Observable (2024) D3 gallery. External Links: Link Cited by: Appendix S2. [39] A. Ottley, A. Kaszowska, R. J. Crouser, and E. M. Peck (2019) The curious case of combining text and visualization. In EuroVis (Short Papers), External Links: Document Cited by: §1, §2.1. [40] S. Pandey and A. Ottley (2023) Mini-vlat: a short and effective measure of visualization literacy. Computer Graphics Forum 42 (3), p. 1â11. External Links: Document Cited by: §2.2. [41] S. Pandey and A. Ottley (2025) Benchmarking visual language models on standardized visualization literacy tests. In Computer Graphics Forum, p. e70137. External Links: Document Cited by: §2.2. [42] P. Parsons (2021) Understanding data visualization design practice. IEEE Transactions on Visualization and Computer Graphics 28 (1), p. 665â675. External Links: Document Cited by: §4.1. [43] M. D. Rahman, B. Doppalapudi, G. J. Quadri, and P. Rosen (2025) A survey on annotations in information visualization: empirical studies, applications and challenges. IEEE Transactions on Visualization and Computer Graphics. External Links: Document Cited by: §1, §1, §2.1, §3.2. [44] M. D. Rahman, G. J. Quadri, B. Doppalapudi, D. A. Szafir, and P. Rosen (2024) A qualitative analysis of common practices in annotations: a taxonomy and design space. IEEE Transactions on Visualization and Computer Graphics. External Links: Document Cited by: §3.1, §6. [45] M. D. Rahman, G. J. Quadri, B. Doppalapudi, D. A. Szafir, and P. Rosen (2024) A qualitative analysis of common practices in annotations: a taxonomy and design space. IEEE Transactions on Visualization and Computer Graphics. External Links: Document Cited by: §1, §2.1, §2.1, §3.2, §3.2, §4.1, §4.2, §6.2. [46] M. D. Rahman, G. J. Quadri, and P. Rosen (2023) Exploring annotation strategies in professional visualizations: insights from prominent us news portals. VisComm Workshop at IEEE VIS. External Links: Document Cited by: §2.1, §3.2, §4.1, §4.1. [47] M. D. Rahman, G. J. Quadri, D. A. Szafir, and P. Rosen (2024) Exploring annotation taxonomy in grouped bar charts: a qualitative classroom study. Inf. Visualization, p. 14738716241270247. External Links: Document Cited by: §1. [48] M. D. Rahman, M. R. Zaman, A. McNutt, and P. Rosen (2025) AnnoGram: an annotative grammar of graphics extension. IEEE Visualization Conference (VIS). External Links: Document Cited by: §1, §2.1, §3.2. [49] M. Rahman, M. T. R. Laskar, S. Joty, and E. Hoque (2025) Text2vis: a challenging and diverse benchmark for generating multimodal visualizations from text. In Empirical Methods in Natural Language Processing (EMNLP), p. 31837â31862. External Links: Document Cited by: §1, §2.2. [50] D. Ren, M. Brehmer, B. Lee, T. Höllerer, and E. K. Choe (2017) Chartaccent: annotation for data-driven storytelling. In IEEE PacificVis, p. 230â239. External Links: Document Cited by: §1, §1, §2.1, §3.2, §3.2. [51] A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer (2016) Vega-lite: a grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics 23 (1), p. 341â350. External Links: Document Cited by: §4.2. [52] A. Satyanarayan, R. Russell, J. Hoffswell, and J. Heer (2016) Reactive vega: a streaming dataflow architecture for declarative interactive visualization. IEEE Transactions on Visualization and Computer Graphics. External Links: Document, Link Cited by: §4.2. [53] S. C. Spivak and M. Tory (2025) Fiction vs friction: challenges in evaluating llms on data visualization tasks. In Human-centered Evaluation and Auditing of Language Models (HEAL) Workshop at CHI 2025, Note: Workshop paper External Links: Link Cited by: §1, §2.1. [54] C. Stokes, C. X. Bearfield, and M. A. Hearst (2023) The role of text in visualizations: how annotations shape perceptions of bias and influence predictions. IEEE Transactions on Visualization and Computer Graphics. External Links: Document Cited by: §1, §2.1. [55] C. Stokes, V. Setlur, B. Cogley, A. Satyanarayan, and M. A. Hearst (2022) Striking a balance: reader takeaways and preferences when integrating text and charts. IEEE Transactions on Visualization and Computer Graphics 29 (1), p. 1233â1243. External Links: Document Cited by: §1, §1, §2.1. [56] B. Tang, A. Boggust, and A. Satyanarayan (2023) Vistext: a benchmark for semantically rich chart captioning. In Findings of the Association for Computational Linguistics (ACL), p. 7268â7298. External Links: Document Cited by: §1, §2.2, §3.1. [57] K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati (2023) On the planning abilities of large language models: a critical investigation. Advances in Neural Information Processing Systems 36, p. 75993â76005. Note: Also available as NeurIPS 2023 proceedings paper External Links: Document, Link Cited by: §6.2. [58] C. Wang, B. Lee, S. M. Drucker, D. Marshall, and J. Gao (2025) Data formulator 2: iterative creation of data visualizations, with ai transforming data along the way. In ACM CHI, p. 1â17. External Links: Document Cited by: §1. [59] C. Yang, Y. Shi, Q. Ma, M. X. Liu, C. KĂ€stner, and T. Wu (2025) What prompts donât say: understanding and managing underspecification in llm prompts. External Links: 2505.13360, Document Cited by: §6.2. [60] D. Yang, L. Zhang, Z. Yue, L. Chen, Y. Xu, W. Wang, and Q. Jin (2025) ChartM3: benchmarking chart editing with multimodal instructions. In ACM International Conference on Multimedia, p. 5001â5009. External Links: Document Cited by: §2.2. [61] J. Yang, A. M. McNutt, and L. Battle (2024) Considering visualization example galleries. In IEEE Visual Languages and Human-Centric Computing (VL/HCC), p. 329â343. External Links: Document Cited by: §4.2, §4.2. [62] X. Zhao, X. Liu, Y. Haoyue, X. Luo, F. Zeng, J. Li, Q. Shi, and C. Chen (2025) Chartedit: how far are mllms from automating chart analysis? evaluating mllmsâ capability via chart editing. In Findings of the Association for Computational Linguistics (ACL), p. 3616â3630. External Links: Document Cited by: §2.2. [63] Q. Zhi, A. Ottley, and R. Metoyer (2019) Linking and layout: exploring the integration of text and visualization in storytelling. Computer Graphics Forum 38 (3), p. 675â685. External Links: Document Cited by: §2.1. [64] Z. Zhu, M. Jia, Z. Zhang, L. Li, and M. Jiang (2025) MultiChartQA: benchmarking vision-language models on multi-chart problems. In Findings of the Association for Computational Linguistics (ACL), p. 11341â11359. External Links: Document, Link Cited by: §2.2. [65] B. Zou, M. Cai, J. Zhang, and Y. J. Lee (2024) Vgbench: evaluating large language models on vector graphics understanding and generation. In Empirical Methods in Natural Language Processing (EMNLP), p. 3647â3659. External Links: Document Cited by: §2.2. Appendix Appendix S1 AnnoBench Browser Capabilities Exploration and Dataset Navigation. AnnoBench supports flexible filtering and selection of dataset charts and benchmark runs directly through the browser interface. Users can select models, datasets, task subsets, chart representations, semantic description levels, and prompt types to define custom experimental conditions. These selections can then be used for exploration, running with newer models, ablation tests, or any other purposes. Experiment Configuration and Execution. Based on the subset selection, users can configure a new configuration for new experiments. Because evaluating chart annotation often involves multiple representations and output formats, the system automatically handles prompt construction and result processing based on the selected configuration. For example, users may provide a single representation (e.g., raster or D3 code) or combine multiple representations within the same prompt. The system adapts the prompt template and output expectations accordingly, enabling direct comparison between unimodal and multi-representation settings. To support fast, practical, large-volume experimentation, the browser integrates with an asynchronous execution pipeline that can run multiple tasks concurrently. All experiment configurations, inputs, outputs, and metadata are stored in a standardized format, allowing results to be reproduced, shared, and re-analyzed. Appendix S2 D3 Observable Gallery Dataset We also collect charts from the D3 Observable Gallery [38], which provides low-level code-based chart representations that serve as canonical source inputs. Because Observable examples are notebook-specific, we standardize them into a consistent JavaScript format suitable for benchmarking. We exclude 56 interactive charts and 24 animated charts that fall outside the scope of our study, and convert the remaining 53 static charts into standardized JavaScript representations. We retain this corpus as an auxiliary resource for chart representations and description conditions rather than as part of the task-bearing benchmark. It broadens the coverage of source representations available in AnnoBench and supports future extensions. AnnoBench is designed to be extensible, and we provide a clear dataset structure and instructions for adding additional datasets in the documentation. Appendix S3 Implementation Details AnnoBench is implemented as a Python library with a web-based interface, following a data-driven design. Each dataset is organized into charts and tasks, where charts include multiple representations (e.g., raster images, SVG, Vega/Vega-Lite specifications, or D3 code), and tasks define annotation objectives, prompt variants, and evaluation settings. This structure allows the same chart to be presented in different modalities while keeping tasks consistent across representations. The benchmark pipeline automatically constructs model inputs by combining the selected chart representations, semantic descriptions, and task prompts. Depending on the configuration, models may receive visual inputs, structured specifications, or code representations. Outputs are then standardized and, when necessary, rendered into images so that both humans and automated methods can evaluate the results consistently across representations. AnnoBench supports multiple evaluation strategies, including rule-based checks, similarity-based comparisons, human evaluation, and VLM-as-a-judge methods. The system is designed to handle errors during generation or rendering and to allow rerunning failed cases, separating model reasoning failures from execution issues. To support large-scale experimentation, the system manages benchmark runs, stores results in a structured format, and enables reproducible analysis across different configurations. The web interface provides tools for launching experiments, inspecting model outputs, and analyzing results across datasets, models, and evaluation criteria. Overall, the implementation is designed to be flexible and extensible, allowing researchers to evaluate diverse models, chart representations, and annotation tasks within a unified framework. Appendix S4 Dataset Creation Process Figure 4 illustrates the dataset construction process for both visualization library charts and professionally produced charts. For library-based charts, we start from publicly available chart specifications or examples, normalize the underlying data, and generate multiple chart representations (e.g., Vega, D3, SVG) using a combination of direct extraction and LLM-assisted conversion. These representations are then compiled into raster images to ensure consistency across representations. Finally, we author chart descriptions and annotation tasks based on the chart content. For professional charts, the process begins with collecting chart images from real-world sources and reconstructing their underlying data and structure. We implement a canonical version of each chart in D3.js, from which additional representations are derived. Similar to the library pipeline, we generate raster outputs and create corresponding chart descriptions and annotation tasks. This two-track pipeline ensures that AnnoBench contains both reproducible, structured examples and visually rich, real-world charts. Figure 4: Overview of the AnnoBench dataset construction and review process. Top panel illustrates the pipeline for charts collected from visualization libraries, where chart images and data are extracted from repositories, additional representations (e.g., Vega, D3, SVG) are translated with LLMs and manually verified, raster versions are compiled, and corresponding captions and annotation tasks are created. The bottom panel shows the pipeline for professionally produced charts, where images are scraped from online sources, data are reconstructed or synthesized, charts are reimplemented in D3.js, and additional representations are generated before producing raster outputs, captions, and tasks. Appendix S5 Annotation Evaluation Examples Figure 5 presents representative examples of model outputs and their evaluation under our rubric. Each example includes the input chart, the annotation task prompt provided to the model, and the resulting annotated output. For each output, we report both human evaluation scores and VLM-as-a-judge scores across all rubric dimensions, along with brief explanations that justify the assigned scores. These examples highlight common success cases and failure modes across models. In particular, they illustrate how errors may arise from incorrect target selection, inaccurate annotation content, poor spatial placement, or stylistic inconsistencies, even when other aspects of the annotation are correct. They also demonstrate differences between human and VLM-based evaluation, where visually plausible outputs may still fail to align with the underlying data or task requirements. Together, these examples provide qualitative insight into how the evaluation framework captures annotation quality and how different models behave across a range of annotation scenarios. Figure 5: We provide a list of examples of an input chart, the annotation task prompt provided to the model, the modelâs output, and the evaluation on the output performed by a human and a VLM-Judge. Below the scorings, we provide human explanation and the VLMs explanation for the scoring that they did. Figure 6: Figure 5 continued. Figure 7: Figure 5 continued. Figure 8: Figure 5 continued.