Paper deep dive
Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents
Srishti Palani, Vidya Setlur
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/17/2026, 4:03:59 AM
Summary
The paper introduces Lexara-RF, a reference-free evaluation framework for Conversational Visual Analytics (CVA) agents. It proposes 13 metrics based on visualization design theory and Gricean cooperative principles to assess outputs using only the user prompt, data, and model response. The framework verifies data fidelity, visual encoding effectiveness, and conversational alignment, demonstrating high correlation with human judgments and outperforming surface-similarity NLG baselines.
Entities (10)
Relation Signals (8)
Data Fidelity → partof → Lexara-RF
confidence 95% · We propose Lexara-RF... 13 reference-free metrics... Data Fidelity verifies that the chart’s values are computationally correct.
Factual Grounding → partof → Lexara-RF
confidence 95% · Factual Grounding (Quality) verifies claims in NL against VV
Visual Encoding → partof → Lexara-RF
confidence 95% · Visual Encoding verifies that non-positional channels... are used appropriately.
Lexara-RF → extends → Lexara
confidence 90% · Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free extension of the Lexara evaluation framework
Lexara-RF → uses → Gricean cooperative principles
confidence 90% · 13 metrics operationalize visualization design theory and Gricean cooperative principles
Lexara-RF → outperforms → BERTScore
confidence 85% · outperforms surface-similarity NLG baselines, BLEU, ROUGE-L, and BERTScore
Lexara-RF → outperforms → BLEU
confidence 85% · outperforms surface-similarity NLG baselines, BLEU, ROUGE-L, and BERTScore
Show Me → usedin → Lexara-RF
confidence 80% · Chart Type verifies... (2) Show Me [16], which produces a ranked set of appropriate marks from the inferred fields.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ended queries. Evaluating these multimodal outputs is challenging: curated reference benchmarks are costly to author, cannot comprehensively capture the space of valid responses, and are unavailable in production. Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free set of metrics that scores CVA outputs using only the prompt, data, and model response. We reformulate evaluation as verification: 13 metrics operationalize visualization design theory and Gricean cooperative principles as computable consistency, intent-alignment, and design validity checks. On a human-rated corpus of CVA test-cases, Lexara-RF achieves alignment comparable to reference-based formulations, outperforms surface-similarity NLG baselines, and localizes structurally grounded failures with high accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2609.17842v1
- Canonical: https://arxiv.org/abs/2609.17842v1
Trouble viewing inline? Open PDF directly →
Full Text
38,848 characters extracted from source content.
Expand or collapse full text
1091 or Algorithm Introduction Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents Srishti Palani †thanks: E-mails: srishti.palani, vsetlur@salesforce.com. Vidya Setlur11footnotemark: 1 Affiliation: Tableau Research, Palo Alto, CA, USA Abstract Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ended queries. Evaluating these multimodal outputs is challenging: curated reference benchmarks are costly to author, cannot comprehensively capture the space of valid responses, and are unavailable in production. Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free set of metrics that scores CVA outputs using only the prompt, data, and model response. We reformulate evaluation as verification: 13 metrics operationalize visualization design theory and Gricean cooperative principles as computable consistency, intent-alignment, and design validity checks. On a human-rated corpus of CVA test-cases, Lexara-RF achieves alignment comparable to reference-based formulations, outperforms surface-similarity NLG baselines, and localizes structurally grounded failures with high accuracy. keywordsConversational Visual Analytics, Evaluation Metrics, Visualization Design, Cooperative Communication Principles. Advances in large language models (LLMs) have enabled conversational visual analytics (CVA) systems that allow users to generate and refine visualizations through natural language [29, 30]. While these systems lower barriers to data exploration, evaluating their outputs remains a challenge. CVA agents produce multimodal outputs, i.e., data-grounded visualizations paired with natural language explanations that must be assessed for data fidelity, encoding effectiveness, and alignment with user intent across a conversation. Existing evaluation approaches [15, 4, 23] compare model outputs against curated references (i.e., acceptable responses for a given input). However, the space of valid CVA responses is inherently open-ended: a single query may admit multiple valid visualizations differing in chart type, encoding, or aggregation, making exhaustive coverage infeasible and introducing false negatives [7]. Moreover, constructing and maintaining references is costly, requiring expert effort to enumerate acceptable outputs and update them as data and tasks evolve [15, 4]. In production settings, references are often unavailable, as user queries and data continuously shift, preventing evaluation at inference time and scalable monitoring of CVA agent quality [28, 3, 11]. To address these limitations, we propose Lexara-RF, a reference-free extension of the Lexara evaluation framework [23] that assesses CVA outputs as intrinsic artifacts grounded in the user prompt and underlying data. We reformulate evaluation as verification: each metric tests whether a response satisfies conditions derived from visualization design theory [17, 6] and conversational principles [8, 30], rather than whether it matches a curated reference output. We contribute: (1) 13 reference-free metrics spanning expressiveness, effectiveness, and conversational alignment, with full pseudocode and judge prompts; (2) a validation on a human-rated CVA corpus (N = 120) showing parity with reference-based metrics at this sample size and substantially higher alignment than surface-similarity NLG baselines (BLEU, ROUGE-L, BERTScore); and (3) evidence that the metrics localize structurally grounded failure modes while surfacing semantically ambiguous ones as distributed signals, supporting hybrid evaluation. Together, these contributions establish reference-free evaluation as a practical complement to reference-based benchmarks for CVA, enabling continuous integration, production monitoring, and rapid prompt iteration where reference authoring is prohibitive. 1 Related Work This work builds on three research themes: (1) theoretical foundations for visualization design, (2) automated visualization evaluation, and (3) conversational quality in analytical dialogue. 1.1 Visualization Design Theory A rich body of work distinguishes between visualization expressiveness (encoding all and only the intended facts) and effectiveness (leveraging perceptual channels for accurate interpretation) [17, 6]. Prior research further formalizes visual encoding channels and their perceptual properties, alongside principles of graphical integrity that emphasize truthful and efficient data representation [2, 36]. More recent frameworks organize visualization design across levels from data abstraction to interaction, outlining how systems support analytical workflows [20, 31]. These insights have been operationalized in systems such as Show Me and Draco, which generate visualizations using heuristics and constraint-based reasoning [16, 19]. Unlike these generative systems, Lexara-RF operationalizes the same principles for evaluation, transforming them into reference-free metrics that verify whether generated visualizations satisfy established design constraints. 1.2 Automated Visualization Evaluation Benchmarks such as nvBench [15], VisEval [4], and NLVCorpus [33] evaluate models against curated reference outputs, while Lexara [23] extends this paradigm with graded metrics spanning visualization and natural language quality. More recent work explores richer evaluation pipelines by decomposing assessments across representation levels, training learned critics from human examples, or using latent similarity representations [26, 24, 13, 32]. However, all ultimately depend on reference visualizations, either explicitly or implicitly via training data. This dependence incurs high annotation costs, penalizes valid alternative responses, and precludes evaluation in production where references are unavailable. Parallel reference-free work in NLP [14, 18, 37, 10, 9] evaluates text without references but do not address the multimodal, design-sensitive nature of CVA. Lexara-RF bridges this gap by grounding reference-free evaluation in data, visualization design principles, and conversational alignment. 1.3 Conversational Quality for Analytical Dialogue CVA systems also produce natural language explanations that must be evaluated for communication quality. Grice’s cooperative principles (Quality, Quantity, Relation, and Manner) provide a foundation for assessing truthfulness, informativeness, relevance, and clarity [8]. Prior work adapts these principles to analytical dialogue, emphasizing factual consistency between text and visualization, context preservation across turns, coherence, and insightfulness [35, 30, 5, 21]. These frameworks are inherently reference-free, evaluating responses based on internal consistency and conversational context rather than comparison to expected outputs. However, they have not been systematically operationalized as computable metrics for CVA systems. Lexara-RF instantiates these principles through verification by checking consistency across text, visualization, data, and conversation history, enabling end-to-end reference-free evaluation. Unlike learned critics [24, 26] and reference-free NLG evaluators [14, 18, 37, 10, 9], which rely on supervision from reference exemplars, Lexara-RF requires no reference data at training or evaluation time and grounds its metrics directly in visualization theory and conversational principles. 2 Lexara-RF: Reference-Free Evaluation Metrics Test User Utterance Model’s CVA Response (V, NLNL) Computed Lexara-RF Metrics Interpretability of Metrics 1 show top 10 customers by profit in 2023 “These are the most profitable customers in 2023.” Data Fidelity = 92%; Field Similarity = 50%; Sort Accuracy = 70%; Filter Accuracy = 100%; Chart Type = 100%; Axis Accuracy = 100%; Visual Encoding = 100%; Interactivity = 100%. Overall Viz 89%. Factual Grounding = 30%; Assumptions Disclosure = 25%; Insightfulness = 40%; Coherence = 90%. Overall NL 46%. Expressiveness and Conversational Alignment failures: field swap. V picks Sales where U asked for Profit affecting Field Similarity, Sort Accuracy, and Data Fidelity scores. Conversational Alignment surfaces the downstream consequence: Factual Grounding and Insightfulness drop because NLNL’s “most profitable” claim is incorrect; Assumptions Disclosure drops because the silent Profit to Sales substitution is never flagged. 2 show profit by state “Profit varies across the United States, with California leading the way.” Data Fidelity = 100%; Field Similarity = 100%; Filter Accuracy = 100%; Sort Accuracy = 100%; Chart Type = 100%; Axis Accuracy = 100%; Visual Encoding = 40%; Interactivity = 100%. Overall Viz 93%. Factual Grounding = 70%; Assumptions Disclosure = 50%; Insightfulness = 30%; Coherence = 90%. Overall NL 60%. Effectiveness and Conversation Alignment failures: encoding mismatch. Sequential single-hue palette on signed Profit (range −$25-\$25K to $7676K) renders profit losses visually indistinguishable from low-profit states, affecting Visual Encoding. Insightfulness, Assumptions Disclosure drop because this obscuring palette choice is not articulated. Expressiveness encodes correct fields and marks. Table 1: Worked examples showing how Lexara-RF reference-free metrics evaluate CVA responses and help diagnose failures. We propose Lexara-RF, a set of reference-free metrics that evaluate CVA outputs as intrinsic artifacts grounded in the user prompt and underlying data source, rather than curated references. Lexara-RF operates on the following runtime inputs: the user utterance U, visualization specification V and natural language explanation NLNL, underlying data source and schema (D,S)(D,S), and conversation history H. While aligned with prior CVA evaluation dimensions, our metrics derive signals directly from visualization theory and conversational principles instead of curated references. They are implemented as verification procedures that test consistency, intent alignment, and design validity, producing graded scores on a common 0–100 scale. Full pseudocode for each metric and additional worked examples spanning aggregation, filter, sort, encoding, and multi-turn failure modes are provided in the paper’s Supplementary Materials. For reproducibility, all implementation choices were fixed a priori: semantic similarity uses sentence-transformers/all-MiniLM-L6-v2 embeddings with cosine similarity [27, 38]; fuzzy string matches use RapidFuzz token-set ratios normalized to [0,1][0,1] [1]; metric outputs are normalized to the shared [0,100][0,100] scale; thresholds include 0.30.3 for Data Fidelity plausibility, ≥0.5≥ 0.5 for filter field/value matching, and 1.5×IQR1.5×IQR for outlier detection; and full 13-metric evaluation requires approximately 18 seconds per response, or about USD 5 in API charges for the 120-response corpus based on the OpenAI API pricing schedule [22], with details in the Supplementary Materials. We employ fixed weights and penalties while scoring the metrics to reflect the relative importance and reliability of different signals, grounded in visualization theory and verification strength rather than learned from data. The weights, 13 metric dimensions, and 0–100 scoring scale are inherited from Lexara [23], preserving interpretability and enabling direct comparison between reference-based and reference-free formulations (§4.2). Signals verified directly against the data receive the highest weight as objective evidence; structural and semantic checks receive moderate weight; multiplicative penalties capture errors with compounding perceptual impact (e.g., axis misuse). We organize our reference-free metrics into three themes: expressiveness, effectiveness, and conversational alignment. Rather than comparing responses to references, the metrics verify consistency with visualization theory and conversational principles. 2.1 Expressiveness of the Visualization Expressiveness captures whether a visualization encodes the intended data and analytical intent. We verify this by checking that the specification V is consistent with the data (D,S)(D,S) and aligned with the user query U. Data Fidelity verifies that the chart’s values are computationally correct. We compute three signals: (1) s1s_1 (structural validity) checks that fields exist in the schema, operations match data types, and fields plausibly correspond to concepts extracted from U; (2) s2s_2 (pipeline re-execution) independently executes V’s transformation pipeline on D and compares the result to the chart output, treating the specification as the claim and the data as the evidence. Alternative aggregations (e.g., SUM vs. COUNT) are evaluated to detect mislabeling; (3) s3s_3 (intent alignment) verifies that the fields and aggregations in V match the analytical intent inferred from U. If s1<50s_1<50, the specification is structurally invalid and the score is s1s_1; otherwise, the score is 0.2s1+0.5s2+0.3s30.2\,s_1+0.5\,s_2+0.3\,s_3, weighting re-execution highest as the strongest correctness signal. Field Similarity verifies that the visualization binds the fields implied by the user’s query. We extract analytical concepts from U (e.g., “revenue,” “category,” “last quarter”) and match each concept to schema columns in S. For each concept, we score candidate fields using a weighted combination of semantic similarity, fuzzy string matching, and type compatibility, selecting the highest-scoring column as the best match. The final score measures how many of these best-matched fields are actually used in V. This approach captures semantic alignment rather than exact matching: a query for “revenue by category” should map to fields like Sales and Category, while a chart using unrelated fields such as Discount or Region scores low despite being syntactically valid. Filter Accuracy verifies that the visualization applies the constraints implied by the user’s query. We extract temporal, categorical, and numeric filter intents from U, augmenting relevant filters from prior turns using conversation history H. We normalize V’s filters and match them to these intents using semantic similarity over fields and values. Scoring uses an asymmetric penalty that prioritizes missing constraints: 100⋅m/|FU|−25⋅miss−15⋅extra100· m/|F_U|-25·miss-15·extra, where missing filters are penalized more than unrequested additions. This reflects that omitted constraints violate user intent, while extra filters only narrow the result. Only explicit and persistent filters are evaluated; implicit data source defaults are ignored. Sort Accuracy verifies that the visualization respects ordering implied by the user’s query. We detect sort signals in U, including superlatives (e.g., “top,” “highest”) and explicit directives (e.g., “sort by X”), and infer the sort field and direction. If no sort is requested, the metric returns 100100 when V is unsorted. Otherwise, we compare the inferred sort to V’s specification. Scoring combines field and direction alignment: 100⋅sf⋅d100· s_f· d, where sfs_f measures similarity between expected and actual sort fields, and d∈1.0,0.5d∈1.0,0.5 rewards correct or reversed direction. Missing required sorting scores 00. 2.2 Effectiveness of the Visualization Effectiveness captures whether a visualization uses perceptual channels appropriately for accurate interpretation, matching encodings to the data and analytical task [17, 6, 2]. Chart Type verifies that the chosen mark type aligns with the user’s intent and data characteristics. We combine two signals: (1) intent inferred from U (e.g., trend, comparison, distribution, correlation) to identify expected chart types, and (2) Show Me [16], which produces a ranked set of appropriate marks from the inferred fields. The score is 100 if V matches the top recommendation, 50 if it appears in the candidate set, and 0 otherwise, with a small bonus for matching the inferred intent. Axis Accuracy verifies that axis assignments support accurate interpretation. Since position is the most precise perceptual channel [6], incorrect assignments can distort meaning. We audit key conventions with multiplicative penalties: (a) temporal intent requires encoding time on a positional channel; (b) time should appear on the x-axis when both axes are present; (c) bar charts should place measures on the y-axis and categories on x; (d) bar and area charts should use a zero baseline; and (e) explicit axis directives in U must be respected. Violations incur multiplicative penalties to reflect their compounding effect on interpretability. Visual Encoding verifies that non-positional channels (color, shape, size, opacity, text) are used appropriately. Perceptual principles recommend matching channels to data types (e.g., hue for categories, size for magnitude) [2, 19]. We score each channel using practice (alignment between channel and data type), and relevance (whether the encoded field is implied by U). The final score is the mean of 0.6⋅0.6· practice and 0.4⋅0.4· relevance across used channels. Mismatched encodings reduce interpretability, while irrelevant fields introduce visual clutter. Interactivity verifies that tooltips provide relevant details on demand [31]. Required fields include those referenced in U and those defining the visualization (e.g., axes, color). We compute two signals: coverage, the fraction of required fields present, and usability, which penalizes duplicate fields and missing units. The final score is 0.7⋅0.7· coverage +0.3⋅+0.3· usability, capturing both completeness and clarity. 2.3 Conversational Alignment CVA responses include natural language explanations that must support analytical reasoning and interpretation. We ground evaluation in Grice’s cooperative principles, adapted for analytical conversation, which assess utterances in context rather than against references [8, 30]. Factual Grounding (Quality) verifies claims in NLNL against V and recomputed values from D, penalizing contradictions more heavily than omissions. Assumptions Disclosure (Quantity) checks whether implicit choices in V (e.g., mappings, filters, aggregations) are surfaced in NLNL. Programmatically extracted assumptions are scored by an LLM judge, since undisclosed assumptions cannot enter shared context [5]. Insightfulness extends Quantity by assessing whether NLNL surfaces meaningful patterns (e.g., trends, outliers), grounded in summary statistics from D and V [21]. Coherence (Manner) evaluates clarity and logical structure of NLNL. Follow-up Relevance (Relation) assesses multi-turn alignment using conversation history H, verifying persistence of filters, analytical scope, and entity references [5]. Overall Viz and Overall NL are the mean of their constituent metrics, with per-metric scores reported for interpretability. 3 Validation of Reference-Free CVA Metrics We evaluate Lexara-RF along three dimensions: Concurrent Validity: alignment with expert human judgments; Convergent Validity: agreement with reference-based evaluation despite not requiring curated expected outputs, and discriminant separation from surface-similarity NLG baselines; and Diagnostic Utility: localization of interpretable failure modes via per-metric score decomposition. All evaluations reuse the human-rated CVA response corpus from Palani & Setlur [23]: N=120N=120 responses stratified by metric score ranges, ambiguity types (syntactic, semantic, pragmatic), and analytical task types (descriptive, comparative, trend), drawn from a generative corpus of evaluation experiments across eight LLMs (GPT-5 family, o3, o4-mini, Claude Opus 4, Claude 3.7 Sonnet, DeepSeek R1), six system prompts, and analyst-supplied data sources spanning finance, education and healthcare domains. Each response carries expert human ratings on a 0–100 scale per metric from two Lexara-trained CVA practitioners (median linear-weighted Cohen’s κ=0.65κ=0.65 for viz metrics, 0.630.63 for NL metrics, per [23]). Because of the sample size, we present this as a descriptive validation rather than an equivalence test; per-metric results are provided in the Supplementary Materials. 3.1 Concurrent Validity: Human Alignment All metrics are computed solely from (U,V,NL,D,S,H)(U,V,NL,D,S,H) without reference outputs. Human scores are the mean of two expert ratings. Inter-rater agreement was substantial, with a median linear-weighted Cohen’s κ=0.78κ=0.78 (IQR 0.710.71–0.830.83) across metrics and corpus-wide κ=0.76κ=0.76, comparable to prior CVA evaluation on the same corpus [23]. Per-metric correlations are reported in the Supplementary Materials, with aggregate values of Spearman’s ρmed=0.60 _med=0.60 (range 0.420.42–0.740.74 across metrics) and Kendall’s τbmed=0.47 _b_med=0.47 (across metrics). Deterministic, execution-grounded metrics (Data Fidelity, Filter Accuracy, Sort Accuracy, Factual Grounding) show the strongest alignment with human judgment (ρ=0.68ρ=0.68–0.740.74), whereas rubric-based LLM-as-Judge metrics (Insightfulness, Coherence) exhibit lower but consistent agreement (ρ=0.42ρ=0.42–0.520.52), reflecting the greater subjectivity of interpretive evaluation [21, 23]. This distinction also provides practical guidance on which metrics are suitable for automated gating versus human review. 3.2 Convergent Validity: Reference-Based Alignment We compare each reference-free metric against (i) its reference-based counterpart from Palani & Setlur [23] and (i) standard NLG baselines (BLEU [25], ROUGE-L [12], BERTScore [39]), all evaluated on the same response set. Agreement with human judgment is assessed using Steiger’s Z-test [34] for dependent correlations, with Holm–Bonferroni correction across the 13 comparisons. Reference-free metrics fail to reject parity with reference-based formulations on 1111 of 1313 dimensions (p>0.05p>0.05 after correction), with mean |Δρ|=0.04±0.01| ρ|=0.04± 0.01 on the remaining two dimensions. Because N=120N=120 provides limited power to detect small effects, this result should be interpreted as indistinguishable performance at the current sample size, not strict measurement equivalence; detecting Δρ=0.05 ρ=0.05 with 80% power would require approximately N≈400N≈ 400. These findings support reference-free metrics as a practical complement to reference-based evaluation when reference authoring is prohibitively expensive. NLG baselines. BLEU, ROUGE-L, and BERTScore correlate with human judgment at ρ=0.18±0.02ρ=0.18± 0.02, 0.24±0.030.24± 0.03, and 0.46±0.030.46± 0.03, respectively; substantially below reference-free and reference-based formulations (see Supplementary Materials). This indicates that surface similarity to a reference NLNL string does not capture the structurally grounded quality of CVA responses, and that the parity result above is non-trivial: any metric correlating at ρ≈0.60ρ≈ 0.60 with humans would already be doing more than a strong text-similarity baseline. Divergent case analysis. We analyzed cases where reference-free and reference-based scores differed by more than 0.15 (90th percentile of pilot-phase score variation), yielding 36 responses. Two annotators independently coded each case (Cohen’s κ=0.74κ=0.74) as valid alternatives, phrasing-invariant disclosures, or true errors, resolving disagreements through discussion [23, 30]. The distribution was 44%, 32%, and 24%, respectively. Although descriptive of this subset only, the low proportion of true errors suggests that most divergences reflect legitimate under-specification in CVA tasks rather than failures of the reference-free metrics. 3.3 Diagnostic Utility: Failure Localization To examine whether individual metrics surface their target failure modes, we analyzed a stratified subset of N=90N=90 responses from the Lexara corpus [23] across performance terciles (high, medium, low) and the three metric pillars. Per-mode counts range from 5 to 11 cases (Supp C Table 2) and reflect the natural distribution of failure types in the source corpus rather than enforced balance; we treat the structurally grounded subset (54 responses across six modes) as our primary basis for localization claims. Two Lexara-trained annotators independently labeled each response using an 11-category failure taxonomy adapted from Lexara (Cohen’s κ=0.79κ=0.79), with disagreements resolved through discussion (Supplementary Materials). The taxonomy was derived independently of the metric implementations in Section 2 and comprised: • Expressiveness: incorrect data values, field mapping, aggregation, filters, and sorting. • Effectiveness: inappropriate chart type, axis errors, and visual encoding mismatches. • Conversational Alignment: factual grounding failures, undisclosed implicit assumptions, and multi-turn context loss. For each labeled response, we identify the metric with the largest negative standardized deviation from its corpus-level distribution and treat it as the primary diagnostic signal, while assessing whether deviations are localized or distributed. Two examples illustrate this behavior. T47 omitted a year filter from the preceding turn produces a sharp drop in Filter Accuracy while other metrics remain near baseline, cleanly isolating the failure. The failure to disclose an implicit population-normalization assumption in T63 produces coordinated declines in Assumptions Disclosure, Insightfulness, and Coherence, reflecting a conversational failure. Across the corpus, structurally grounded failures (e.g., missing filters, aggregation errors, baseline violations) produce localized deviations in their corresponding metrics, whereas semantically ambiguous failures (e.g., undisclosed assumptions, pragmatic interpretation) produce correlated declines across conversational alignment metrics. Thus, the framework localizes well-defined structural errors while surfacing ambiguity as distributed signals. Per-failure-mode breakdown in the Supplementary Materials. 4 Discussion: Limitations & Future Work These results support a measured claim: Lexara-RF aligns with expert judgment at correlations comparable to reference-based methods (median ρ=0.60ρ=0.60), exceeds surface-similarity NLG baselines, and localizes structurally grounded failure modes at ≥90%≥ 90\% accuracy without curated references. At N=120N=120, observed parity reflects limited statistical power rather than strict equivalence; accordingly, we position Lexara-RF as a complement to reference-based evaluation that eliminates per-test-case ground-truth authoring. Rubric-based metrics show lower agreement and weaker localization, warranting different gating thresholds and continued human review. Three factors plausibly explain why rubric-based metrics underperform: (i) inherent subjectivity in interpretive tasks like Insightfulness and Coherence [21]; (i) LLM-as-Judge variance even at temperature 0 (details in supplementary materials); and (i) scope overlap among Insightfulness, Coherence, and Assumptions Disclosure, which manifests as correlated rather than localized signals when failures are semantically ambiguous (§4.3). Promising improvement paths include ensemble judging across model families, calibration of rubric anchors against held-out human ratings, and decomposing broader rubrics into narrower verification primitives. Only 24% of reference-free vs. reference-based disagreements (§3.2) are true metric errors, whereas 44% are valid alternatives, suggesting much of what reference-based evaluation flags is legitimate under-specification rather than metric failure. Per-metric decomposition effectively localized failure dimensions in our case studies, suggesting promise for regression triage and prompt debugging in CVA pipelines. Three limitations bound our claims: First, the framework is validated only on Vega-Lite-style specifications; extending metrics that inspect encodings or interaction requires adaptation to other grammars (e.g., ggplot, D3). Second, evaluation is limited to English utterances and judge prompts. Third, validation uses the Lexara corpus with inherited metric weights; broader datasets spanning grammars, languages, and analytical tasks are needed for data-driven calibration. Future work should target a larger validation corpus spanning multiple grammars and languages, data-driven weight calibration, and extension to richer CVA tasks such as multi-table joins and calculated fields. Our results highlight a hybrid evaluation paradigm: automated, reference-free metrics like Lexara-RF can reliably handle structurally grounded correctness checks, while human judgment remains essential for interpretive and subjective aspects of analytical communication. Data Availability. The 120-response validation corpus is available at ://github.com/SrishtiPalani/Lexara-CVA-Eval/tree/main/lexara_rf_corpus. Supplementary materials have all details for reproducibility. References [1] M. Bachmann (2025) RapidFuzz: Rapid Fuzzy String Matching in Python and C++. Note: ://github.com/rapidfuzz/RapidFuzzAccessed 2026-06-24 Cited by: §2. [2] J. Bertin (1983) Semiology of graphics: diagrams, networks, maps. University of Wisconsin Press, Madison, WI. Note: ISBN: 978-0299090609 Cited by: §1.1, §2.2, §2.2. [3] E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley (2017) The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. In IEEE International Conference on Machine Learning and Systems, External Links: Document Cited by: Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [4] N. Chen, Y. Zhang, J. Xu, K. Ren, and Y. Yang (2025) VisEval: A Benchmark for Data Visualization in the Era of Large Language Models. IEEE Transactions on Visualization and Computer Graphics 31 (1), p. 1301–1311. External Links: Document Cited by: §1.2, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [5] H. H. Clark and S. E. Brennan (1991) Grounding in Communication. In Perspectives on Socially Shared Cognition, L. B. Resnick, J. M. Levine, and S. D. Teasley (Eds.), p. 127–149. External Links: Document Cited by: §1.3, §2.3, §2.3. [6] W. S. Cleveland and R. McGill (1984) Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods. Journal of the American Statistical Association 79 (387), p. 531–554. External Links: Document Cited by: §1.1, §2.2, §2.2, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [7] S. Fu, K. Xiong, X. Ge, S. Tang, W. Chen, and Y. Wu (2020) Quda: Natural Language Queries for Visual Data Analytics. External Links: Document Cited by: Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [8] H. P. Grice (1975) Logic and Conversation. In Syntax and Semantics, Vol. 3: Speech Acts, P. Cole and J. L. Morgan (Eds.), p. 41–58. Note: ISBN: 978-0126235500 Cited by: §1.3, §2.3, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [9] T. S. Kim, H. Lee, Y. Lee, and J. Kim (2026) Evalet: Evaluating Large Language Models through Functional Fragmentation. In Conference on Human Factors in Computing Systems, External Links: Document Cited by: §1.2, §1.3. [10] W. Kryscinski, B. McCann, C. Xiong, and R. Socher (2020) Evaluating the Factual Consistency of Abstractive Text Summarization. In ENLP, B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), p. 9332–9346. External Links: Document Cited by: §1.2, §1.3. [11] H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale (2011) Empirical Studies in Information Visualization: Seven Scenarios. IEEE Transactions on Visualization and Computer Graphics 18 (9), p. 1520–1536. External Links: Document Cited by: Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [12] C. Lin (2004) Rouge: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, p. 74–81. Cited by: §3.2. [13] C. Liu, C. Da, X. Long, Y. Yang, Y. Zhang, and Y. Wang (2025) SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding. arXiv preprint arXiv:2506.21319. Cited by: §1.2. [14] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In ENLP, H. Bouamor, J. Pino, and K. Bali (Eds.), p. 2511–2522. External Links: Document Cited by: §1.2, §1.3. [15] T. Luo, C. Huang, L. Shen, B. Li, S. Shen, W. Zeng, N. Tang, and Y. Luo (2025) nvBench 2.0: A Benchmark for Natural Language to Visualization under Ambiguity. External Links: Document Cited by: §1.2, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [16] J. Mackinlay, P. Hanrahan, and C. Stolte (2007) Show Me: Automatic Presentation for Visual Analysis. IEEE Transactions on Visualization and Computer Graphics 13 (6), p. 1137–1144. External Links: Document Cited by: §1.1, §2.2. [17] J. Mackinlay (1986) Automating the Design of Graphical Presentations of Relational Information. ACM Transactions on Graphics 5 (2), p. 110–141. External Links: Document Cited by: §1.1, §2.2, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [18] P. Manakul, A. Liusie, and M. Gales (2023) SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In ENLP, External Links: Document Cited by: §1.2, §1.3. [19] D. Moritz, C. Wang, G. L. Nelson, H. Lin, A. M. Smith, B. Howe, and J. Heer (2019) Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. IEEE Transactions on Visualization and Computer Graphics 25 (1), p. 438–448. External Links: Document Cited by: §1.1, §2.2. [20] T. Munzner (2009) A Nested Model for Visualization Design and Validation. IEEE Transactions on Visualization and Computer Graphics 15 (6), p. 921–928. External Links: Document Cited by: §1.1. [21] C. North (2006) Toward Measuring Visualization Insight. IEEE Computer Graphics and Applications 26 (3), p. 6–9. External Links: Document Cited by: §1.3, §2.3, §3.1, §4. [22] OpenAI (2026) OpenAI api pricing. Note: ://openai.com/api/pricing/Accessed 2026-06-24 Cited by: §2. [23] S. Palani and V. Setlur (2026) Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics. In Conference on Human Factors in Computing Systems, External Links: Document Cited by: §1.2, §2, §3.1, §3.1, §3.2, §3.2, §3.3, §3, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [24] B. Pan, Y. Fu, K. Wang, J. Lu, L. Pan, Z. Qian, Y. Chen, G. Wang, Y. Zhou, L. Zheng, Y. Tang, Z. Wen, Y. Wu, J. Lu, B. Zhu, M. Zhu, B. Zhang, and W. Chen (2025) VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation. External Links: Document Cited by: §1.2, §1.3. [25] K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002) BLEU: A Method for Automatic Evaluation of Machine Translation. p. 311–318. External Links: Document Cited by: §3.2. [26] L. Podo, M. Ishmal, and M. Angelini (2024) Vi(E)va LLM! A Conceptual Stack for Evaluating and Interpreting Generative AI-based Visualizations. External Links: Document Cited by: §1.2, §1.3. [27] N. Reimers and I. Gurevych (2019) Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In ENLP, Cited by: §2. [28] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J. Crespo, and D. Dennison (2015) Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28. Cited by: Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [29] V. Setlur, S. E. Battersby, M. Tory, R. Gossweiler, and A. X. Chang (2016) Eviza: A Natural Language Interface for Visual Analysis. In ACM Symposium on User Interface Software and Technology, p. 365–377. External Links: Document Cited by: Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [30] V. Setlur and M. Tory (2022) How Do You Converse with an Analytical Chatbot? Revisiting Gricean Maxims for Designing Analytical Conversational Behavior. In Conference on Human Factors in Computing Systems, p. 1–17. External Links: Document Cited by: §1.3, §2.3, §3.2, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents, Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents. [31] B. Shneiderman (1996) The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. In IEEE Symposium on Visual Languages, p. 336–343. External Links: Document Cited by: §1.1, §2.2. [32] Y. Song, X. Zhao, and R. C. Wong (2024) Marrying Dialogue Systems with Data Visualization: Interactive Data Visualization Generation from Natural Language Conversations. In Conference on Knowledge Discovery and Data Mining, p. 2733–2744. External Links: Document Cited by: §1.2. [33] A. Srinivasan, N. Nyapathy, B. Lee, S. M. Drucker, and J. Stasko (2021) Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations. In Conference on Human Factors in Computing Systems, p. 1–10. External Links: Document Cited by: §1.2. [34] J. H. Steiger (1980) Tests for Comparing Elements of a Correlation Matrix. Psychological Bulletin 87 (2), p. 245. External Links: Document Cited by: §3.2. [35] M. Tory and V. Setlur (2019) Do What I Mean, Not What I Say! Design Considerations for Supporting Intent and Context in Analytical Conversation. In Visual Analytics Science and Technology, p. 93–103. External Links: Document Cited by: §1.3. [36] E. R. Tufte (1983) The visual display of quantitative information. Graphics Press. Note: ISBN: 978-0961392147 Cited by: §1.1. [37] A. Wang, K. Cho, and M. Lewis (2020) Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), p. 5008–5020. External Links: Document Cited by: §1.2, §1.3. [38] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou (2020) MiniLM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, Vol. 33. External Links: Document Cited by: §2. [39] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi (2019) BERTscore: Evaluating Text Generation with BERT. External Links: Document Cited by: §3.2.